implementing lessons learned: establishing a culture of evidence-based assessment
DESCRIPTION
Implementing Lessons Learned: Establishing A Culture of Evidence-Based Assessment. Marilee J. Bresciani, Ph.D. Associate Professor, Postsecondary Education and Co-Director of the Center for Educational Leadership, Innovation, and Policy San Diego State University 3590 Camino Del Rio North - PowerPoint PPT PresentationTRANSCRIPT
Implementing Lessons Learned: Establishing A Culture of Evidence-Based Assessment
Marilee J. Bresciani, Ph.D.Associate Professor, Postsecondary Education andCo-Director of the Center for Educational Leadership, Innovation, and PolicySan Diego State University3590 Camino Del Rio NorthSan Diego, California, [email protected]
My Lens
Former K-12 Instructor Doctorate in administration,
curriculum, and instruction Primarily an administrator; teaching 1-
2 courses a year; now a FT faculty member
OBA consultant and Reviewer for WASC and Australia
My research is in exploring systematic implementation of OBA and its effects
My Context
Participants are primarily instructors from 2- and 4-year public institutions with some proprietary institutions representation
You teach users of technology You care about what your learners learn You use what you know to inform
designers of technology and curriculum how to do a better job with their products
Overview Overview of Outcomes-Based
Assessment Discuss its History and Importance Share Strategies for Implementing
Sustainable Evidence-Based Decision Making within your Locus of Control
Applying Lessons Learned to Implementation
Questions to Consider as You Return Home
Bresciani, M.J.
Ask Yourself These Questions
How would you describe what outcomes-based assessment is to your faculty, staff, and students?
How would you describe its importance to what is happening in the classroom, your institution, and the student’s next steps?
The Assessment Cycle (Bresciani, 2006)
The key questions…• What are we trying to do and why? or• What is my program supposed to accomplish? or• What do I want students to be able to do and/or
know as a result of my course/workshop/orientation/program?
• How well are we doing it?• How do we know?• How do we use the information to improve or
celebrate successes?• Do the improvements we make contribute
to our intended end results?
The IterativeSystematicAssessment CycleBresciani, 2006
Mission/Purposes
Goals
Outcomes
Implement Methods to
Deliver Outcomes and
Methods to Gather Data
Gather Data
Interpret Evidence
Make decisions to improve programs; enhance student learning and development;
inform institutional decision-making, planning,
budgeting, policy, public accountability
Outcomes-Based Assessment Discoveries (Bresciani, 2006)
Most people do capitalize on their innate intellectually curiosity to find out what works
Most people just don’t articulate their intended end results (e.g., outcomes) ahead of time
Most people don’t document the decisions made based on their results
Most people don’t follow up later to see if their decisions made the intended improvement
This Process Works for:(Bresciani, 2007)
Evaluating the effectiveness of the technology Via outcomes for the users
Evaluating the effectiveness of the technology training Via evaluating outcomes of student learning
Evaluating how well the technology contributes to the improvement of student learning Via crossing the outcomes
Students can Participate (Bresciani, 2006)
In every phase of the assessment process
By having them do so provides them opportunities to evidence their problem solving, critical thinking, and analytical reasoning, and communication skills
And get a better picture of what you want them to learn and why
and take more responsibility for their learning
The Purpose (Bresciani, 2006)
Outcomes-Based assessment does not exist for assessment’s sake
It is taking what most of us already do, and making it systematic
Its purpose is to reflect on the end result of doing - - are we accomplishing that which we say we are?
The Purpose, Cont. (Bresciani, 2006)
It is NOT personnel evaluation It can be both formative and summative Planning is incorporated into it All types of assessment have value
(e.g., needs, utilization, satisfaction, learning and development) but we have to pay attention to evaluating student learning
Drivers of Outcomes-Based Assessment (Ewell, 2003)
To improve the underperforming student Competency Movement in Business and Industry Bologna Declaration of 1999 Government Conversation, 1985
The Higher Education Re-authorization Act Testimonies in USA, 2002 and 2006
Response to NCLB Legislation
Drivers, Cont.National Commission on the
Future of Higher Education (2006)
Demand for Public Information about Performance
• Transparency of outcomes and results• Comparable measures of quality• Demonstration of value-added of the
entire educational experience
Drivers, Cont. Accountability requirements handed
back to statesPerformance indicatorsStandardized tests
Discipline Standards could be designed by disciplines
Increased focus on private education Still no focus on evaluating
transferability of learning
How does this context change your answers to these previously posed questions?
How would you describe what outcomes-based assessment is to your faculty, staff, and students?How would you describe its importance to what is happening in the classroom, your institution, and the student’s next steps?
How do you make this process, effective, efficient, and enduring?
Consider how you would apply this criteria to what you do in the classroom or the design of curriculum and/or technology.
Criteria for Good Practices for Outcomes-Based Assessment Program Review (OAPR) (Bresciani, 2006)
Criterion 1: Clear understanding of purpose, goals, & expectations for OAPR Why, who, what, when, and how?
Criterion 2: Collaboration within and across Outcomes, Evaluation methods, Criteria,
Interpretation of Results, Decisions and Recommendations, Values
Criterion 3: Use of results To improve programs; to inform performance
indicators; to address value-added; to inform transferability of learning
Criteria, Cont. (Bresciani, 2006)
Criterion 4: Awards and recognition Common sense awards and recognition,
position descriptions and hiring criteria, evaluation of job performance criteria
Criterion 5: Resources to support program review Technological Professional development Individual consultation
Criterion 6: Coordination of the process
Criteria, Cont. (Bresciani, 2006)
Criterion 7: FlexibilityDelicate balance between structure
and creativity Criterion 8: Addressing barriers Criterion 9: Evaluation of the
program review processShort range and long range goalsMeta-Assessment
Inverted Funnel of Application to Lessons Learned
How would you apply these good practice criterion to the lessons you learned at this conference or vice versa?
Technical Disclaimer
I have not used any of these products so feel free to help me out here…
Assistance with Input Assessment
SAM 2007 E-Portfolio Management System
Assistance with Formative Assessment
SAM 2007 E-Portfolio Management System Case Grader Office 2007 Office 2007 “In a Flash”
Assistance with Summative Assessment
E-Portfolio Management System SAM 2007 Case Grader Office 2007 – if you
used course-embedded program assessment
Strategies for Collaboration
E-Portfolio Management System
Strategies for Coordination of the Process
E-Portfolio Management System
Questions to Consider (Bresciani, 2007)
ProcessWhen implementing an
effective, efficient, and enduring outcomes-based assessment process in your organization, consider the questions on the handout in the back of the room
Questions to Consider, Cont. (Bresciani, 2007)
When considering the use of Technology to promote outcomes-based assessment…Does the use of this technology reinforce the
reasons we are engaged in outcomes-based assessment?
How does the use of this technology promote formative assessment, summative assessment, or the assessment of transferable skills/knowledge?
Questions to Consider, Cont. (Bresciani, 2007)
Is our professional development to train users of our technology coupled with the professional development for outcomes-based assessment or instructional design?
How does the use of this technology enable us to better use the results to improve student learning and development?
Questions to Consider, Cont. (Bresciani, 2007)
How does the use of this technology improve efficiencies? And who benefits from those improved efficiencies?
How does this technology constrain or promote creativity/flexibility within each discipline?
How does this technology aid the coordination of the process? Or does it contribute to isolated thinking and doing?
Questions to Consider, Cont. (Bresciani, 2007)
How can data that this technological tool gathers help us evaluate the extent that we are transforming our organization through the use of evidence-based decision-making?“Keep your eye on the prize”
Challenge Questions to Consider
How does this technology help illustrate the transferability of learning/skills?
How does this technology help us illustrate, evaluate, and improve the learning/skills as they relate to the requisite skills required by employers?
What do you need to refine in order to make this happen?
Questions?
References
Bresciani, M.J. (2006). Outcomes-Based Academic and Co-Curricular Program Review: A Compilation of Institutional Good Practices. Sterling, VA: Stylus Publishing