basic skills, outcomes & fostering a culture of evidence and inquiry
DESCRIPTION
Basic Skills, Outcomes & Fostering a Culture of Evidence and Inquiry. Rob Johnstone, Skyline College; Janet Fulks, Bakersfield College; Bob Pacheco, Barstow College; Ian Walton, Mission College. Sponsored by the William & Flora Hewlett Foundation Research conducted by the RP Group. Agenda. - PowerPoint PPT PresentationTRANSCRIPT
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Basic Skills, Outcomes & Fostering a Culture of Evidence and InquiryRob Johnstone, Skyline College; Janet Fulks, Bakersfield College; Bob Pacheco, Barstow College; Ian Walton, Mission College
Sponsored by the William & Flora Hewlett FoundationResearch conducted by the RP Group
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Agenda
Opening Dialog: Why are we struggling?
Challenges identified through the Basic Skills Outcomes Capacity (BSOC) Study
Strategies for responding to these challenges
Closing Dialog: What would help colleges strengthen cultures of inquiry?
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
BSOC FindingThere is a perception gap between
administrators and faculty regarding the
availability and use of assessment data.
College administrators believe that information
is more widely available and integrated into
implementation planning than those who work
more closely with students.
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
The Information Gap
Administration
Deans / Managers
Faculty/Staff
No data needs being met
All data needs being met
Information Gap
Information Gap
Gap
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Opening Dialog
Findings from the BSOC survey suggest we are struggling to get the right information to the right people at the right time.
What are your thoughts?
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
BSOC Finding
Colleges are data rich and information poor.
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
The Data We Collect
SLO / Assessment
Productivity / Efficiency
Enrollment Mgmt
Data Not Being Used
Data Frequently Being Used
Unmet
Unmet Data Needs
Unmet Data Needs
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
The Action Gap“Our college is quite successful at
gathering data and putting reports together based on that data. Where we struggle is with executing action plans as a result of this data. I don’t really think we do a good job of ‘doing’ anything with the data, or implementing programmatic changes or initiatives based on the data.”
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
BSOC Finding
There is a pattern of gaps in the use of data and evidence across community colleges – gaps that might be inhibiting institutional learning.
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Institutional Data Fills Two Important Gaps
Elevated Level of Organizational
Awareness
Improved Processes & Performance
Knowledge Gap Performance Gap
Strategic Planning Function
Institutional Effectiveness & Student Success Function
(Poor Planning) (Good Planning) (Good Performance)
Classroom and Service Area Assessment
Standard Level of Organizational
Awareness
Institutional Outcomes and Benchmarks
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
BSOC Finding
Colleges may be focusing on the wrong data.
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Typical Twenty Year Trend for Common Institutional Outcome Measures
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Retention Rate
Success Rate
Consider the multitude of changes over this period:
- faculty/staff turnover- program successes/failures- changing student demographics - budget contractions/expansion- evolving state & accreditation mandates- leadership turnover
and yet these performance metrics remained amazingly stable.
What does that tell us about the usefulness of these metrics in setting institutional strategies?
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Assessment and the Organizational Learning Map
Desired Outcome(Pass transfer-level course)Program Level Assessment
• Assess at the end of the semester• Modifications tend to be structural • Change is slow & deliberate
Assessment within the Classroom • Assess both during and at the end of the semester• Modifications are both structural & procedural• Change is frequent & free form
Entry Point (Three levels
below transfer)
Did the learning community for Math 100 work?
Within Math 100, did a group assignment produce stronger learning than a lecture?
Learning Pathways under Three Assessment Frameworks
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
BSOC Finding
Institutional research supports institutional metrics to a higher degree than classroom assessment.
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Level of Involvement in the Use of Data
Chief Instructional Officer Classroom Faculty
High Level of Involvement Low Level or No Involvement
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
BSOC Finding
Much of what is needed to support good decision making lies beyond the current grasp of our institutional databases.
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Students Progress in a Non-Linear Fashion
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
BSOC Finding
Over the last decade, most of the progress in gathering and utilizing data has been at the institutional level.
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Degree to Which Evidence is Used in Decision Making
High
Low
Us
e o
f D
ata
fo
r In
sti
tuti
on
al M
etr
ics
Use of Data in Classroom/Service Area Assessment
Low High
Strong Sense of Direction but Poor Understanding of
Program Performance
Much Context-Specific Knowledge but Poor Integration
Poor Planning & Weak
Understanding of Programs
Closing the LoopRobust Planning &
High Levels of Innovation
Outward Looking
Inward Looking
Forward LookingCluster of
California Community
Colleges
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
BSOC Recommendations
Research should be brought to the intervention level, at the classroom.
1
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Expanded Definition of Research Capacity
Production
Dissemination
InterpretationTranslation
Closing & Widening the
Loop
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
BSOC Recommendations
• Research offices should receive support that will enable them to address multiple domains of research capacity.
• Technical assistance should be provided to colleges that will help them to strengthen cultures of evidence and inquiry.
The Bridging Research, Information, and Culture (BRIC) Project
2
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
The BRIC Project Developing a Better Understanding of How to Close the Performance Gap
Standard Level of Organizational
Awareness
Elevated Level of Organizational
Awareness
Improved Processes & Performance
Knowledge Gap Performance Gap
Strategic Planning Function
Institutional Effectiveness & Student Success Function
(Poor Planning) (Good Planning) (Good Performance)
Institutional Outcomes and Benchmarks Classroom and Service Area Assessment
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Closing Dialog
The BRIC project will allow us to provide tailored support at colleges.
How could an external team of faculty, administrators, and researchers help deepen the culture of inquiry at your institution?
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP
Group
Contact & Information
RP Group Website
www.rpgroup.org
BSOC/BRIC Project ContactsRobert Johnstone, Project [email protected]
Priyadarshini Chaplot, Project [email protected]
Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group