the proof is in the portfolio: an architecture of the good, the bad and the mediocre
DESCRIPTION
Terrel L RhodesTRANSCRIPT
The Proof is in the Portfolio: An Architecture of the Good, the
Bad, and the Mediocre
Terrel L RhodesAssociation of American Colleges and Universities
AustraliaOctober 18, 2011
What is AAC&U? Founded in 1915, AAC&U is
dedicated to making the aims of liberal learning a vigorous and constant influence on institutional planning and educational practice in college. It is a meeting ground for all sectors of higher education and brings together faculty, academic and student affairs leaders and presidents across sectors, divisions, and disciplines to explore the aims of education, the future of the academy, and strategies for institutional change and higher student achievement.
2
The Twenty-First Century Academy
Context:Changing Designs for College
Learning
A Curriculum in Transition:Rethinking educational purposes and practices to better prepare all students for
• complexity and contingency
• global interdependence
• innovation in the workplace
• diverse democracy
3
4
US Economy Defined by Greater Workplace Challenges and Dynamism
Every year, more than 1/3 of the entire US labor force changes jobs.
Today's Students Will Have 10-14 Jobs by the Time They Are 38.
50% of Workers Have Been With Their Company Less Than 5 Years.
Every year, more than 30 million Americans are working
in jobs that did not exist in the previous quarter.
5
DOL-BLS
What Employers Say
“[Employers] generally are...frustrated with their inability to find ‘360 degree people’ who have both the
specific job/technical skills and the broader skills (communication and problem-solving skills, work ethic, and ability to work with others) necessary to promise
greater success for both the individual and the employer.”
From Peter D. Hart Research Associates, Report of Findings Based on Focus Groups
Among Business Executives (AAC&U, 2006)
6
The Growing Demand for Higher Order SkillsSource: Council on Competitiveness, Competitiveness Index
7
HARTRESEARCH
P e t e r D
A S S O T E SC I A
Raising The BarEmployers’ Views On College Learning In The Wake Of The Economic Downturn
Key findings from survey among 302 employersConducted October 27 – November 17, 2009for
Employers’ Expectations of Employees Have Increased
88%
88%
90%
91%
% who agree with each statement
Our company is asking employees to take on more responsibilities and to use a broader set of skills than in the past
Employees are expected to work harder to coordinate with other departments than in the past
The challenges employees face within our company are more complex today than they were in the past
To succeed in our company, employees need higher levels of learning and knowledge today than they did in the past
Broad Skills/Knowledge AND Specific Skills/ Knowledge Are Needed for
Career Success
20%
20%
59%
Which is more important for recent college graduates who want to pursue advancement and long-term career success at your company?
BOTH in-depth AND broad range of skills and knowledge
Broad range of skills and knowledge that apply to a range of fields or positions
In-depth knowledge and skills that apply to a specific field or position
70%
70%
71%
75%
75%
79%
81%
89%
Employers’ Top Priorities For Student Learning Outcomes In
College% saying two- and four-year colleges should place MORE emphasis on helping students develop these skills, qualities, capabilities, knowledge
Effective oral/written communication
Critical thinking/ analytical reasoning
Knowledge/skills applied to real world settings
Analyze/solve complex problems
Connect choices and actions to ethical
decisionsTeamwork skills/ ability
to collaborateAbility to innovate and
be creativeConcepts/developments in
science/technology
Higher Level Liberal Education Skills and
Abilities =
Higher Wages
12
Data from Georgetown University Center for Education and the Workforce
Center on Education and the Workforce
The Salary Premium for Liberal Education Outcomes
From a federal database analyzing qualifications for 1,100 different jobs, there is consistent evidence that the highest
salaries apply to positions that call for intensive use of liberal education capabilities, including (random order):
Inductive and Deductive ReasoningJudgment and Decision MakingProblem SolvingSocial/Interpersonal SkillsMathematicsOriginalityWriting
13
Source: Georgetown University Center on Education and the Workforce
Mean Earnings of Jobs that Emphasize Writing
14
0
35,000
70,000
q1(low) q2 q3 q4 q5(high)
earn
ings
quintiles
Mean earnings of writing quintiles
Earnings
Source: Georgetown University Center for Education and the Workforce
Mean Earnings of Jobs that Emphasize Speaking
15
0
30,000
60,000
q1(low) q2 q3 q4 q5(high)
earn
ings
quintiles
Mean earnings of speaking quintiles
Earnin…
Source: Georgetown University Center for Education and the Workforce
Mean Earnings of Jobs that Emphasize Originality
16
0
30,000
60,000
q1(low) q2 q3 q4 q5(high)
earn
ings
quintiles
Mean earnings of originality quintiles
Earnings
Source: Georgetown University Center for Education and the Workforce
Mean Earnings of Jobs that Emphasize Problem Solving
17
0
35,000
70,000
q1(low) q2 q3 q4 q5(high)
earn
ings
quintiles
Earnings of complex problem solving quintiles
Earnings
Source: Georgetown University Center for Education and the Workforce
Mean Earnings of Jobs that Emphasize Physical Ability
18
0
30,000
60,000
q1(low) q2 q3 q4 q5(high)
earn
ings
quintiles
Mean earnings of physical ability quintiles
Earnings
Source: Georgetown University Center for Education and the Workforce
Where We Are Now
In short, we know “what works” – to foster both learning gains and greater
completion
But many students aren’t doing “what works”
19
E-Portfolios Can Capture:Essential Learning OutcomesA Guiding Vision and National Benchmarks for College Learning and Liberal Education in the 21st Century
High Impact PracticesHelping Students Achieve the Essential Learning Outcomes
Authentic AssessmentsProbing Whether Students Can APPLY Their Learning – to Complex Problems and Real-World Challenges
20
Narrow Learning is Not EnoughThe LEAP Essential Learning Outcomes
Knowledge of Human Cultures and the Physical and Natural World
Focused on engagement with big questions, enduring and contemporary
Intellectual and Practical Skills
Practiced extensively across the curriculum, in the context of progressively more challenging problems, projects, and standards for performance
Personal and Social Responsibility
Anchored through active involvement with diverse communities and real-world challenges
Integrative Learning
Demonstrated through the application of knowledge, skills, and responsibilities to new settings and complex problems
21
Aims/Outcomes AddressedAcross the Curriculum
First to Final Year
Integrating Liberal and Professional Learning
Co-Curriculum as Well
Assessments that Deepen Learning
Sustained Focus on Underserved Students
Expecting students to complete a significant project before graduation that demonstrates their depth of knowledge in their major AND their acquisition of analytical, problem-solving, and communication skills (62% help a lot)
Expecting students to complete an internship or community-based field project to connect classroom learning with real-world experiences (66%)
Ensuring that students develop the skills to research questions in their field and develop evidence-based analyses (57%)
Expecting students to work through ethical issues and debates to form their own judgments about the issues at stake (48%)
Source: Raising the Bar (AAC&U, 2010)
Employers Assess the Potential Value of Emerging Educational
Practices% saying each would help a lot/fair amount to prepare college students for success
84%
81%
81%
73%
High-Impact Practices (HIPs)
Correlate Highly with NSSE Gains on Student Learning Outcomes
Provide “Compensatory Benefit” for Students With Lower Test Scores and/or
High Drop-Out RatesKuh, High Impact Practices: What They Are, Who Has Access to Them, and Why They Matter (AAC&U 2008)
High Impact Practices First-Year Seminars and Experiences
Common Intellectual Experiences
Learning Communities
Writing-Intensive Courses
Collaborative Assignments and Projects
Undergraduate Research
Diversity/Global Learning
Service Learning, Community-Based Learning
Internships
Capstone Courses and Projects
The Eleventh High Impact Practice
Electronic portfolios
26
2.00
2.25
2.50
2.75
3.00
3.25
3.50
3.75
4.00
-2 -1 0 1 2Educationally Purposeful Activities
(standardized)
Fir
st-
year
GP
A
ACT 28 ACT 24 ACT 20
Impact of Educationally Purposeful Practices on First Academic Year GPA by Pre-College
Achievement Level
0.50
0.55
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
-2 -1 0 1 2
Educationally Purposeful Activities (standardized)
Pro
ba
bily
t o
f R
etu
rnin
g
African American
White/Caucasian
Impact of Educationally Purposeful Practices on the Probability of Returning for the Second Year of College by
Race
*Findings developed by LEAP National Leadership Council Member George Kuh and used with permission
Huber, CSU - NorthridgeFigure 4. Impact of Participation in High Impact Practices on Percentage of
Senior NSSE Respondents Graduating on Time by Racial & Ethnic Background
38%
54%
48%
63%65%
68%
73%
69%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
Latina/o Respondents Other Respondents
Pe
rata
ge
Gra
du
ati
ng
"o
n T
ime
" (i
.e.,
in 2
00
6-0
7)
None 1 HIP 2 HIPs 3 or more HIPs
[V=.109 (.094)][V=.255 (.007)]
Multiple HIPs distributed through general education and major programs would, we are convinced, “require only small curricular changes.” Such “modest change” can yield a significant increase in student success and persistence.
29
Participation Levels in High Impact Practices
First Year (NSSE Data)
Learning Communities 17%
Service Learning 36%
Participation Levels in High Impact Practices
Seniors (NSSE Data)Research With Faculty 19%
Internship 53%
Service Learning 46% Study Abroad 19%
Senior Culminating Work 32%
The Challenge Ahead –
Making High Impact Practices Central
Rather than Optional
The Key Elements for a Compelling Quality Framework Already Are in
HandConsensus Aims and Outcomes
Practices that Foster Achievement AND Completion
Evidence on “What Works” for Underserved Students
Assessments That Raise – and Reveal – the Level of Learning
The Power of Rubrics and E-Portfolios as Tools for Both Assessment and High-Impact Learning
• Rubrics to help guide students and faculty• Places individual faculty judgment within national shared
experience; national benchmarks• E-portfolios to gather students’ best work, encourage self-
assessment, and allow for mining of samples for assessment purposes
• E-portfolios are portable, allow for cumulative learning and assessment, can complement other high-impact practices
• Can build up from course level to institutional reporting needs AND down from general to specific program/course context
34
VALUE Projecthttp://www.aacu.org/programs/VALUE
An obvious place to begin:
Help Students Understand What They
Are Expected to Accomplish
Rubrics Basics
Criteria
VALUE Rubrics & AssessmentVALUE Rubrics & Assessment
Rubrics Basics
Levels
VALUE Rubrics & AssessmentVALUE Rubrics & Assessment
Rubrics Basics
Performance Descriptors
VALUE Rubrics & AssessmentVALUE Rubrics & Assessment
Validity and Usability• Over 2000 distinct institutions have downloaded one
or more of the VALUE rubrics for use• Over 11,000 distinct individuals have downloaded one
or more of the VALUE rubrics for use• 3 major consortia are using VALUE rubrics for cross
institutional collaboration – Connect2Learning – LaGuardia College/AAEEBL (FIPSE) – 23 campuses; Integrative Portfolio Process – Michigan (FIPSE) – 6 campuses; RAILS – Syracuse (Institute for Museum and Library Studies ACRL) – 10 campuses
• Major state university systems are using VALUE rubrics• Being used worldwide, e.g. Japan, Hong Kong,
Australia, United Arab Emirates
Reliability Study• 40 Faculty• 4 Traditional Disciplinary Divisions –
Humanities, Social Sciences, STEM, Professions
• Three VALUE rubrics – Critical Thinking, Civic Engagement, Integrative Learning
• Common set of student portfolio work• Agreement = .66 without norming; .8
normed• Another set of 5 campuses, using same set
of rubrics with 500 samples of student work – still analyzing
Building the Evidentiary Base
• University of Kansas – Representing Results
Per
cen
t o
f R
atin
gs
Critical Thinking: Issues, Analysis, and Conclusions
Inter-rater reliability = >.8
Building the Evidentiary Base
• University of Kansas – Representing Results
Per
cen
t o
f R
atin
gs
Critical Thinking: Evaluation of Sources and Evidence
Building the Evidentiary Base
Per
cen
t o
f R
atin
gs
“VALUE added” for 4 years - writing
Building the Evidentiary Base
• University of Kansas – • “analysis of the data from the AACU VALUE rubrics
affirmed that a team approach to course design can yield larger improvement in some forms of student writing and thinking”
• “We also saw that the rubrics work best when there is close alignment between the nature of the assignment and the dimensions of intellectual skill described in the rubric”
• “Finally, at a practical level we are very encouraged that this process is manageable and sustainable”
Building the Evidentiary Base
• University of Kansas –
• “Interestingly, the patterns that were visible in the VALUE rubric scores were not mirrored in the CLA scores; students in the team-designed and traditional courses performed no differently on the CLA. Students’ performance on the CLA, moreover, was generally unrelated to the VALUE rubric ratings of their coursework, as well as the instructors’ grading of the same coursework. In contrast, the latter two measures were highly correlated, suggesting that the VALUE rubrics capture qualities of critical thinking and writing that fit well with what faculty members VALUE in their students’ work.”
Roanoke College
Writing
Reliability = >.8
Using the Results“…we excluded the scores for those instructors and ran
frequencies and descriptive statistics on the categories again. We found the means for the rubric categories of Focus and Thesis and Organization remained close to 2.5 with the scores of 2 and 3 occurring most often (approx. 2.4) while most scores for the categories of Evidence and Reasoning were 2s. The category of Style and Mechanics was in the middle with a mean of 2.3. Our adjusted results support what most faculty believe about the writing of our first-year students, they can learn to develop a thesis and organize a paper more effectively than they can provide convincing evidence and strong reasoning to support the thesis.”
Building the Evidentiary Base
University of North Carolina - Wilmington
Dimension % of Work Products Scored 2 or higher
% of Work Products Scored 3 or Higher
IL1 Determine Information Needed 87.2% 46.2%
IL2 Access Needed Information 89.6% 46.8%
IL3 Evaluate Information and Sources 88.5% 39.7%
IL4 Use Information Effectively 85.9% 43.6%
IL5 Access and Use Information Ethically 93.6% 59.0%
Table 1. Information Literacy Results
Table 1 Information Literacy ResultsTable 1 Information Literacy ResultsTable 1 Information Literacy Results
Inter-rater reliability = >.8
Building the Evidentiary Base
University of North Carolina - Wilmington
Dimension % of Work Products Scored 2 or higher
% of Work Products Scored 3 or Higher
CT1 Explanation of Issues 68.3% 35.5%
CT2 Evidence Year1 65.0% 28.2%
CT2 Evidence Year 2*Interpreting and Analysis
Questioning viewpoint72.8%40.9%
38.6%13.6%
CT3 Influence of Context and Assumptions
48.8% 21.2%
CT4 Student’s Position 54.5% 24.0%
CT5 Conclusions and Related Outcomes 47.7% 17.0%
Table 2. Critical Thinking Results
Building the Evidentiary Base
Long Island University – Brooklyn Campus
The results of this initial assessment are early benchmarks important for all institutions that expect integrative learning to emerge among students’ upper level accomplishments. These results help us see how students incorporate unmediated experiential learning into deeper transference and comprehension. In fact, students welcome the opportunity to integrate life experience, course work, and texts early in their undergraduate experience, if invited to do so.
• VSA institutions can select from CAAP, CLA, or ETS Proficiency Profile
• Important considerations for selection– Acceptance by students, faculty,
administrators or other policy makers– Trade-offs in cost, ease of administration– Utility of the test for other purposes -
supporting campus activities and services or providing guidance on improving learning
55
Implications: VSA and CLA
One Approach to Capturing and Tracking
Learning
E-Portfolios
Freshman Inquiry (FRINQ)UNST 100-level class
A year-long, theme-based course with a mentor section that corresponds with the class.
Transfer TransitionUNST 200- and 300- level classes
One-term course designed specifically for transfer students, with a mentor section that corresponds with the class.
Sophomore Inquiry (SINQ)UNST 200-level classes
One-term courses with a mentor section that corresponds with the class.
Sophomore Inquiry Sophomore Inquiry Sophomore Inquiry
Upper Division ClusterClasses designated with a "U" offered by academic departments. Three courses in one cluster linked to one of the
Sophomore Inquiry courses.
Cluster Course One Cluster Course Two Cluster Course Three
Senior CapstoneUNST 400-level class
A 6-credit, community-based learning class.
Portland State University
Queensborough College
• Remedial/Developmental Communications Course
• First Year Writing Course• Disciplinary Course• Web-based portfolio/wiki
58
Literature, Language & Communicati
on
We have had our why's, how's, and what's upside-down,
focusing too much on what should be learned, than on how, and
often forgetting the why altogether.
In a world of nearly infinite information, we must firstaddress why, facilitate how, and let the what
generatenaturally from there.
Michael Wesch, “From Knowledgeable to Knowledge-able,” Academic Commons, January 2009 (academiccommons.org)
64
Curricular and Pedagogical Innovations – Led by Faculty – Are Creating a 21st Century Vision and
Practice for Liberal Education
Directly Connected to the Needs and Experiences of Today’s Diverse
Students, our Diverse Democracy, and Interdependent Global Community
65