non traditional measures in assessment 090712 (1)
TRANSCRIPT
1
BEYOND STANDARDIZED TESTS:
Assessing Creativity, Common Sense, Wisdom, and More in Admissions
2
Contact Information
Robert J. SternbergProvost and Senior Vice President George Kaiser Family Foundation Chair in Ethical
LeadershipRegents Professor of Psychology and EducationOklahoma State University;Honorary Professor, Heidelberg UniversityPresident, Federation of Associations in the Behavioral and
Brain Sciences;[email protected]
3
4
Goals of the Presentation
To show that, in admissions, in addition to memory and analytical skills, it is important rigorously to assess creative, practical, and wisdom-based/ethical skills of students
To show how to assess these additional skills in a rigorous way
To show data regarding what happens when an assessment process rigorously assesses these additional skills
5
Organization of Talk
Introduction Five Projects with Non-Traditional Measures
The Rainbow Project The Kaleidoscope Project The Panorama Project Advanced Placement Project The Choate Rosemary Hall Project
Implementing Non-Traditional Measures in Your Institution
Conclusions
6
The Problem
Traditional standardized-test measures in assessment, and even school grades, give us good information about some valued skills of students, but practically no information about other valued skills
7
In Particular…
If we wish to develop students who will be the active citizens and future leaders of tomorrow, we need to measure a broader range of the skills important to future success—not just the memory and analytical skills measured by standardized tests (such as IQ tests, the SSAT, SAT, and the ACT), but also creative, practical, and wisdom-based skills
8
What Skills do Standardized Tests Measure Well?
Memory skills Who? What? Where? When? Why? How?
9
What Else Do Standardized Tests Measure Well?
Analytical skills Analyze Critique Judge Compare and contrast Evaluate
10
What Skills Don’t Standardized Tests Measure Well?
Creative skills Invention Exploration Discovery Imagination Supposition
11
What Skills Don’t Standardized Tests Measure Well?
Practical Skills Application Utilization Implementation Putting ideas into practice Persuasion
12
What Skills Don’t Standardized Tests Measure Well?
Wisdom-Based Skills Ethical thinking Seeing multiple points of view Understanding how what is true
can change over time Understanding long-term as well as
short-term implications Thinking for the common good
13
Why These Particular Skills?
Motivation for the “Theory of Successful Intelligence” Alice Barbara Celia Diane
Zachary
14
Assessments on which Talk is Based
Rainbow (admissions) National project involving roughly
1000 diverse seniors in high school and college freshmen
Kaleidoscope (admissions) Project at Tufts University involving
over 30,000 applicants to the freshman class over a period of 5 years
15
Assessments on which Talk is Based Panorama (admissions)
New project at Oklahoma State University—no data yet
Advanced Placement (achievement) Project at diverse high schools in
the United States measuring (so far) achievement among several hundred students at high levels in psychology, statistics, and physics
16
Assessments on which Talk is Based Choate Rosemary Hall Assessment
An assessment to help select students who most can profit from, and in turn profit, the environment of the preparatory school
17
Augmented Theory of Successful Intelligence
People are successfully intelligent to the extent that they make the most of their lives
They do so by figuring out their strengths and weaknesses, and (a) capitalizing on the strengths while (b) compensating for or correcting weaknesses
They do so through a mix of analytical, creative, practical, and wisdom-based/ethical skills applied to a knowledge base
18
In particular…
Individuals are successfully intelligent to the extent they display Creative skills to generate novel ideas Analytical skills to ascertain whether the
ideas are good ones Practical skills in order to implement their
ideas and persuade others of their value Wisdom-based/ethical skills in order to
ensure the ideas help to achieve a common good based upon positive ethical principles
19
Source for Details
Sternberg, R. J. (2010). College Admissions for the 21st Century. Cambridge, MA: Harvard University Press.
20
The Rainbow Project
This project was conducted at Yale University in the early 2000s on roughly 1000 students with freshmen from 13 colleges and seniors from 2 high schools. The institutions were geographically very dispersed and were of greatly varying levels of selectivity, from community colleges to highly prestigious ones. There was a high level of ethnic diversity. Assessments were proctored in the students’ own institutions but scored at Yale.
21
Analytical Assessments
Learn meanings of words from context (The blen arose early in the
morning on the horizon.) Complete a number series
(3, 7, 13, 21, ….) Complete a figural matrix
22
Practical Assessment Sample ItemCollege Life Tacit-Knowledge Inventory
You are enrolled in a large introductory lecture course. Requirements consist of three term-time exams and a final. Please indicate how characteristic it is of your behavior to spend time doing the following, if your goal is getting an A in the course.
23
Practical Assessment Sample ItemCollege Life Tacit-Knowledge Inventory
Rate on a 1 (low) to 9 (high) scale:
___Attending class regularly.___Attending optional weekly review
sessions, if there are any, with the T.A.___Reading assigned text chapters
thoroughly.___Taking comprehensive class notes.___Speaking with the Professor after class
and during office hours.…
24
Practical Assessment Sample ItemGeneral Workplace/Common Sense Tacit-Knowledge Inventory
You’ve been assigned to work on a project for a day with a fellow employee whom you really dislike. He is rude, lazy, and rarely does a proper job. What would be the best thing for you to do?
25
Practical Assessment Sample ItemGeneral Workplace/Common Sense Tacit-Knowledge Inventory
Rate on a 1 (low) to 9 (high) scale:
___Tell the worker that you think he is worthless.
___ Warn the worker that, if he is not “on his toes” today, you will complain to the supervisor.
___ Avoid all conversation and eye contact with the other worker.
___ Be polite to the other worker and try to maintain as business-like a manner as possible so that hopefully he will follow your example for the day.
…
26
Practical Everyday Situational Judgment - Movies
Examinees see seven digitized movies depicting various real-life situations that college students confront or may confront: The Party: Entering a party where one does not know anyone
A Fair Portion: Discussing shares of rental payments for a flat
Professor’s Dilemma: Asking for a letter of recommendation from a professor who does not know you very well
27
Creative Written Stories
SHORT STORY TASK: TITLES
“A Fifth Chance”“2983”“Beyond the Edge”“The Octopus’s Sneakers”“It’s Moving Backwards”“Not Enough Time”
28
Creative Oral Stories
SHORT STORY TASK
Students see several pictorial collages. They have to tell a story about one of them.
29
30
Creative Cartoon Titles
Examinees see five cartoons and need to provide titles for three of the five.
31
________________________________________________________________________
________________________________________________________________________
Exploratory Factor Analysis: Rainbow Tasks
Promax rotation: 62.8% variance explained
Factor 1 Factor 2 Factor 3Oral Stories 0.57 -0.06 -0.06Written Stories 0.79 0.01 -0.02Cartoons 0.20 0.28 -0.08STAT-creative 0.00 0.73 0.09STAT-analytic -0.06 0.80 -0.04STAT-practical 0.03 0.81 -0.02Movies 0.12 0.05 0.52College Life -0.13 0.01 1.00Common Sense 0.12 -0.01 0.92
33
Predicting College GPA: SAT + Analytical
Step 1: SAT-Verbal, SAT-Math
Step 2: Analytic (STAT)
9.8 9.9
0
5
10
15
20
Step 1 Step 2
R s
qu
are
d (
%)
34
Predicting College GPA: SAT +Practical
Step 1: SAT-Verbal, SAT-Math
Step 2: Practical (STAT + performance)
9.810.7
0
5
10
15
20
Step 1 Step 2
R s
qu
are
d (
%)
35
Predicting College GPA: SAT +Creative
Step 1: SAT-Verbal, SAT-Math
Step 2: Creative (STAT + performance) 9.8
12.8
0
5
10
15
20
Step 1 Step 2
R s
qu
are
d (
%)
36
Predicting GPA: SAT + Analytic, Creative, Practical
Step 1: SAT-Verbal, SAT-Math
Step 2: All Rainbow Project Items(STAT Analytic, Practical, Creative, Practical performance,Creative performance)
9.8
19.9
0
5
10
15
20
Step 1 Step 2
R s
qu
are
d (
%)
37
Predicting GPA: All measures (practical before creative)*
Step 1: SAT-M SAT-V HSGPA
Step 2: + Analytic
Step 3: + Practical
Step 4: + Creative
*Controlling for school quality independent variable
15.6 15.2 15.9
24.8
0
5
10
15
20
25
30
Step1
Step2
Step3
Step4
R s
qu
are
d
38
In sum
In the Rainbow sample, Adding Rainbow measures over SAT
roughly doubles prediction of college success
Adding Rainbow measures over SAT + High School GPA increases prediction by roughly half
39
Amount of Each Measure That Is Predicted by Racial / Ethnic Differences (ω²)
0.09
0.04
0.00
0.02
0.01
0.02
0.03
0.01
0.02
0.01
0.03
0.00
0.05
0.10
Ver
bal
Mat
h
ST
AT
ST
AT
Writ
ten
Sto
ries
Ora
lS
torie
s
Car
toon
s
ST
AT
Mov
ies
Col
lege
Life
Com
mon
Sen
se
SAT Analytic Creative Practical
Pro
po
rtio
n e
xp
lain
ed
by
ra
ce
dif
fere
nc
es
40
In Sum
In the Rainbow sample: Rainbow measures reduce ethnic-
group differences relative to the SAT alone
The new measures reduce differences because different ethnic groups show different average patterns
Differences are not eliminated, however
41
The Kaleidoscope Project
Project at Tufts University over a period of five years involving over 30,000 applicants to the freshman class. Additional admissions exercises were optional. Questions were placed as an optional Tufts-specific supplement to the Common Application. The study was done in collaboration with Dean of Admissions Lee Coffin and his staff.
42
Essay Prompts: Year 1
The late scholar James O. Freedman referred to libraries as “essential harbors on the voyage toward understanding ourselves.” What work of fiction or non-fiction would you include in personal library? Why? (Analytical)
An American adage states that “curiosity killed the cat.” If that is correct, why do we celebrate people like Galileo, Lincoln, and Gandhi, individuals who imagined longstanding problems in new ways or who defied conventional thinking to achieve great results? (Analytical)
43
Essay Prompts: Year 1
History’s great events often turn on small moments. For example, what if Rosa Parks had given up her seat on that bus? What if Pope John Paul I had not died after a month in office in 1978? What if Gore had beaten Bush in Florida and won the 2000 U.S. Presidential election? Using your knowledge of American or world history, choose a defining moment and imagine an alternate historical scenario if that key event had played out differently. (Creative)
44
Essay Prompts: Year 1
Create a short story using one of the following topics: The end of MTV Confessions of a Middle School Bully The Professor Disappeared The Mysterious Lab (Creative)
Using an 8.5x11 inch sheet of paper, illustrate an ad for a movie, design a house, make an object better, or illustrate an ad for an object of your choice. (Creative)
Do a creative YouTube Video and send us the web address (Creative)
45
Essay Prompts: Year 1
Describe a moment in which you took a risk and achieved an unexpected goal. How did you persuade others to follow your lead? What lessons do you draw from this experience? You may reflect on examples from your academic, extracurricular or athletic experiences. (Practical)
A high school curriculum does not always afford much intellectual freedom. Describe one of your unsatisfied intellectual passions. How might you apply this interest to serve the common good and make a difference in society? (Wisdom)
Creative Essay: “What if…”
If the Trojans had heeded Laocoon’s advice and thrown Odysseus’ wooden horse into the sea, they would have defeated the Greeks at Troy. Aeneas would then never have had reason to flee the city, and he would never have ventured to Italy to found Rome. Without Rome, neither the Roman Republic nor a Roman Empire would have existed. Concrete, the arch, plumbing, and the sauna might never have been invented. The modern implications of Rome never having existed are indeed drastic. Lacking even concrete floors, people would resort to sleeping in the mud, and, without plumbing or saunas, they would be perpetually filthy and, generally, quite chilly. France could not have built the base of the Eiffel Tower without arches, so tourists would be unable to purchase miniature collectible Towers in Parisian convenience stores.
Good but Uncreative Essay:“What if…”
What if the ratification of the nineteenth amendment did not pass and women were never given the right to vote? What would life for women, like me, be like in the United States? For one thing, I probably would not be writing this essay. If women were not given their right to vote, I probably would stop going to school after this year and it would be unlikely that I would receive a college education. Without suffrage, my career options would be limited, if a career were a possibility at all. My accepted practices would be limited to staying home and taking care of the family. Rather than being equals, women would be subservient to men. I might not drive, I might not dress in the way in which I choose to, and I might not be able to live my life the way that I can in the twenty-first century.
48
Pilot Study Data
Number of applications rose Bottom third of old application pool
greatly diminished; many more top applicants
Average SATs rose slightly African-American applications up 25%,
acceptances up 30% Hispanic-American applications and
acceptances up 15%
49
Pilot Study Data
There were no significant ethnic-group differences on Kaleidoscope
Kaleidoscope correlated moderately with rated leadership/extracurricular activities (.44)
Being rated for Kaleidoscope was associated with higher freshman GPA, holding constant high school GPA and SATs
50
Pilot Study Data
Kaleidoscope predicted extracurricular, leadership, active-citizenship participation
Greater customer satisfaction Message to students, parents,
teachers, and counselors that Tufts is looking for more than just the high-SAT, high-GPA student
51
The Panorama Project
This is a new project at Oklahoma State University. All applicants are being given the opportunity optionally to answer questions that assess creative, analytical, practical, and wisdom-based skills. Each applicant who participates answers three questions, similar to those in Kaleidoscope. Scoring will be similar to that for Kaleidoscope. Ratings will be used for those evaluated by holistic admissions and for scholarship consideration.
52
Panorama Sample Analytical Question
An army colonel once stated, "Leadership is about comforting the disturbed and disturbing the comfortable." What did he mean? What do you think true leadership is? When have you been a leader, and how did you exercise leadership?
53
Panorama Sample Practical Question
If you were able to open a local charity or business of your choice, what type of organization would it be and whom would it benefit? Describe your start-up process.
54
Panorama Sample Creative Question
Write a story or poem that includes one of the following sets of words:
Purple, panic, panda, petunia, and popcorn
A stick, a light bulb, the Great Wall of China, and water
A bicycle, a clock, the Wild West, and duct tape
55
Panorama Sample Wisdom Question
After submitting a class project, you realize one of your partners committed plagiarism. Your teacher previously announced that if he or she learned that cheating had occurred, all members of the work group would receive an F grade. How would you handle the situation and what would be your ideal outcome?
56
The Advanced Placement Project This is a project to infuse measurement
of creative, analytical, and practical skills into tests of achievement given in high schools. Study has been run so far with Psychology, Statistics, Physics.
57
An Example from Psychology: Creative
Imagine that you had to produce a TV sitcom to illustrate Freud’s personality theory.
Which of the following characters would best represent the superego? (a) A Fire Fighter (b) An action-movie hero (c) A nurse (d) An artist *(e) A Supreme Court judge
58
An Example from Statistics: Practical
Mr. Smith, a politician, argues that the average family income in his state is $4,500 a month and that therefore complaints about massive poverty are ill-founded. What is wrong with Mr. Smiths’ claim?
(a) $4,500 a month is not very high. (b) Mr. Smith’s statement is obviously not true. *(c) Mr. Smith does not take into account the
standard deviation of incomes. (d) Mr. Smith is not an expert on poverty and
hence has no credentials to make any claim about it.
59
Choate Rosemary Hall Project
This battery included a variety of measures to enhance prediction of academic success in the environment of Choate Rosemary Hall
The samples that follow are from the School Life Questionnaire
School-Life Questionnaire1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
In day schools you rarely see your teachers outside of class. Some of them might be engaged in sports or other extra-curricular activities, but mostly you only see them in school-related circumstances. At boarding school the situation is quite different, because many teachers live on campus and you get to see them outside the classroom a lot.
School-Life Questionnaire1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
Given this situation, rate the quality of the following behavior choices:
____ (a) Always greet teachers and smile, but avoid seeing them outside of class.
____ (b) Take advantage of this situation to talk to teachers about your school-related problems.
____ (c) Wait and see if teachers approach you, and if so, what kinds of things they talk to you about.
School-Life Questionnaire1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
____ (d) Talk to your teachers but avoid discussing your problems as this might give them a negative impression of you.
____ (e) Try to be sensitive and make a distinction between situations when teachers are available and unavailable to you.
____ (f) Always try to be noticed—the more teachers talk to you, the better your grades will be.
____ (g) Always ask whether it is a good time or not to discuss your problems with teachers.
1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
You are taking a math class that gives you a lot of trouble. On the first two tests you did poorly, and for tomorrow you have a homework problem that you are not quite sure how to solve. .
1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
Given this situation, rate the quality of the following behavior choices:
____ (a) Try to find a solution, or at least some explanation of how to arrive at a solution, in the book .
1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
____ (b) Give it a try; if you can’t solve the problem you’ll just have to tell the teacher you didn’t understand it.
____ (c) Ask some of your classmates if they’ve found the solution, telling them you want to compare it with yours’ (although you don’t actually have one).
1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
____ (d) Try hard, and if it doesn’t work, give up on it; you can always pretend you had forgotten you had homework.
____ (e) Suggest to some of your classmates that you study together.
1 2 3 4 5 6 7
Not a very good Average A very good
choice choice
____ (f) Go to see the one student in the class who you know is really smart and ask him for help.
____ (g) Try hard, take a lot of notes, then come to the teacher before class and tell her/him that you tried and failed.
Predicting Choate GPA: Hierarchical Regression
31.3
50.9
0
10
20
30
40
50
60
Choate GPA 12/ 99 (adj Rsq)
Pre-Choate GPA, SSAT (all) Yale indicators (all)
Predicting Choate GPA: Hierarchical/Stepwise Regression
24.730.9
52.7
0
10
20
30
40
50
60
Choate GPA 12/ 99 (adj Rsq)
Pre-Choate GPA SSAT(Q) Yale indicators (3)
Predicting Choate GPA: Hierarchical Regression
42.5
52.858
0
10
20
30
40
50
60
Choate GPA 12/ 99 (adj Rsq)
Locus of Control Choate Tacit Knowledge Self-confidence
71
Main Results
Adding analytical, creative, and practical questions: Increases content validity Increases face validity Decreases ethnic-group differences
72
Putting Theory into Practice: Creating Rubrics
Test users are busy people. So it is preferable to use “holistic” ratings rather than trying to be excessively micro-analytic.
You need to decide what you value in responses
Some rubrics we have used are the following:
73
Analytical Rating
The extent to which the response is Analytical Organized Logical Balanced
74
Creative Rating
The extent to which the response is Novel Compelling Task-appropriate (either yes or no)
75
Practical Rating
The extent to which the response is compelling
The extent to which the response is practical with respect to Human resources Material resources Time and place
76
Wisdom Rating
The extent to which the response reflects A common good A balance of one’s own, others’,
and larger interests Thinking over the long-term as well
as the short-term Positive ethical values
77
A Common Error to Avoid
In scoring, you do not want merely to measure general academic skills; therefore, do not take into account grammar, punctuation, capitalization, and spelling except for the analytical rating. Otherwise, you create a general factor (halo effect) that has little to do with what you want to rate and that will increase correlation of the non-traditional assessments with conventional standardized tests
78
Conclusion
Creative, practical, and wisdom-based skills, like memory and analytical skills, can be rigorously assessed
Measurements such as those described here provide a means for rigorous assessment
Assessing such additional skills provides incremental prediction of academic and personal success and reduces ethnic-group differences
79
Conclusion
In order to develop the active citizens and leaders of tomorrow, who will make a positive, meaningful, and enduring difference to the world, one reliably and validly can assess in school creative, analytical, practical, and wisdom-based skills.
80