assessing higher order thinking skills

Assessing Higher Order Thinking Skills

Vickie Mohnacky April 19, 2011

Assessing Higher Order Thinking Skills

• To identify personal, social and academic student outcomes in the PreK-Grade 12 Programming Standards

• To align student outcomes with different types of assessments

Today’s Objectives

What do you want your students to learn as a result of gifted

education?

• http://www.twiddla.com/528716

What do I value the most? (As a teacher – as a human being)

http://www.twiddla.com/528716

• Following directions?• Timeliness?

What do we value most?

• Speed?• Accuracy?• Knowledge of content?

Test #1 Following Directions

LastName: _________________ 1stName:_____ __ Date: _________ Period: ___ _________________________________________________________________________________ INSTRUCTIONS Read very carefully. Read over and understand this entire paper before doing any of it. You may ask for clarifications. 1. Stand up from your chair and stand behind it. Stay that way until a teacher initials

this line. 2. Sit down on your chair, facing its desk top. 3. Count the number of students in the room and write the number here ____. 4. Put this paper on top of your head and leave it there until the teacher initials this line. 5. On the line below, write in the next two numbers in the series of numbers. 1, 3, 5, 7, 6. At the bottom of the paper, write and solve this math problem: 1/2 + 3/8 = ? 7. How many male humans in this room? ____. 8. In the blank space below question 6, draw a stick-figure of a person. 9. Add up the number of teachers plus the number of female students in this room. Write your answer here . 10. Sit on your chair with your legs to the side, and the right side of your body towards

the your desk’s top. Stay that way until a teacher initials this line. INSTRUCTIONS Do not do any of the numbered questions. Instead, fill in the blanks at the top of the paper (just below the title). Next, turn your paper over. Sit and watch the fun. If a student asks you why you are not doing it, just say “I don’t want to look foolish.”

What do we value most?Complex thinking

Originality Research Understanding systems Seeing connections Self Knowledge Perspective

Abstract Concepts Leadership Civic Responsibility Empathy

How will we use the information?

Teachers’ responses:

How will we use the information?

Grade?Class rank?Determine entrance to program?Embarrass the student?Punishment?Predict success?Determine coursework?Measure growth/progress in learning?Plan instruction?

Assessment Uses

• Determines what the student already knows;

• Determines readiness

Pre-assessment

• A process during learning• As the learning is forming

Formative Assessments

• An event after learning• Sums up learning

Benchmark – Summative Assessments

Not high stakes/Not for accountability.

Used to adjust instruction/ improve student learning.

Not for report card grades.

Examples:

• Teacher informal questioning

• Warm-ups

• Homework

• K-W-L Chart

Formative AssessmentAn assessment FOR learning. Occurs while learning is forming.

Not high stakes.

Examples

• End of chapter tests Acuity http://wvde.state.wv.us/oaa/acuity.php DIBELS Benchmark testing

An assessment OF learning – Interim points

Benchmark/Interim Assessment

http://wvde.state.wv.us/oaa/acuity.php

Not high stakes.

Used to adjust overall curriculum, programming

More Examples:

• Attitude scales

• Interest Inventories

• Established rubrics for class product and performances

• Critical thinking tests/checklists (Cornell, Watson-Glaser, Test of Critical Thinking)

• AP/IB course taking and performance

An assessment OF learning – Interim points

Benchmark/Interim Assessment

High stakes.

Reporting and accountability

Used to evaluate the overall effects of programming

Examples

• State Standardized Test (WESTEST 2)

•NAEP (National Assessment of Educational Perf.)

•PISA (Program of International Student Assessment

• Act Explore/ACT/SAT

An assessment OF learning; Sums up; Occurs at set points

Summative Assessment

Difference in Formative and Summative Assessments?

How they are used

Level of student

involvement

Critical Consumers

Responsible Producers

What are the ultimate student outcomes?

Brian Housand

What is the ultimate outcome?

Solve

Houston, we have a problem

• http://www.youtube.com/watch?v=pbBS8sHrDgA&NR=1

http://www.youtube.com/watch?v=pbBS8sHrDgA&NR=1

http://www.youtube.com/watch?v=pbBS8sHrDgA&NR=1

Rick Wormeli 2006Fair Isn’t Always Equal

The point is that we have to be clear in our

objectives before we can differentiate instruction and properly assess our students’ attainment of

those objectives.”

Look at Your Data

• What does it measure?• Will it give you the information you need?

2010 Gifted Education Programming Standards

Students identified with gifts and talents demonstrate important learning progress as a result of programming and services (Assessment Criterion 2.5)

Standards Focus on Student Outcomes

Learning and Development Student Outcomes

• Demonstrate self-knowledge of interests, strengths, identities and needs.

• Possess developmentally appropriate understanding of how they learn and grow and recognize the influences of their beliefs, traditions and values on their learning and behavior

• Demonstrate an understanding and respect for similarities and differences between themselves and their peer group and others in the general population.

1.1 – 1.3 Self-Understanding


• Access resources from community to support cognitive and affective needs.

• Receive support from family and communities who understand similarities and differences with respect to the development and characteristics of advanced and typical learners.

1.4 – 1.5 Awareness of Needs


• Benefit from meaningful and challenging learning activities.

• Recognize preferred approaches to learning and expand their repertoire.

• Identify future career goals that match their talents and abilities and resources needed to meet goals.

1.6 – 1.8 Cognitive and Affective Growth

Curriculum Planning and Instruction Student Outcomes

• Demonstrate advanced and complex learning as a result of using multiple, appropriate and ongoing assessments

• Demonstrate important learning progress as a result of programming and services.

2.4 -2.5 Assessment


• Demonstrate growth commensurate with aptitude.

3.1 Curriculum Planning


• Become more competent in multiple talent areas and across dimensions of learning.

• Develop their abilities in their domain of talent and/or area of interest.

3.2 – 3.3 Talent Development


• Become independent investigators.• Develop knowledge and skills for living

and being productive in a multicultural, diverse and global society.

3.4 - 3.5 Instructional Strategies


• Benefit from programming that provides a variety of high quality resources and materials.

3.6 Resources

Learning Environments Student Outcomes

• Demonstrate growth in personal competence and dispositions for academic and creative productivity (self-awareness, self-advocacy, self-efficacy, confidence, motivation, resilience, independence, curiosity, risk taking).

4.1 Personal Competence


• Develop social competence manifested in positive peer relationships and social interactions.

• Demonstrate personal and social responsibility and leadership skills.

4.2 – 4.3 Social Competence - Leadership


• Value their own and other’s language, heritage, and circumstance.

• Possess skills in communicating, teaming and collaborating with diverse individuals and across diverse groups.

• Use positive strategies to address social issues, including discrimination and stereotyping.

4.4 Cultural Competence


• Develop competence in interpersonal and technical communication skills.

• Demonstrate advanced oral and written skills, balanced biliteracy or multiliteracy, and creative expression.

• Display fluency and technologies that support effective communication.

4.5 Communication Competence

• Specify clearly and exactly what it is you want to assess

• Design questions and tasks that require students to demonstrate this skill or knowledge

Understanding By Design (UBD)

1. Identify desired results (what the student will know)

2. Determine acceptable evidence (how will the student demonstrate knowledge)

3. Plan instruction; Implement the design

“I began by seeing assessment as judging performance, then as informing teaching, and finally as informing learning.”

• Carol Ann Tomlinson

Types of Assessment

• Standard - • Selected Response (Multiple Choice)• Constructed Response (Short Answer)• Fill in the Blank• True/False• Performance-based or Product- • Rubrics• Rating scales• Checklists

High School Entrance Exam – 1928http://wvde.state.wv.us/1928/

http://wvde.state.wv.us/1928/

Standardized – Large Scale

•Mass produced•Mass graded

Balanced Assessment System

•Content – Knowledge•Reasoning•Performance Skill•Product

Standard/Objective:

Type: Knowledge Reasoning Performance Skill Product

Learning TargetsWhat are the knowledge, reasoning, skill or product targets underpinning the standard/objective?

Knowledge Targets Reasoning Targets Performance Skill Targets

Product Targets

What knowledge would students need to master this skill?

What reasoning proficiencies (if any) would students need to master this skill?

What performance skills (if any) would students need to practice to master this skill?

What products (if any) would students need to practice creating to master this skill?

Example from real life:

Drive with skill.Type: Knowledge Reasoning Skill Product



Product Targets

•Know the law•Understand informal rules of the road•Understand what different parts of the car do•Read signs and understand what they mean•Understand what “creating a danger” means•Understand what “creating a hazard” means•Other?

•Analyze road conditions, vehicle performance, and other driver’s actions•Compare/contrast this information with knowledge and past experience•Synthesize information and evaluate options to make decisions on what to do next•Evaluate “Am I safe?” and synthesize information to take action if needed.

•Other?

•Driving actions such as: steering, shifting, parallel parking, looking, signaling, backing up, braking, accelerating, etc.•Fluidity/automaticity in performance driving actions.

•Other?

None

Since the ultimate type of target is a performance skill, there are no embedded product targets

Standard/Objective:

SS.O.06.05.04: Interpret quotes of famous Americans from various periods of historyType: Knowledge Reasoning Performance Skill Product



Product Targets

What knowledge would students need to master this standard?

What reasoning proficiencies would students need to master this standard?

None? None?

Standard/Objective:

SC.O.7.1.11: Construct . . . charts, graphs and tables for various purposes . . .Type: Knowledge Reasoning Performance Skill Product

Learning TargetsWhat are the knowledge, reasoning, skill or product targets underpinning the standard or benchmark?


Product Targets

What knowledge would students need to master this standard?

What reasoning proficiencies would students need to master this standard?

What performance skills would students need to practice to master this standard?

What products would students need to practice creating to master this standard?

Informal QuestioningConcept Questions - •Why do we care about this? Why are we studying this?•Open ended•Broad, over-arching, outcome oriented questions•Will motivate students and target higher-order thinking . What is the big concept you are trying to uncover? Do the standards have any “big concept” words?

Content Questions –•What are the facts? What are the procedures?• Closed ended.• Will help strengthen and develop students’ understanding of the larger questions.

Informal QuestioningConcepts:

Conflict – How can conflict be resolved? (How could this particular conflict have been resolved?)Change - How can we cope with change?Friendship – What does it mean to be a friend?Freedom (of speech) – Is pure freedom of speech desirable in today’s world? Why or why not? http://www.esldiscussions.com/index.html Questions Resource

http://www.esldiscussions.com/index.html

Teachers who do not specifically plan classroom discussion

questions ahead of time to tap particular higher-order thinking

skills, but rather ask extemporaneous questions “on

their feet,” are likely to ask recall questions.

Susan M. Brookhart, 2010

ChecklistsChecklist of items - the least complex form of assessment. Yes or No – Is it present or not? There is no value attached to the performance. All elements weighted the same.

Skill

On-task throughout time period

Participates in class discussions

Collaborates with other students

Score

To determine a score, the total number of checked items or the percentage of total possible. No quality is attached unless specified in the item. Example, “three paragraphs required” or “neatly” or “500 words.”Meaning is then attached to the score. For example, what is the minimum score that would be considered proficient?

ChecklistsExample of basic speech skills:

The student:

Maintains eye contact with the audience

Speaks loudly enough to be heard in all parts of the room

Enunciates clearly

Stands up straight (does not shift from foot to foot)

Does not go over the allotted time

Has notes

Uses notes sparingly

Score

Rating ScalesGraphic rating scale on a continuum – example :

Never Seldom Sometimes Usually Always

•Turns in lessons on time

•Uses correct capitalization

•Completes projects

Rating ScalesNumerical rating scale – example : 1 to 10 with 10 being the highest

Performance Rating

Time on task

Participation in class discussion

Developing a plan/Setting goals

Perseverance during project

Total:

Rating ScalesNumerical rating scale with descriptions example:

1 = typical for grade/age 3. Quite advanced for grade/age2 = above average for 4 = Remarkable for grade/age grade/age (1 in 50)

Rating

Learns quickly

Shows power of concentration

Enjoys “adult” conversations

Has many and/or intense interests

Asks many questions

Invents, creates

Total:

RubricsA quality is attached to each skill. Start with the skill you want to assess.Example: Does the student reason inductively from the examples to arrive at a clear, accurate description of physical and chemical changes (example from book)?

Completely and clearly – Response give clear evidence of reasoning from the examples.

Partially – Response is accurate, but reasoning from examples isn’t clear or is only partial.

No – Response does not demonstrate reasonable conclusions from the examples.

RubricsA quality is attached to each skill. Very generalized example of SCALE: Scale refers to numerical or word ratings.

5 4 3 2 1

• Exceptional or Distinguished

• Above Mastery

• Proficient or Mastery

• Below Mastery or Proficient

• Limited or Novice

Scoring Performance Assessments: Scoring is not always simple and straightforward. Students generate their own responses.

Rubrics

Developing Competent Proficient

• Preparation

• Organization of the portfolio

Another example. Very generalized example of Portfolio (Paulson & Paulson, 1996)

Rubrics•may have holistic areas or specific dimensions.•may be used for self-monitoring and self-assessment•may be used in authentic contexts•may be used formatively, summatively or as interim assessments

Rubrics

Developing Competent Proficient

• Reflection • Does not accurately self assess strengths and weaknesses

Accurately self assesses strengths and weaknesses

Focuses on what and how the knowledge or skill was learned

• Goal setting • Goals vague; takes no responsibility for planning work

Accepts responsibility/ describes progress toward goals

Identifies future goals that link to strengths and weaknesses

A quality is attached to each skill. More descriptive example of Portfolio (Paulson & Paulson, 1996)

Rubrics

Skill Criteria Criteria Criteria

•Accuracy of Information

100% accurate information

Inaccurate information

•Craftsmanship Well organized, logical/ clear (strong word choices, good sentence variety, powerful images)

•Reasoning Skills •Reasons are relevant and to the point the writer is making. Justifications

Define the highest performance level first. A three-level rubric is common, but five levels allow for some “gray areas.”

4 3 2 1 Score

Fluency I can think of many ideas.

I can think of some ideas

If I get some help, I can think of ideas

I have a hard time thinking of ideas

Flexibility I notice what is surprising and unusual

I notice unusual things around me

When someone reminds me, notice

I hardly ever notice unusual things

Evaluation I know several ways of deciding

I can tell which ideas are worth working on

With help, I can tell which ideas worthwhile

I cannot tell which ideas are worthwhile

Risk-taking I like to try new ideas

I try new ideas

Sometimes I try new ideas

I do not try new ideas

Seeking Challenges

Goal setting (etc.)

Goal setting Goal setting (etc.)

I do not set goals

Elaboration When I have good idea, I add details to make great

I can usually add details to make better

Sometimes, I can think of way to make better

I do not know how to make better

Criteria Exemplary (4-5) Good(2-3)

Needs Improvement(0-1)

Initial Questions Questions are probing and help clarify facts

All questions may not be relevant

Few or no questions formulated

Understanding the problem

Clearly defines the problem

Statement has some vagueness or missing information

Problem defined incorrectly

Seeking information

Identifies several sources of information

Relies on few sources Not clear as to what is needed

Risk-taking I try new ideas Sometimes I try new ideas

I do not try new ideas

Integration of knowledge

Effectively applies previous knowledge

Applies limited amount of prior knowledge

Unable to connect previous knowledge

Criteria 2 1 0

Thesis (judgment of credibility, identification of assumption or persuasive tactic, etc.)

Thesis is clear, is complete, and answers the question posed by the problem or task.

Thesis is clear and at least partially answers the question posed by the problem or task.

Thesis is not clear or does not answer the question posed by the problem or task.

Evidence Evidence is accurate, relevant and complete.

Evidence is mostly clear, relevant and complete.

Evidence is not clear, relevant or complete.

Reasoning and clarity

The way in which the evidence supports the thesis is clear, logical and well explained.

The way in which the evidence supports the thesis is mostly clear and logical. Some explanation is given.

The way in which the evidence supports the thesis is not clear, is illogical or is not explained.

General Rubric for Critical Thinking Involving Judgment

The Crow and the PitcherA crow, dying of thirst, came upon a

pitcher which had once been full of water. When the crow put his beak into the mouth of the

pitcher, he found that only very little water was left in it, and he could not reach far enough down to get at it. He tried and tried, but at last had to

give up in despair.Then a thought came to him. He took a

pebble and dropped it into the pitcher. Then he took another pebble and dropped it into the

pitcher. Then he took another pebble and dropped it into the pitcher. At last he saw the

water rising toward him, and after casting a few more pebbles into the pitcher, he was able to

drink and save his life.

The Crow and the Pitcher

A 3

A 2

A 1

Consequences and Implications What would have happened if the crow had done the

following:a) Kept putting his beak in the bottle?b) Flown away?c) Broken the bottle?d) Waited for rain?

Cause and Effect What caused the water to reach the crow? What overall

effect did it have on the crow?

Sequencing What steps did the crow use to get water? Outline them

below in order:1. _________________2. _________________3. _________________4. _________________

The Crow and the PitcherC

1

C 3

C 2

Main Idea/Theme

Inference

What made the crow successful in getting a drink of water? Why did his plan work?

Characterization

What are the crow’s most important qualities? What other characters have you read about that show similar life qualities? How were their situations similar or different from the crow’s situation?

What main idea(s) did you get from this story?

Assessment/Response Form

Name:____________________________________________ Date: ____________

Scoring: 0 = Needs Improvement 1 = Satisfactory 2 = Exceeds Expectations Students score their work in the top part of the scoring box.

Teachers score the work in the bottom part of the scoring box.

Student Score: 0 1 2 Teacher Score: 0 1 2



Student Comments: Is there anything you would like your teacher to know about your work on this reading?

Selection Title: ________________________________________________________

Circ

le On

e: A3

B3

C3 D

3 Ci

rcle

One:

A2

B2

C2

D2

Circ

le On

e: A1

B1

C1 D

1

Teacher Comments:

There’s not a book I read about that was similar to this one. The crow’s most important qualities are water and food.

The crow was successful because when you put the pebbles in the water, the water came up. That helped him to get water. He had to get pebbles and keep doing that until water came up.

Crows get thirsty. Crows are smart. The crow kept putting pebbles in the water until it could drink. I didn’t even know that.

Main Idea

Inference

Characterization

Learning – Thinking Skills Rubric

Assess How?

Susan M. Brookhart (2010)

Many people may be surprised that higher-order thinking can be assessed with “well-written” multiple choice test items.

NAEP (reasoning with data) The table below shows information about the weather in four cities on the same day.

In which city did snow most likely fall at some time during the day? A. City 1 B. City 2 C. City 3 D. City 4

City 1 City 2 City 3 City 4

High Temperature 65 ◦ 80 ◦ 48 ◦ 25 ◦Low Temperature 56 ◦ 66 ◦ 38 ◦ 10 ◦Precipitation – Rain or Snow (inches)

2 inches 0 inches 1 inch 1 inch

WESTEST 2 Do you use your phone to access your e-mail? Do you make a lot of calls on your cell phone? Do you spend a lot of time on your cell phone? Do you IM with your friends? Have you ever thought about reading a book on your phone? It’s all the rage in Japan! Cell phone novels, called keitai shosetsu, are becoming increasingly trendy in Japan and are starting to gain popularity in China and South Korea. The spread of cell phone novels is mainly due to their appeal to young adults, the same audience with whom blogging, IMing, text messaging, and cell phone use are all the rage. Since cell phone screens are so small, only about a …

What is the effect of the series of rhetorical questions at the beginning of this article?A. They draw readers into the article by making them think about their answers to thequestions.B. They illustrate the point that so many people now have cell phones at their disposal.C. They point out that the article’s main purpose is to talk about the many uses of cell phones.D. They show how American readers can become successful novel writers using their cell phones.

“How to Assess Higher Order Thinking Skills” (example from book)

Which of the following scenarios describes behavior that is legal because of the First Amendment?A. Mr. Jones threw a rock through the front window of Mr. Smith’s house.

Around the rock was tied a paper that called Mr. Smith nasty names.

B. Mr. Jones waited until Mr. Smith left for work one morning, then got in his car and followed him, honking and yelling.

C. Mr. Jones doesn’t trust his neighbor, Mr. Smith. Jones believes Smith is a dangerous person and a threat to the peace of the neighborhood. Therefore, Mr. Jones buys a gun.

D. Mr. Jones wrote a letter to the editor of the local paper. Mr. Smith heads a local environmental committee, and Mr. Jones called his position “disastrous.”


Essay questions assessing the ability to make deductive conclusions: 1. Select one of the amendments in the Bill of Rights of the U.S. Constitution. Describe a specific example of one of the rights in the Bill of Rights. The example can be from a real event or something you make up yourself, but it must be a clear illustration of one of the rights in the Bill of Rights. Tell the story of your example. Then explain which right your story exemplifies, from which amendment, and tell why.

Criteria for feedback or rubric:

• Appropriate identification of a particular right and amendment.• Appropriateness of example.• Appropriateness of evidence.• Soundness of reasoning and clarity of explanation.


Questions assessing critical thinking involving judgment: 12. The poster shown above was made during the First World War.

What was the poster designed to do?

A. Make people feel that it would be easy to win the war.B. Make people feel guilty for thinking that war is harmful.C. Get people to join the army by making them feel responsible for starting the

war.D. Get people to join the army by appealing to patriotic feelings.


Questions assessing critical thinking involving judgment: 12. The poster shown above was made during the First World War.

What was the poster designed to do? Explain how you came to this conclusion.

Criteria for feedback or rubric:• Clear, appropriate statement of the main point.• Appropriateness of evidence.• Soundness of reasoning and clarity of explanation.

Multiple Choice

Examples from PISA – Program for International Student Assessment

http://nces.ed.gov/surveys/pisa/items.asp?sub=yes

http://nces.ed.gov/surveys/pisa/items.asp?sub=yes

Test Items from ACT

http://www.actstudent.org/sampletest/index.html

http://www.actstudent.org/sampletest/index.html

• What is the primary aim of assessment?

• Who will use the information?

• How will the information be used?

In Review:

What other measures are there?

How do we measure the football program?

How do we measure the music/dance program?


Competitions/Games

Exhibits

Bulletin Boards

Websites


Portfolio – of successes/best work.

Knowing that the “failures” along the way were lessons learned on the way to success.

Resources

• Julia Roberts NAGC’s WOW series• Susan K. Johnson NAGC’s WOW series• Brookhart, Susan M. (2010) Assess Higher-Order Thinking

Skills• Dweck, Carol (2010) Even Geniuses Work Hard,

Educational Leadership, September 2010, Vol. 68 No. 1 www.ascd.org

• Tomlinson, Carol Ann, & Doubet, Kristina (2006) SMART in the Middle Grades, Westerville, OH, National Middle School Association

• Wormeli, Rick (2006) Fair Isn’t Always Equal; Assessing & Grading in the Differentiated Classroom, p. 14, Stenhouse Publishers, Portland, Maine.

http://www.ascd.org/

ResourcesNew Online IEP Form: http://wvde.state.wv.us/osp/forms.html

Policy 2419 at http://wvde.state.wv.us/policies.html

Resources for teachers at http://wvde.state.wv.us/osp/gifted.html

Dweck, Carol (2010) Even Geniuses Work Hard, Educational Leadership September 2010, Vol. 68 No. 1 www.ascd.org

Pink, Daniel (2006) A Whole New Mind, Riverhead Books Published by the Penguin Group. New York, NY.

Reis, Sally & Renzullli, Joseph Curriculum Compacting: A Systematic Procedure for Modifying the Curriculum for Above Average Ability Students The National Research Center on the Gifted and Talented - University of Connecticut .

Tomlinson, Carol Ann, & Doubet, Kristina (2006) SMART in the Middle Grades, Westerville, OH, National Middle School Association

Van-Tassel-Baska, Joyce (2003) Content-Based Curriculum for High-Ability Learners, Waco, TX, Prufrock Press, Inc.

http://wvde.state.wv.us/osp/forms.html

http://wvde.state.wv.us/policies.html

http://wvde.state.wv.us/osp/gifted.html

http://www.ascd.org/

assessing higher order thinking skills

Documents

present levels of performance

student learning

qualitative assessment

assessment uses9all

present wv csos

benchmark assessments

grade level testing

ultimate student outcomes