measuring critical reasoning - southern business...

Measuring Critical Reasoning

Prof. Ram NetaUniversity of North Carolina, Chapel Hill

Prof. W. Stewart Thomas, Ph.D., CPA, CGMAUniversity of North Carolina, Pembroke

R. Teague Henry, Ph.D.University of North Carolina, Chapel Hill

10/25/2017 REASZON, Inc, proprietary 1

Introduction: Who Are We?

• REASZON, Inc was founded in 2014 to create an accurate and unbiased assessment of Critical Reasoning ability

• Ram Neta lead the development of the product. He has been researching reasoning for decades, and his online course on critical reasoning has been taken by over a million students around the world.

• Prof. W. Stewart Thomas is Executive Vice Chancellor of Finance for the University of North Carolina, Pembroke.

• Dr. R. Teague Henry is a Postdoctoral Fellow in Psychometrics at the University of North Carolina, Chapel Hill.


What we will talk about today

I. What is Critical Reasoning?II. Why is it important?III. How does it work?IV. How can it be measured?V. Conclusion: an opportunity


I. What Is Critical Reasoning?

• What characteristics make Sherlock Holmes so good at solving crimes?

• He doesn't have a credential and he is not a quick thinker

• But... he notices clues, he can figure out what they mean, and he is not relying on intuition, since he can always clearly explain his thinking.


Intuitive Thinkers vs. Critical Reasoners

• Four decades of research in cognitive psychology have shown that there are two cognitive systems in humans.

• System 1 "is the brain's fast, automatic, intuitive approach“

• System 2 "is the mind's slower, analytical mode... where reason dominates."– Nobel Prize winner Daniel Kahneman, Thinking Fast and Slow (Farrar, Strauss, and Giroux: 2011)

• Critical reasoning is the operation of system 2.


II. Why Critical Reasoning Matters

• Early humans needed to be able to find food, water, and shelter, and to escape from predators or opponents. This relies heavily on system 1.

• In a rapidly changing world, today's humans need to be able to set long‐term goals, and to plan risks and rewards over the long‐term. This relies heavily on system 2.


Evidence that Critical Reasoning Matters

• The importance for work and schools is widely known–Menkes, J. (2005). Hiring for Smarts. Harvard Business Review, 83, November, 1‐10.

– Schmidt, F. L. & Hunter, J. (2004). General mental ability in the world of work: Occupational attainment and job performance. Journal of Personality and Social Psychology, 86, 162‐173.


III. How Does Critical Reasoning Work?

• For decades, teachers and researchers have thought about critical reasoning in the following way:


INFORM + DESCRIBE

NEGOTIATE +

COOPERATE

TEST + REVISE

INTEGRATE + APPLY

INFORM + DESCRIBE

CRITICAL THINKING

What Does This Image Get Right?

• Critical reasoning is the basis for functional cognitive fitness– It is the core of all other thinking skills, just as core body strength is the basis for functional physical fitness

• Improved Critical Reasoning improves all other thinking skills (memory, problem solving, reading comprehension, search), and helps them to work together smoothly


If we want to measure, we need to understand CR in more detail

• Although the image accurately portrays critical reasoning as the core of our other thinking skills, it does not provide enough detail to enable us to measure critical reasoning ability.

• It's like telling us that raising the temperature can make things melt or evaporate, but not giving us enough information to build a thermometer.


Example Of An Assessment QuestionA prominent statesman in Country X has recently claimed that the increased amount of recycling indicates that Country X's natural resources are no longer diminishing and that the amount of waste is no longer increasing. This specious argument is as illogical as it would be to assert that the ever increasing rate at which rain forests are being cut down demonstrates a lack of danger to that resource. The real cause of the increased amount of recycling is an increase in overall consumption, including that of recyclable materials such as paper and plastic.The above argument, if true, best supports which of the following as a conclusion?

The statesman's claim is a deliberate fallacy intended to promote his political interests.

Country X recycles 100% of the waste generated by its population and industries.

Country X's natural resources continue to diminish and the amount of waste continues to grow.

Recycling is the reason for the increasing consumption of recyclable materials in Country X.

The more waste Country X's population and industry generate, the more rain forests need to be cut down.


What’s Wrong With This Approach?

• Other things being equal, someone who's good at critical reasoning will have an easier time answering the test question. But so will someone who's good at reading, or someone who has a big vocabulary, or someone who understands the politics of recycling.

• Measuring critical reasoning by using story problems is like measuring a person's weight by eyeballing their visible size: what you observe ends up being a product of many factors, and not just the factors that you're trying to measure.


IV. How Can We Measure CR Accurately?

• Stanovich, West, and Toplak have developed a measure they call the "Rationality Quotient”

• Measures how rational someone is by listing the most frequent errors that occur when people need to rely on system 2

• Then tests to see how often someone makes those errors

This is like measuring health by checking how many common disorders your patient suffers from, and how severely.


The Rationality Quotient

• Involves 34 measured parameters– Base rate neglect– Conjunction Fallacy– Framing Effects– Anchoring Effect– Sample Size Awareness– Gambler's Fallacy–Myside bias– 4‐card selection task, and so on



Comprehensive Assessment of Rational Thinking (CART)

This is the most recently developed test of RQ. It contains 412 verbal questions, and it takes 3 hours to complete. The questions are written by hand and validated individually, so the test works only so long as the questions are not memorized, and the questions are understood by the test taker.

4‐card selection test

Consider the statement:If a card has a vowel on one side, then it has an even number on the other side.

Which of these four cards would you need to flip over to determine if that statement is true of these cards?

A D 4 7


Conjunction test

Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti‐nuclear demonstrations.

Which is more probable?1.Linda is a bank teller.2.Linda is a bank teller and is active in the feminist movement.


Myside bias test

Consider the following sequence of three numbers:

2, 4, 6

These numbers follow a rule. Figure out what rule they follow by proposing other sequences of three numbers...


The Rationality Quotient (Continued)

• The 34 parameters of RQ all boil down to three fundamental variables– Deduction– Filtering out noise from signal– Pattern recognition

• If we can measure those three abilities directly, without relying on other abilities, then that will give us the most accurate measure of someone's critical reasoning ability


Isolating And Measuring These Traits

• In order to measure the three fundamental capacities directly, we need to develop measures that– Don't rely on the test taker's vocabulary or reading ability or culturally specific knowledge

– Don't rely on the test taker's speed or stamina (questions are not rushed, but very few questions needed for measurement)

– Don't get any easier as a result of memorization


The REASZON Assessment

• The algorithm generates questions that are almost entirely non‐verbal, and so eliminates the role of reading comprehension.

• Generates a new test every time, and so eliminates the role of memory.

• Computer adaptive, and so can detect the user's ability within just a few questions.


The Empirical Case for Test Validity

• REASZON was tested against ETS's newest critical reasoning assessment in a sample of business school students.

• REASZON performs as well or better than the other assessment for predicting relevant indices such as SAT math scores and GPA.


Example: Predicting SATM


Conclusions

• REASZON has comparable predictive performance to competitor product in important outcome measures.

• REASZON is shorter, quicker to get results back, culture blind and simple to translate into any language.


V. Conclusion: An Opportunity• Only approach that measures Critical Reasoning

Directly• Computerized

– “Disruptive Technology” which can be offered at more reasonable price than competitors

• Automatically graded with immediate results– Competitor exams take weeks and months to grade

• Fast (average time to take assessment is 30 minutes)• Eliminates bias of major competitors

– Competitors are verbal / passage based– REASZON is conditional and visual based


Try REASZON for free

Use REASZON to measure the critical reasoning skills of your student body: all use in 2017 is free, and you incur no obligation whatsoever for future use.For more information, find us at www.REASZON.com, or email us [email protected]@[email protected]


Sample Puzzle 1


Solution to Practice Puzzle 1


Sample Puzzle 2


Solution to Practice Puzzle 2


measuring critical reasoning - southern business...

Documents