assessing quantitative reasoning in student writing: a quirky experience

Assessing Quantitative Assessing Quantitative Reasoning in Student Reasoning in Student

Writing: A QuIRKy ExperienceWriting: A QuIRKy Experience

Nathan D. GraweNathan D. GraweCarleton CollegeCarleton College

Serc.carleton.edu/quirkSerc.carleton.edu/quirk

With support from the US Department of Education’s Fund for the Improvement of Post-With support from the US Department of Education’s Fund for the Improvement of Post-Secondary Education, the National Science Foundation, and the WM Keck Foundation.Secondary Education, the National Science Foundation, and the WM Keck Foundation.

What is QR?What is QR?


The habit of mind to consider the The habit of mind to consider the power and limitations of quantitative power and limitations of quantitative evidence in the evaluation, evidence in the evaluation, construction, and communication of construction, and communication of arguments in public, professional, arguments in public, professional, and personal life.and personal life.


Four facets of QR:Four facets of QR:

1) 1) QR requires a QR requires a basic skill setbasic skill set



1) QR requires a 1) QR requires a basic skill setbasic skill set

2) 2) QR demandsQR demands application in context application in context

"The test of numeracy, as of any literacy, is "The test of numeracy, as of any literacy, is whether a person naturally uses appropriate whether a person naturally uses appropriate

skills in many different contexts" skills in many different contexts"

-National Council on Education and the -National Council on Education and the Disciplines (2001) Disciplines (2001)




2) QR demands2) QR demands application in context application in context

3) 3) QR involves QR involves argumentargument






““Deploying numbers skillfully is as Deploying numbers skillfully is as important to communication as important to communication as deploying verbs.”deploying verbs.”

-Max Frankel, -Max Frankel, The New The New York Times MagazineYork Times Magazine






““Numbers [are] the principal language Numbers [are] the principal language of public argument.” of public argument.”

-BBC Program -BBC Program More or More or LessLess





3) QR involves 3) QR involves argumentargument

4) 4) QR is a QR is a habit of mindhabit of mind

““[QR] is not a discipline but a way of [QR] is not a discipline but a way of thinking….” thinking….” -Lynn -Lynn SteenSteen

Achieving Quantitative LiteracyAchieving Quantitative Literacy

What do the numbers show?What do the numbers show?

How representative is that? How representative is that?

Compared to what?Compared to what?

How was the variable How was the variable operationalized?operationalized?

Who is in the Who is in the measurement sample?measurement sample?

Registered Voters

Eligible Voters

2006 Midterm Elections

Is the outcome statistically Is the outcome statistically significant?significant?

What’s the effect size?What’s the effect size?

What’s the effect size?What’s the effect size?

Consider two statements:Consider two statements: A) A) With a p<.001 With a p<.001 we can conclude that whites we can conclude that whites

in MN, WI, and IA were in MN, WI, and IA were more likelymore likely to vote for to vote for Obama than whites in LA, MS, and ALObama than whites in LA, MS, and AL

B) The fraction of whites who voted for B) The fraction of whites who voted for Obama in MN, WI, and IA was Obama in MN, WI, and IA was at least 35 at least 35 percentage points higherpercentage points higher than that in LA, MS, than that in LA, MS, and AL (<15% vs. >50%) –forget the p value!and AL (<15% vs. >50%) –forget the p value!

What’s the research design What’s the research design (correllational or (correllational or experimental)?experimental)?

Fact: Those who work with Fact: Those who work with computers earn 15-20% more than computers earn 15-20% more than others.others.

““Thus, computer training may, at Thus, computer training may, at least in the short run, be a profitable least in the short run, be a profitable investment for public and private job investment for public and private job training programs.”training programs.”

What’s the research design What’s the research design (correllational or (correllational or experimental)?experimental)?

Other interesting returns:Other interesting returns:

Calculator = 12.8%Calculator = 12.8%

Telephone = 11.4%Telephone = 11.4%

Pencil/Pen = 11.2%Pencil/Pen = 11.2%

Work while sitting = 10.1%Work while sitting = 10.1%

Controlling for what?Controlling for what?

Controlling for what?Controlling for what?

• About 50% of the differential has to do with About 50% of the differential has to do with different career choices….different career choices….

• 25% percent involves greater time women 25% percent involves greater time women spend on care-taking…. spend on care-taking….

• The other 25 percent is due to bias and The other 25 percent is due to bias and prejudice….”prejudice….”


“…“…sophisticated reasoning with sophisticated reasoning with elementary mathematics more than elementary mathematics more than elementary reasoning with elementary reasoning with sophisticated mathematics.”sophisticated mathematics.”

-Lynn Steen-Lynn Steen


Implications for AssessmentImplications for Assessment




3) QR involves 3) QR involves communicationcommunication

4) QR is a 4) QR is a habit of mindhabit of mind


““[QR] is largely absent from our [QR] is largely absent from our current systems of assessment and current systems of assessment and accountability.” accountability.”

-NCED (2001)-NCED (2001)





3) QR involves 3) QR involves communicationcommunication

4) QR is a 4) QR is a habit of mindhabit of mind


Application in context:Application in context:

““Standardized conditions are decontextualized Standardized conditions are decontextualized by design"by design"

-Grant Wiggins-Grant Wiggins

“ “'Get Real!': 'Get Real!': Assessing Assessing for Quantitative for Quantitative Literacy ”Literacy ”


Communication:Communication:

While traditional assessment tools While traditional assessment tools effectively measure comprehension, effectively measure comprehension, the ability to read others’ QR the ability to read others’ QR exposition does not guarantee the exposition does not guarantee the ability to engage in the creation of ability to engage in the creation of QR arguments.QR arguments.


Habit of mind:Habit of mind:

““As in book literacy, evidence of students’ ability As in book literacy, evidence of students’ ability to play the messy game of the [QR] discipline to play the messy game of the [QR] discipline depends on seeing whether they can handle tasks depends on seeing whether they can handle tasks without specific cues, prompts, or simplifying without specific cues, prompts, or simplifying scaffoldsscaffolds from the teacher-coach or test designer.” from the teacher-coach or test designer.”

-Grant -Grant WigginsWiggins “ “'Get Real!': 'Get Real!': Assessing Assessing for for Quantitative Literacy”Quantitative Literacy”


"[QR] requires creativity in "[QR] requires creativity in assessment, since neither course assessment, since neither course grades nor test scores provide a grades nor test scores provide a reliable surrogate." reliable surrogate."

-Lynn -Lynn SteenSteen



"The interdisciplinary and contextual nature "The interdisciplinary and contextual nature of [QR] cries out for a cross cutting of [QR] cries out for a cross cutting approach." approach."

-Lynn Steen-Lynn SteenAchieving Quantitative LiteracyAchieving Quantitative Literacy

““We want to regularly assess student work with We want to regularly assess student work with numbers and numerical ideas numbers and numerical ideas in the field….in the field….””

-Grant Wiggins-Grant Wiggins““'Get Real!': Assessing 'Get Real!': Assessing

for Quantitative Literacy”for Quantitative Literacy”

The QuIRK RubricThe QuIRK Rubric

QuIRK’s idea:QuIRK’s idea:

Measure QR in the “natural” context Measure QR in the “natural” context of papers written in courses across of papers written in courses across the curriculum—papers written for the curriculum—papers written for “authentic” purposes.“authentic” purposes.


Sophomore Writing PortfolioSophomore Writing Portfolio

3-5 papers plus reflective essay written in 3-5 papers plus reflective essay written in 2 of the 4 college divisions 2 of the 4 college divisions

-observation-observation

-analysis-analysis

-interpretation-interpretation

-documented sources -documented sources

-thesis-driven argument -thesis-driven argument


Writing requirement assessment:Writing requirement assessment:30 faculty members, 3 days, 450 30 faculty members, 3 days, 450 portfoliosportfolios

QR assessment:QR assessment:6-8 faculty members, 3 days, random 6-8 faculty members, 3 days, random sample of portfolios, 1 paper each sample of portfolios, 1 paper each drawn from analysis, interpretation, or drawn from analysis, interpretation, or observation categoriesobservation categories


1. A little book-keeping1. A little book-keeping


2. Did the student take the paper in 2. Did the student take the paper in a QR-relevant direction?a QR-relevant direction?


Central Use:Central Use:

Use of numbers to address a central Use of numbers to address a central question, issue, or themequestion, issue, or theme

Peripheral Use:Peripheral Use:

Use of numbers to provide useful Use of numbers to provide useful detail, enrich descriptions, present detail, enrich descriptions, present background, or establish frames of background, or establish frames of referencereference


The importance of the periphery:The importance of the periphery:““Even for works that are not inherently Even for works that are not inherently quantitative, one or two numeric facts quantitative, one or two numeric facts can help convey the importance or can help convey the importance or context of your topic.”context of your topic.”

-Jane -Jane Miller Miller

The Chicago Guide to The Chicago Guide to Writing About NumbersWriting About Numbers


3. If relevant, to what extent did the 3. If relevant, to what extent did the student actually use QR?student actually use QR?


4. If present, rate the holistic quality 4. If present, rate the holistic quality of the QR.of the QR.


5. Spotlight on several recurring 5. Spotlight on several recurring problems:problems:


6. Does assignment explicitly call for 6. Does assignment explicitly call for QR?QR?


Inter-Rater Agreement StatisticsInter-Rater Agreement Statistics

QR Relevance:QR Relevance: 75%75%

QR Extent:QR Extent: 82%82%

QR Quality:QR Quality: 67%67%

Problem Characteristics:Problem Characteristics:68%68%

Examples from Student WorkExamples from Student Work

Example of QR-irrelevant paper:Example of QR-irrelevant paper:

““The Maiden who Needs No The Maiden who Needs No Saving”—an analysis of Keat’s Saving”—an analysis of Keat’s treatment of helplessness and power treatment of helplessness and power in “La Belle Dame Sans Merci”in “La Belle Dame Sans Merci”


Example of ineffective peripheral QR Example of ineffective peripheral QR paper:paper:

Title: UntitledTitle: Untitled

Thesis: A synthesis of 3 alternative Thesis: A synthesis of 3 alternative views on poverty provides a more views on poverty provides a more complete picture than any 1 complete picture than any 1 perspective alone perspective alone

Examples from Student WritingExamples from Student Writing

$192 billion $60 billion1/65th of GNP 1/230th of GNP


Example of effective peripheral QR Example of effective peripheral QR paper:paper:

Title: “Les Banlieues d’Islam”Title: “Les Banlieues d’Islam”

Thesis: Thesis:


Example of ineffective centrally QR Example of ineffective centrally QR paper:paper:

Title: UntitledTitle: Untitled

Thesis: Based on data from a time Thesis: Based on data from a time diary and other sources, the American diary and other sources, the American quality of life is better than that in quality of life is better than that in less-developed societies.less-developed societies.


Example of ineffective centrally QR Example of ineffective centrally QR paper:paper:

““Day Care and Development”Day Care and Development”

Topic: Literature review of effects of Topic: Literature review of effects of day care participation on child day care participation on child developmentdevelopment


Even though quantitative evidence is Even though quantitative evidence is referenced extensively, student referenced extensively, student doesn’t appreciate the real power of doesn’t appreciate the real power of the actual numbers:the actual numbers:

Ex: “The amount of time that Ex: “The amount of time that children spent in day-care was children spent in day-care was positively correlated with reports of positively correlated with reports of how many friends they had.”how many friends they had.”


Example of effective centrally QR paper:Example of effective centrally QR paper:

Title: “Modeling the Implicit Learning of Title: “Modeling the Implicit Learning of Language Acquisition”Language Acquisition”

Thesis: Even if people aren’t aware of Thesis: Even if people aren’t aware of the systems undergirding language, the systems undergirding language, they implicitly learn the system through they implicitly learn the system through experience.experience.


““In phase two, the mean number of correct string In phase two, the mean number of correct string classifications the subjects gave was 17.3 out of 24; all classifications the subjects gave was 17.3 out of 24; all but two participants correctly identified more than 12 but two participants correctly identified more than 12 strings, and the mode number of correct responses strings, and the mode number of correct responses was 18….”was 18….”

[Discussion of statistical test.][Discussion of statistical test.]

““It was expected that the participants would be able to It was expected that the participants would be able to correctly classify more than 12 of the 24 strings in correctly classify more than 12 of the 24 strings in phase two, so the result is not especially surprising. phase two, so the result is not especially surprising. Still, the data provide a striking example of implicit Still, the data provide a striking example of implicit learning. ”learning. ”

Infusing QR Across the Infusing QR Across the CurriculumCurriculum

“…“…numeracy is not something mastered in a numeracy is not something mastered in a single course….Thus quantitative material needs single course….Thus quantitative material needs to permeate the curriculum, … so that students to permeate the curriculum, … so that students have opportunities to practice their skills and see have opportunities to practice their skills and see how useful they can be in understanding a wide how useful they can be in understanding a wide range of problems.” range of problems.”

-Derek Bok (2006)-Derek Bok (2006)

“…“…authentic and enduring learning…can rarely authentic and enduring learning…can rarely succeed one course at a time.”succeed one course at a time.”

-Lee Shulman (1997)-Lee Shulman (1997)


““If [QR] remains the responsibility If [QR] remains the responsibility solely of mathematics departments—solely of mathematics departments—especially if it is caged into a single especially if it is caged into a single course such as ‘Math for Liberal course such as ‘Math for Liberal Arts’—students will continue to see Arts’—students will continue to see [QR] as something that happens only [QR] as something that happens only in the mathematics classroom.”in the mathematics classroom.”

-Lynn -Lynn SteenSteen


Infusing QR Across the Infusing QR Across the CurriculumCurriculumWhere are Where are centrallycentrally relevant papers relevant papers

written? written? ((Population: All papers submitted to writing

portfolio) LegendGreen = Math & Natural SciencesBlue = Social SciencesRed = Other

Dark = Centrally RelevantMedium = Peripherally RelevantLight = Irrelevant

Infusing QR Across the Infusing QR Across the CurriculumCurriculumWhere are Where are peripherallyperipherally relevant papers relevant papers

written? written? ((Population: All papers submitted to writing

portfolio)Legend

Green = Math & Natural SciencesBlue = Social SciencesRed = Other

Dark = Centrally RelevantMedium = Peripherally RelevantLight = Irrelevant


• 30% of gen ed papers are centrally 30% of gen ed papers are centrally QR relevantQR relevant• Of these, nearly 50% are outside natural Of these, nearly 50% are outside natural

sciencessciences

• Another 25% are peripherally soAnother 25% are peripherally so• Of these, 90+% are Of these, 90+% are outsideoutside natural natural

sciences and more than half are in Arts, sciences and more than half are in Arts, Lit, & HumanitiesLit, & Humanities


By QR relevance, what fraction of By QR relevance, what fraction of papers use QR little/none, some, and papers use QR little/none, some, and extensively?extensively?

Centrally Relevant

Peripherally Relevant Either

Little/None 19.5% 94.7% 52.1%

Some 17.1% 5.3% 12.0%

Extensive 63.4% 0.0% 35.9%


““Weasely” examples:Weasely” examples:A) “The downtown areas of large cities A) “The downtown areas of large cities are often home to copious franchises.”are often home to copious franchises.”

B) Recall Daycare PaperB) Recall Daycare Paper

Our Curricular ResponseOur Curricular Response

Our Curricular ResponseOur Curricular ResponseDisciplines Currently Disciplines Currently RepresentedRepresented::

Our Curricular ResponseOur Curricular ResponseComing Soon!:Coming Soon!:

Our Curricular ResponseOur Curricular ResponseExample 1: An Ill-Structured Physics LabExample 1: An Ill-Structured Physics Lab

Example 2: Intro to Lintina/o Studies PaperExample 2: Intro to Lintina/o Studies Paper

Example 3: History Paper on India’s Example 3: History Paper on India’s Colonial CensusColonial Census

Example 4: Exploring and Architectural Example 4: Exploring and Architectural RemodelRemodel

Our Curricular ResponseOur Curricular ResponseExample 5: Environmental History PaperExample 5: Environmental History Paper

Example 6: Development Economics Example 6: Development Economics PaperPaper

Example 7: Native-American Literature Example 7: Native-American Literature AnalysisAnalysis

Our Curricular ResponseOur Curricular Response

assessing quantitative reasoning in student writing: a quirky experience

Documents

communication4 qr

argument4 qr

assessmentfour facets

basic skill

quantitative reasoning

sophisticated reasoning

elementary reasoning

sophisticated mathematics