Evaluating Writing for Evaluating Writing for Quantitative Quantitative ReasoningReasoning

Integrative Learning Project 2005 Summer InstituteIntegrative Learning Project 2005 Summer InstituteCarleton College: Scott Bierman, Liz Ciner, Jackie Lauer-Glebov, Carleton College: Scott Bierman, Liz Ciner, Jackie Lauer-Glebov,

Carol Rutz, Mary SavinaCarol Rutz, Mary SavinaEmails: Emails: crutz@carleton.educrutz@carleton.edu

msavina@carleton.edumsavina@carleton.edusbierman@acs.carleton.edusbierman@acs.carleton.edu

eciner@acs.carleton.edueciner@acs.carleton.edujlauergl@acs.carleton.edujlauergl@acs.carleton.edu

Carleton’s ILP ProjectCarleton’s ILP Project

►Cross-cutting literacies and skillsCross-cutting literacies and skills►Tie in with faculty “workload” – Tie in with faculty “workload” –

improving faculty members’ improving faculty members’ knowledge of what everyone else is knowledge of what everyone else is doing.doing.

Today: One piece – Connecting Today: One piece – Connecting quantitative literacy and writingquantitative literacy and writing

Questions for youQuestions for you

►How do you evaluate students’ writing How do you evaluate students’ writing on your campus?on your campus?

►How do you evaluate students’ How do you evaluate students’ quantitative reasoning (or literacy or. . quantitative reasoning (or literacy or. . . ) on your campus?. ) on your campus?

►How might you combine the two?How might you combine the two?

OutlineOutline

►Background: Background: QuQuantitative antitative IInquiry, nquiry, RReasoning, and easoning, and KKnowledge (nowledge (QUIRKQUIRK) ) initiativeinitiative

►Background: Writing Portfolio initiativeBackground: Writing Portfolio initiative►Writing about QUIRK: First-year Writing about QUIRK: First-year

seminarsseminars►Developing criteria and reading student Developing criteria and reading student

writingwriting► Intersecting the cross-cutting literacies: Intersecting the cross-cutting literacies:

what the future may hold at Carletonwhat the future may hold at Carleton

What is QuIRK?What is QuIRK?

►InquiryInquiry►KnowledgeKnowledge►ReasoningReasoning

►http://www.go.carleton.edu/http://www.go.carleton.edu/quirk>www.go.carleton.eduquirk>www.go.carleton.edu/quirk/quirk

Background: Writing Portfolio Background: Writing Portfolio InitiativeInitiative

►May 2001 – Faculty vote to institute May 2001 – Faculty vote to institute writing portfolio requirement. writing portfolio requirement. Volunteers from class of 2004 submit Volunteers from class of 2004 submit portfolios in May 2002.portfolios in May 2002.

►May 2003 – All sophomores in the May 2003 – All sophomores in the class of 2005 submit writing portfolios.class of 2005 submit writing portfolios.

►May 2004 – All sophomores in the May 2004 – All sophomores in the class of 2006 submit writing portfolios.class of 2006 submit writing portfolios.

The portfolio/QUIRK cycleThe portfolio/QUIRK cycle

Carleton College 2004, FIPSE proposal

Background: History of QuIRKBackground: History of QuIRK► 2001- present– faculty meet informally to systematically 2001- present– faculty meet informally to systematically

discuss concerns about students’ quantitative literacy.discuss concerns about students’ quantitative literacy.► May 2003 – Background white paper presented to Dean and May 2003 – Background white paper presented to Dean and

PresidentPresident► September 2004 – FIPSE grant approved .September 2004 – FIPSE grant approved .► September 2004 and continuing – First-year seminars offered, September 2004 and continuing – First-year seminars offered,

course development funds for faculty, speakers, QuIRK faculty course development funds for faculty, speakers, QuIRK faculty workshopsworkshops

► June 2005 - QUANT squad formed, portfolio reading and rubric June 2005 - QUANT squad formed, portfolio reading and rubric revised revised

► August 2005 – workshop featuring College of San Mateo August 2005 – workshop featuring College of San Mateo learning community – Tools for Thought learning community – Tools for Thought (Jean Mach and (Jean Mach and Mike Burke)Mike Burke)

► December 2005 – Writing workshop “Writing with Numbers”, December 2005 – Writing workshop “Writing with Numbers”, John Bean, facilitatorJohn Bean, facilitator

► May 2006 – First writing examples from QuIRK first-year May 2006 – First writing examples from QuIRK first-year seminars expected in writing portfoliosseminars expected in writing portfolios

How can we help students How can we help students appreciate and strengthen QR?appreciate and strengthen QR?

►First-year seminarsFirst-year seminars►QuIRK across the curriculum, examples:QuIRK across the curriculum, examples:

Improve QR in Biology lab reports.Improve QR in Biology lab reports. Develop a data project on the trans-Atlantic Develop a data project on the trans-Atlantic

slave trade for use in a History course.slave trade for use in a History course. Improve a QR assignment in a Writing Improve a QR assignment in a Writing

Course.Course. Create a new Political Science course Create a new Political Science course

emphasizing comparative electoral analyses.emphasizing comparative electoral analyses.►Faculty development and campus eventsFaculty development and campus events

First-year seminarsFirst-year seminars

► IDSC 100, Measured Thinking: IDSC 100, Measured Thinking: Principles of Quantitative ReasoningPrinciples of Quantitative Reasoning

►POSC 100, Media and Electoral PoliticsPOSC 100, Media and Electoral Politics►SOAN 100, Myths of CrimeSOAN 100, Myths of Crime►ENTS 100, Geology and Human HealthENTS 100, Geology and Human Health

Other QR assignments in first-year Other QR assignments in first-year classes, e.g. English 109 (Rutz, classes, e.g. English 109 (Rutz, Shuffleton)Shuffleton)

Evaluation of First-Year Evaluation of First-Year Seminars Seminars (thanks to Jackie Lauer-Glebov, (thanks to Jackie Lauer-Glebov,

Carleton Office of Institutional Research)Carleton Office of Institutional Research)► 38 students in QR seminars and 45 control 38 students in QR seminars and 45 control

students in other FYS seminars did not differ students in other FYS seminars did not differ at the pretest on seven QR-related questions at the pretest on seven QR-related questions on first-week survey (e.g. “I have the skills to on first-week survey (e.g. “I have the skills to read and understand a statistical analysis of read and understand a statistical analysis of data.”)..data.”)..

► Compared to pretest scores, the posttest Compared to pretest scores, the posttest responses to the QR questions by both QR and responses to the QR questions by both QR and control students were more positive.control students were more positive.

► The QR seminar students had more positive The QR seminar students had more positive responses to the QR questions than the responses to the QR questions than the control students.control students.

Evaluating QuIRK in student Evaluating QuIRK in student writing – May/June 2005writing – May/June 2005

►Norming sessions for “Quant SquadNorming sessions for “Quant Squad™™””►Quant SquadQuant Squad™™ reads 281 portfolios (of reads 281 portfolios (of

about 480 total), flagging 381 essays about 480 total), flagging 381 essays with some QuIRK content from 102 with some QuIRK content from 102 courses, in 25 departments and courses, in 25 departments and programs.programs.

►QuIRK taskforce revises criteria, QuIRK taskforce revises criteria, articulates goals, attends record-setting articulates goals, attends record-setting Minnesota Twins game.Minnesota Twins game.

What we saw in student What we saw in student writingwriting

► Students are aware of the power of Students are aware of the power of quantitative claims and quantitative reasoning.quantitative claims and quantitative reasoning.

► Many student do not put numbers in context – Many student do not put numbers in context – “is this number a big one?” “is this number a big one?”

► Many students are less specific than they Many students are less specific than they should be and they overuse “most,” “many,” should be and they overuse “most,” “many,” “few,” “seldom.”“few,” “seldom.”

► Many students had problems interpreting Many students had problems interpreting numbers and using them to advance numbers and using them to advance arguments.arguments.

Some faculty development Some faculty development implicationsimplications

► Articulate the importance of quantitative Articulate the importance of quantitative reasoning to all faculty (choice of workshop reasoning to all faculty (choice of workshop topics).topics).

► Rewrite assignments to encourage students to Rewrite assignments to encourage students to report and interpret the quantitative content report and interpret the quantitative content of their sources.of their sources.

► Demonstrate what we mean by “good Demonstrate what we mean by “good interpretation” of data (e.g. for gall fly papers).interpretation” of data (e.g. for gall fly papers).

► Highlight excellent assignments in the arts and Highlight excellent assignments in the arts and humanities that encourage students to use humanities that encourage students to use and interpret numbers.and interpret numbers.

June 2005 goals articulationJune 2005 goals articulation

►Goals:Goals: Thinks quantitativelyThinks quantitatively Implements “quantitative analysis” Implements “quantitative analysis”

competentlycompetently Interprets and evaluates thoughtfullyInterprets and evaluates thoughtfully Communicates effectivelyCommunicates effectively

Revised criteriaRevised criteria► States questions under consideration in States questions under consideration in

numerical/quantitative/measurable terms;numerical/quantitative/measurable terms;► Identifies appropriate quantitative/numerical/empirical Identifies appropriate quantitative/numerical/empirical

evidence to address questions and issues;evidence to address questions and issues;► Generates, collects, or accesses appropriate data;Generates, collects, or accesses appropriate data;► Investigates questions and issues byInvestigates questions and issues by selecting and carrying selecting and carrying

out appropriate quantitative or numerical methodsout appropriate quantitative or numerical methods;;► Uses quantitative methods correctly;Uses quantitative methods correctly;► Presents and/or reports the quantitative data appropriatelyPresents and/or reports the quantitative data appropriately;; ► Focuses analysis appropriately on relevant data;Focuses analysis appropriately on relevant data;► Interprets resultsInterprets results to address questions and issuesto address questions and issues under under

consideration;consideration;► Assesses the limitations of the methods employed, if Assesses the limitations of the methods employed, if

appropriate to the task or assignment.appropriate to the task or assignment.

What’s next at Carleton? – Using What’s next at Carleton? – Using student writing to evaluate. . . student writing to evaluate. . .

► Visual literacy? (Working group formed January 2005Visual literacy? (Working group formed January 2005

Visual representations of evidence should be governed by the Visual representations of evidence should be governed by the principles of reasoning about quantitative evidence. For information principles of reasoning about quantitative evidence. For information displays, design reasoning must correspond to scientific reasoning. displays, design reasoning must correspond to scientific reasoning. Clear and precise seeing becomes as one with clear and precise Clear and precise seeing becomes as one with clear and precise thinkingthinking – Edward Tufte– Edward Tufte (1997), Visual Explanations(1997), Visual Explanations

► Information literacy? (Mellon pilot project 2001-2004)Information literacy? (Mellon pilot project 2001-2004)► Group work? (ECC review in progress)Group work? (ECC review in progress)► Integrative learning in individual courses? Integrative learning in individual courses?

(HHMI/CISMI)(HHMI/CISMI)► Ethical inquiry/reflection and civic engagement?Ethical inquiry/reflection and civic engagement?► ????????

Thanks to . . . Thanks to . . .

► Neil Lutsky, Sam Patterson, Jackie Lauer-Neil Lutsky, Sam Patterson, Jackie Lauer-Glebov and the others who wrote the FIPSE Glebov and the others who wrote the FIPSE grant proposalgrant proposal

► Carol RutzCarol Rutz► The QUANT SquadThe QUANT Squad► Corrine Taylor, Director of the Quantitative Corrine Taylor, Director of the Quantitative

Reasoning Program at Wellesley CollegeReasoning Program at Wellesley College► Liz Ciner, John Ramsay and the others in the Liz Ciner, John Ramsay and the others in the

Dean of the College officeDean of the College office► Lynn Steen, Randy Richardson and others Lynn Steen, Randy Richardson and others

outside Carleton who’ve helped usoutside Carleton who’ve helped us