assessing (and improving) critical thinking and quantitative reasoning in ma 103
DESCRIPTION
Assessing (and Improving) Critical Thinking and Quantitative Reasoning in MA 103. Gerald Kruse and David Drews Juniata College Huntingdon, PA [email protected] [email protected]. MA 103, Quantitative Methods, aka “QM ” - PowerPoint PPT PresentationTRANSCRIPT
MA 103, Quantitative Methods, aka “QM”“Mathematics 103 prepares students to be quantitatively literate citizens in today's world. By learning to think critically about quantitative issues, students will be able to make responsible decisions in their daily lives. …as well as to present quantitative output and verbal arguments. ”
MA 103, Quantitative Methods, aka “QM”“Mathematics 103 prepares students to be quantitatively literate citizens in today's world. By learning to think critically about quantitative issues, students will be able to make responsible decisions in their daily lives. …as well as to present quantitative output and verbal arguments. ”
Three Projects during the semester- began using CLA performance tasks Spring
2009- authentic and open-ended
MA 103, Quantitative Methods, aka “QM”“Mathematics 103 prepares students to be quantitatively literate citizens in today's world. By learning to think critically about quantitative issues, students will be able to make responsible decisions in their daily lives. …as well as to present quantitative output and verbal arguments. ”
Three Projects during the semester- began using CLA performance tasks Spring 2009- authentic and open-ended
Pre and Post Assessment (Skills and Attitudes)- 55 min exam given on the first and last class of semester- Fall 2009 transition from math skills to CLA
performance task
“The Collegiate Learning Assessment (CLA) provides one possible example of an assessment that fits a situated notion of QR.”Richard Shavelson, “Reflections on Quantitative Reasoning, an Assessment Perspective,” In B.L. Madison & L.A. Steen (Eds.), Calculation vs. context: Quantitative literacy and its implications for teacher education. MAA.
“The Collegiate Learning Assessment (CLA) provides one possible example of an assessment that fits a situated notion of QR.”Richard Shavelson, “Reflections on Quantitative Reasoning, an Assessment Perspective,” In B.L. Madison & L.A. Steen (Eds.), Calculation vs. context: Quantitative literacy and its implications for teacher education. MAA.
Being quantitatively literate is being “able to think and reason quantitatively when the situation so demands”
Evaluating Evidence
Higher-Order Skills AssessedHigher-Order Skills Assessed
Evaluating Evidence
Analysis / Synthesis / Conclusion
Higher-Order Skills AssessedHigher-Order Skills Assessed
Evaluating Evidence
Analysis / Synthesis / Conclusion
Presenting / “creating” evidence
Higher-Order Skills AssessedHigher-Order Skills Assessed
Evaluating Evidence
Analysis / Synthesis / Conclusion
Presenting / “creating” evidence
Acknowledging alternatives to THEIR conclusion
Higher-Order Skills AssessedHigher-Order Skills Assessed
Evaluating Evidence
Analysis / Synthesis / Conclusion
Presenting / “creating” evidence
Acknowledging alternatives to THEIR conclusion
Completeness
Higher-Order Skills AssessedHigher-Order Skills Assessed
Experimental DesignExperimental Design
Instrument Section 01 Section 02 Section 03
Pre-Assessment QR Assessment QR Assessment Traditional Math Skills
Project 01 Performance Task, based on CLA
Traditional Traditional
Project 02 Performance Task, based on CLA
Traditional Traditional
Project 03 Performance Task, based on CLA
Traditional Traditional
Post-Assessment QR Assessment QR Assessment QR Assessment
• Explicit scoring guidelines (based on rubric) established.
Fall 2009, Version 1.oFall 2009, Version 1.o
• Explicit scoring guidelines (based on rubric) established.
• Scoring guidelines gave “good” reliability.
Fall 2009, Version 1.oFall 2009, Version 1.o
• Explicit scoring guidelines (based on rubric) established.
• Scoring guidelines gave “good” reliability.
• Encouraging, but not statistically significant, results indicated that students in the section with performance task based projects showed more improvement in critical thinking skills.
Fall 2009, Version 1.oFall 2009, Version 1.o
• Explicit scoring guidelines (based on rubric) established.
• Scoring guidelines gave “good” reliability.
• Encouraging, but not statistically significant, results indicated that students in the section with performance task based projects showed more improvement in critical thinking skills.
• Use results to prepare for next round of assessment in Spring 2011.
Fall 2009, Version 1.oFall 2009, Version 1.o
• The content of the course Fall 2009 vs. Spring 2011 remained the same, as did 95% of the “classroom experience,” but the course was reframed with a focus on quantitative reasoning (q. r.) and critical thinking (c. t.)
Modifications for V2.0Modifications for V2.0““How was the student experience different in Spring 2011 vs. Fall How was the student experience different in Spring 2011 vs. Fall
2009?”2009?”
• The content of the course Fall 2009 vs. Spring 2011 remained the same, as did 95% of the “classroom experience,” but the course was reframed with a focus on quantitative reasoning (q. r.) and critical thinking (c. t.)- syllabus - assignments- opportunities during lecture- “salt and pepper”
Modifications for V2.0Modifications for V2.0““How was the student experience different in Spring 2011 vs. Fall How was the student experience different in Spring 2011 vs. Fall
2009?”2009?”
• The content of the course Fall 2009 vs. Spring 2011 remained the same, as did 95% of the “classroom experience,” but the course was reframed with a focus on quantitative reasoning (q. r.) and critical thinking (c. t.)- syllabus - assignments- opportunities during lecture- “salt and pepper”
• The pre/post assessment was modified: - better linkage with specific learning outcomes- more open-ended scenario- names- one prompt
Modifications for V2.0Modifications for V2.0““How was the student experience different in Spring 2011 vs. Fall How was the student experience different in Spring 2011 vs. Fall
2009?”2009?”
• Present the idea of a rubric:- objective assessment, “trust”- familiarize with elements- create one for “chips”
Modifications for V2.0 continuedModifications for V2.0 continued““How was the student experience different in Spring 2011 vs. Fall How was the student experience different in Spring 2011 vs. Fall
2009?”2009?”
Scoring GuidelinesScoring GuidelinesHigher Order Skill None (0) Emerging (1, 2) Developing (3, 4) Mastering (5, 6)
Evaluating evidence provided Identifying relevant evidence, evaluating evidence credibility, reliability, relevance. Keep scores low for people that misunderstand what evidence says/can say. Document A is relevant and from an acceptable source Document B is anecdotal Document C is biased and not relevant Document D is relevant and from an acceptable source
Evaluation of 1 document with: acceptable evaluations (1) If discussing documents A and D, and ignoring B and C, there is some implied reason that these documents are relevant.(1) Evaluation of 2 documents with: acceptable evaluations (2)
Evaluation of 3 documents with: acceptable evaluations (3) Evaluation of 3 documents with: good evaluations (4) Document A is relevant and from an acceptable source Document B is anecdotal Document C is biased and not relevant Document D is relevant and from an acceptable source
Evaluation of 4 documents with: acceptable evaluations (5) Evaluation of 4 documents with: good evaluations (6) Document A is relevant and from an acceptable source Document B is anecdotal Document C is biased and not relevant Document D is relevant and from an acceptable source
Analysis/synthesis /conclusion I. Analysis 1. Document A contains outliers, must conclude that no correlation between headaches and Aspartame, Sucralose, or Placebo consumption 2. Document D has conditional probability table which shows that headaches and Aspartame consumption are independent II. Conclusion - must make one Blowing it on one of the documents is worse than ignoring the document altogether
Implies or directly incorrectly agrees with Sauer, or conclusion is unclear: with no evidence (1) or some evidence, not just agreeing/quoting Sauer (2).
If analyzed Document A (outliers) or Document D (headaches and aspartame independent) well, but no conclusion (2)
Implies or directly disagrees with Sauer, or concludes “more research needed,” analyzing Document A (outliers) OR Document D (headaches and aspartame independent), but analysis is inaccurate/unclear/incomplete (3) or uses A AND D but misinterprets one of them (3) or analysis of A OR D is good (4).
Implies or directly disagrees with Sauer, or concludes “more research needed,” analyzing Document A (outliers) AND Document D (headaches and aspartame independent), but analysis is:
unclear or incomplete(5)
OR
good (6).
Presenting/”creating” evidence Creating a graph, calculating conditional probability
Tries to reorganize information to new format but does so poorly (1).
Makes some reasonable effort to reorganize information but could do more/better, hitting % (2)
Reorganizes information to create compelling graphs/tables or conditional probability, Pr(A|H) (3)
Acknowledging alternatives to their conclusion If disagree w/Sauer and not ban Aspartame, acknowledge that Aspartame could be bad, just substituting sucralose is not an alternative
Acknowledges possibility of alternative(s) to their conclusion, but does not elaborate or with poor reason(s) (1).
Describes an alternative possibility/interpretation to their conclusion, without acceptable rationale or support (2).
Describes an alternative possibility/interpretation to their conclusion, with acceptable rationale or support (3).
Completeness Disagree with Sauer or argue for more research
Completeness Agree with Sauer
Only makes conclusion about Sauer’s claim, but uses no or inappropriate evidence (1).
Only makes conclusion about Sauer’s claim, but uses no or inappropriate evidence (1).
Deals with Sauer’s claim plus minimal attention to strengths and weaknesses of argument (2)
Deals with Sauer’s claim and gives a plan OR strengths and weaknesses of argument (2)
Deals with Sauer’s claim plus good attention to strengths and weaknesses of argument (3)
Deals with Sauer’s claim and gives a plan AND strengths and weaknesses of argument or plan (3)
Mechanics/persuasiveness Exhibiting facility with the English language, especially sentence structure and overall organization.
Distracting errors, poor sentence structure, poor organization (1).
Few errors, generally well-constructed sentences, fair-good organization (2).
Hardly any errors, good sentence structure, good overall organization AND Memo format, From:/To: etc. (3).
• Present the idea of a rubric:- objective assessment, “trust”- familiarize with elements- create one for “chips”
• Improved feedback for projects completed during the semester: - students used guideline to score their work- compared this to my scoring- general trends discussed with entire class- scheduled time to meet for specific feedback
• Each of the three projects emphasized different quantitative content (what we were doing at the time) as well as different categories in the rubric
Modifications for V2.0 continuedModifications for V2.0 continued““How was the student experience different in Spring 2011 vs. Fall How was the student experience different in Spring 2011 vs. Fall
2009?”2009?”
Results for V2.0Results for V2.0
Total Score Pre Post % change
% of possible
improvement
my section 9.46 13.25 40.06 32.84other section 8.2 9.24 12.68 8.13
Evaluating Evidence Pre Post % change
% of possible improvement
my section 1.68 2.5 48.81 18.98other section 0.96 1.24 29.17 5.56
Results for V2.0Results for V2.0
Analysis, Synthesis, Conclusion Pre Post
% change
% of possible improvement
my section 2.86 4 39.86 36.31other section 2.6 3.28 26.15 20.00
Presenting, Creating Evidence Pre Post % change
% of possible
improvement
my section 0.38 1 163.16 23.66other section 0.16 0.6 275.00 15.49
Results for V2.0Results for V2.0
Acknowledging Alternatives to Their Conclusion Pre Post
% change
% of possible
improvement
my section 0.71 0.96 35.21 10.92other section 0.8 0.44 -45.00 -16.36
Completeness Pre Post
% change
% of possible improve-
mentmy section 1.89 2.28 20.63 35.14other section 1.84 1.84 0.00 0.00
Rubric ReliabilityRubric ReliabilityDimension % spot on % +/- 1Evaluation 56.7 40.0
96.7Anal/Synth/Concl 58.6 34.5
93.1Create 73.3 26.7
100.0Alternatives 51.7 37.9
98.7Completeness 62.1 34.5
96.6Total Score 39.2 35.7
75.0
22.520.017.515.012.510.07.55.0
22.5
20.0
17.5
15.0
12.5
10.0
7.5
5.0
J
D
Scatterplot of D vs J
Rubric ReliabilityRubric ReliabilityCorrelation of Total ScoresCorrelation of Total Scores
Pearson correlation of D and J = 0.927 P-Value = 0.000
Critical thinking involves Critical thinking involves evaluating/making good evaluating/making good argumentsargumentsMaking good arguments involves. . .
Clearly stating a conclusionEvaluating and selecting evidenceCreating links between evidence and
conclusion
We consider quantitative reasoning as critical thinking involving numbers/data…
MA 103, Quantitative MethodsMA 103, Quantitative Methods
at Juniata Collegeat Juniata College• Juniata has a Quantitative Skills Requirement,
Q = QM + QS
• MA 103, Quantitative Methods, is offered for students who don’t fulfill the “Q” in their POE
• Three Projects during the semester- began using CLA performance tasks
Spring 2009- authentic and open-ended
Making good arguments involves. . .
Clearly stating a conclusionEvaluating and selecting evidenceCreating links between evidence and
conclusion
We can then consider quantitative reasoning as critical thinking involving numbers/data…
Critical thinking involves Critical thinking involves evaluating/making good evaluating/making good
argumentsarguments
Identifying relevant evidence, evaluating evidence credibility, reliability, relevance
Not Attempted (0)Emerging (1,2)
Mentions one or two documents, with:- No or wrong evaluation on both (1)- cursory-to-OK eval on document C, flawed on other (2)
Developing (3,4)Mentions two documents (one must be C), with:- cursory-to-OK evaluation on both (3)- good evaluation on both (4)
Mastering (5,6)Evaluation of C is good, and evaluates two other doc with:- acceptable evaluations (5)- good evaluations (6)
Evaluating Evidence Category on Evaluating Evidence Category on RubricRubric
Identifying relevant evidence, evaluating evidence credibility, reliability, relevance
Not Attempted (0)Emerging (1,2)
Incorrectly implies , or states directly, agrees that “banning aspartame would improve the health of the state’s citizens” with:- no evidence (1)- evidence (2)
Developing (3,4) Implies, or directly disagrees, with Sauer, noting inconsistency of claim with data in doc C but reason is:- inaccurate/unclear or incomplete(3)- good (4)
Mastering (5,6) - Says C doesn’t support claim and is clear about reason and uses F reasonably well (5)- Satisfactorily uses conditional probability when discussing relationship between headaches and aspartame usage (6)
Analysis/Synthesis/Conclusion Category on Analysis/Synthesis/Conclusion Category on RubricRubric
•Sen. Nathan Dulce is running for re-election vs. Pat Sauer•Proposed bill to ban aspartame, an artificial sweetener, from being added to any soft drink or food product, Dulce opposes, Sauer approves.•Pat Sauer made two arguments during a recent TV interview:(1) Strong correlation between the number of people who consume aspartame and headaches, so,“banning aspartame would improve the health of the state’s citizens.” (2)“Aspartame should be banned and replaced with sucralose.” Pat Sauer supported this argument by referring to a news release.
Performance Task Scenario for Pre and Post-Assessment
22.520.017.515.012.510.07.55.0
22.5
20.0
17.5
15.0
12.5
10.0
7.5
5.0
J
D
Scatterplot of D vs J
22.520.017.515.012.510.07.55.0
22.5
20.0
17.5
15.0
12.5
10.0
7.5
5.0
J
D
Scatterplot of D vs J