engaging faculty in the assessment and improvement of critical …€¦ · - course now required of...
TRANSCRIPT
![Page 1: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/1.jpg)
Engaging Faculty in the Assessment and Improvement of Critical Thinking using the CAT Instrument SACS/COC 2014 ANNUAL MEETING
National Science Foundation’s TUES Program under grant 1022789.
PRESENTED BY:
Kevin HarrisTennessee Tech University
Marcus GillespieSam Houston State University
![Page 2: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/2.jpg)
Importance of Critical ThinkingInformation Overload
![Page 3: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/3.jpg)
Remembering Information
Remembering Information
Finding Relevant InformationFinding Relevant Information
The Changing Nature of Education
Understanding & Evaluating Information
Understanding & Evaluating Information
Using Information EffectivelyUsing Information Effectively
![Page 4: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/4.jpg)
What is Critical Thinking?
Classic Emphasis
Evaluate Arguments and Conclusions
Reasoning
![Page 5: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/5.jpg)
Expanded Contemporary EmphasisClassical Emphasis
Evaluate Arguments and Conclusions
Evaluate Arguments and Conclusions
ReasoningReasoning
Evaluate IdeasAnd Plans
Evaluate IdeasAnd Plans
Problem SolvingProblem Solving
CommunicationCommunication
CreativityCreativity
Evaluate One’s OwnUnderstanding
Evaluate One’s OwnUnderstanding
Life-Long Learning SkillsLife-Long Learning Skills
What is Critical Thinking?
![Page 6: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/6.jpg)
Need to Measure Success for Accountability
Why Assess Critical Thinking?
Assessment Drives Improvement Efforts
How We Assess - Determines What Students Learn
![Page 7: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/7.jpg)
History of CAT Development
Collaborate With Other Institutions To Refine CAT
2004 - 2007
Collaborate With Other Institutions To Refine CAT
2004 - 2007
Develop Training Methods forNational Dissemination & Collect Norms
2007 - 2010
Develop Training Methods forNational Dissemination & Collect Norms
2007 - 2010
Expand National Dissemination& Support Assessment in NSF Projects
2010 - 2014
Expand National Dissemination& Support Assessment in NSF Projects
2010 - 2014
Preliminary WorkAt TTU
2000 - 2004
Preliminary WorkAt TTU
2000 - 2004
![Page 8: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/8.jpg)
Over 200 Institutions Collaborating
![Page 9: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/9.jpg)
Designing the CAT Instrument
Faculty Driven:High Face ValidityInvolved in Scoring
Faculty Driven:High Face ValidityInvolved in Scoring
Construct Validity:Learning Sciences
Construct Validity:Learning Sciences
Engaging for Students
Engaging for Students
Reliable &Consistent ScoringEssay Responses
Reliable &Consistent ScoringEssay Responses
![Page 10: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/10.jpg)
Skills Evaluated by CAT Instrument
Evaluating InformationSeparate factual information from inferences.
Interpret numerical relationships in graphs.
Understand the limitations of correlational data.
Evaluate evidence and identify inappropriate conclusions.
Creative ThinkingIdentify alternative interpretations for data or
observations.
Identify new information that might support or contradict a hypothesis.
Explain how new information can change a problem.
Learning & Problem SolvingSeparate relevant from irrelevant information.
Integrate information to solve problems.
Learn & apply new information.
Use mathematical skills to solve real-world problems.
CommunicationCommunicate ideas effectively.
![Page 11: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/11.jpg)
One hour examMostly short answer essayFaculty scored in workshopsDetailed scoring guideReliableValid
CAT Features
![Page 12: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/12.jpg)
National Dissemination Model
InstitutionFaculty Scoring
Session(8-14 Faculty)
2 - 3 Representatives
CATRegionalTraining
![Page 13: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/13.jpg)
Sample Disclosed Question
![Page 14: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/14.jpg)
Assessment Uses of CAT
Informal LearningExperiences
Informal LearningExperiences
Classroom LearningExperiences
Classroom LearningExperiences
Program OutcomesProgram Outcomes
College OutcomesCollege Outcomes
![Page 15: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/15.jpg)
Closing the Loop in Assessment and Quality Improvement
Increase Faculty Awareness of Effective Practices
Increase Faculty Awarenessof Student Weaknesses(Faculty Participate in Test Scoring)
Improve Student Learning(Faculty Implement Effective Practices)
Assess Student Performance
![Page 16: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/16.jpg)
Some Successful Projects
Clemson University
Duquesne University
Purdue University
Sam Houston State University
See Others @ CriticalThinkingTest.org
![Page 17: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/17.jpg)
Sam Houston State University’s QEP to Improve Critical Thinking
Critical-thinking Assessment Test
Scientific reasoning
![Page 18: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/18.jpg)
improve criticalthinking skills
the importance ofevidence and logic
engender scientifichabits of mind
General Goals
![Page 19: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/19.jpg)
Why Did We Choose this QEP Topic?Carnegie Institution Report
> 93% of American adults are scientifically illiterate.
> 78% of college graduates are scientifically illiterate.
![Page 20: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/20.jpg)
Specific Course Goals
Distinguish Science from Pseudoscience
Scientific Content & Terminology from Several Disciplines
Science as a Way of Knowing
Enhance Critical Thinking
![Page 21: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/21.jpg)
Pedagogies:
Case Studies & Team-based Learning
![Page 22: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/22.jpg)
Ex: “Tragic Choices: Autism, Measles, and the MMR Vaccine”
![Page 23: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/23.jpg)
alpha centauri
In addition to standard science topics, we use extraordinary claims to engage the students’
attention and increase motivation…
![Page 24: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/24.jpg)
Students Work in Groups
Groups Share Ideas
Peer Evaluation
![Page 25: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/25.jpg)
Why did we choose the CAT?
CAT specifically addresses scientific reasoning and it is not discipline-specific.
Students are given information in the form of scenarios and asked:• To what degree does the evidence support the
conclusion?• Are there alternative interpretations/hypotheses?
(MWH)• How would you test the idea? What additional
evidence would you need to evaluate the claim?
![Page 26: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/26.jpg)
Examples of FoS QuestionsRico wanted to find out if the majority of people in Texas do not support gun
control laws. So, he surveyed 25 of his friends at the local shooting range.
He found that 90% of them are strongly opposed to gun control laws. Rico
concluded that “Texans strongly oppose gun control laws”. Which of the
following is true?a. Based on his survey results, Rico’s conclusion must be correct.b. The sample size of Rico’s survey is appropriate.c. The group Rico surveyed is appropriate for the purposes of
determining how most Texans feel about gun control laws.d. The survey Rico conducted is not adequate to support his conclusion. e. A, B, and C are correct
![Page 27: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/27.jpg)
Megan believes that eating corn silk from a corn plant (like that shown below) will improve the strength and luster of her hair because the corn silk looks like hair.
What logical fallacy has Meagan committed?
a. appeal to ignoranceb. post hoc ergo propter hoc (false cause)c. faulty analogyd. argument from ignorancee. none of the above – her logic is correct
![Page 28: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/28.jpg)
Assessing CT Gains
Treatment vs. Control
Pre-Test Post-Test DesignUsing CAT Instrument
![Page 29: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/29.jpg)
![Page 30: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/30.jpg)
Perspective
Gains in FoSClass
Gains in FoSClass => Typical Gains
Over 4 YearsTypical GainsOver 4 Years
![Page 31: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/31.jpg)
Benefits of use of CAT to SHSUGraders, from multiple disciplines, have incorporated CAT-like questions into their assignments
CAT results Validated the effectiveness of the course: - Led to presentations on campus focused on CT and
alternative pedagogies, as well as 2 seminars
- Course now required of EC-6 Education majors at SHSU
Grading sessions foster communication among faculty; Enjoy the sessions
Test reveals reading comprehension and writing deficiencies
![Page 32: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/32.jpg)
- Presentations to many universities and contacts with others
Benefits beyond SHSU
- Info on CT has been presented at Correctional Management Institute of Texas seminars; Juvenile Justice and other law enforcement personnel; CT isn’t discipline specific
Validity of CAT made these things possible
![Page 33: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/33.jpg)
Suggestions and Lessons LearnedGive background information in PPt presentation to graders about the test and the rationale for its use at your institution.
Welcome to the CAT Grading Session
CAT = Critical thinking Assessment Test
![Page 34: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/34.jpg)
Repeat graders can become lax…
Keep sample size appropriate…
Limit number of graders to about 12
Try to select graders that are focused and ‘analytical’ – details matter
Suggestions and Lessons Learned
![Page 35: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/35.jpg)
Assistance
CAT Center spot checks representative sample of the tests for accuracy
Can statistically correct results if the score on a question falls outside the range of acceptable variation
CAT Center VERY HELPFUL with any questions we’ve had
![Page 36: Engaging Faculty in the Assessment and Improvement of Critical …€¦ · - Course now required of EC-6 Education majors at SHSU Grading sessions foster communication among faculty;](https://reader036.vdocuments.mx/reader036/viewer/2022081613/5fb5fb3fe9545f60bc1f37d4/html5/thumbnails/36.jpg)
Thank You
www.CriticalThinkingTest.org
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.