Scientific Reasoning Competency Assessment in Higher Education in Virginia

Harold Gellerhgeller@gmu.edu

Department of Physics and AstronomyGeorge Mason University

What I’m Going to Talk About

• SCHEV Mandate for Core Competencies

• Scientific Reasoning Competency

• Basis of GMU Response– Definition– Standards– Methodology– Analysis and Presentation

• Commentary

SCHEV Mandate for Core Competencies

• The State Council of Higher Education for Virginia (SCHEV)

• Meeting on core competencies in November 2001– See state issued guidelines

• Original deadlines modified in 2002– Responses required were delayed to 2003

Scientific Reasoning Competency

• Not easily defined– One approach

• Hazen and Trefil (1991)– Ability to comprehend science article in

newspaper» Does this really define scientific reasoning?» Can this be left to the media?» Is this definition an end-all?» What grade level does this represent?

– GMU approach• See GMU Assessment Proposal

Basis of GMU Response

• Issues hashed out in committee– Definition– Implementation

• Which students take exam• How many students take it• How administered

– Ability to measure across disciplines• Lowest common denominator• Content independence

GMU Definition of Scientific Literacy[by committee]

• Based upon 8 defined learning goals– Identify different scientific ways of knowing– Develop and test hypotheses– Interpret data– Recognize primary and secondary sources– Utilize qualitative and quantitative techniques– Recognize power and limitations of science process– Communicate among scientists and general public– Recognize role of science in decision making

GMU Standards[by committee]

• To be set and established by committee consisting of– All faculty that teach general education

science courses

– Members of the Scientific Reasoning Competency Assessment Committee

GMU Methodology[by committee]

• Develop common test items– Reflect goals in definition

• Pilot questions– Use in final examinations of sample of general

education science courses– Count as part of grade in class

• Analyze results of pilot testing• Develop standards based upon results in

sample of classes

As Part of Methodology(a syllabus addendum)

[by committee]

• Sample supplement to all general education science classes’ syllabi– “One of the goals of the general education program at

George Mason University is to develop students competent in scientific reasoning. This course will help you achieve that goal. In order to judge university success in reaching this goal, a test will be given in late March or early April that will focus on scientific reasoning. This required multiple choice test will take approximately half an hour or so to complete and will count toward your grade.”

Sample Question (vetoed)[by Geller]

• Johannes Kepler (1571-1630) derived his three Laws of Planetary Motion solely from the observations of Tycho Brahe (1546-1601). This represents which of the following ways of scientific knowing?

• A inductive• B deductive• C empirical• D theoretical• E none of the above

Sample Question (vetoed)[by Geller]

• The above graphs represent a phase diagram for water and carbon dioxide. Using the phase diagram for carbon dioxide, what phase would carbon dioxide be found at a pressure of 0.5 atmosphere and a temperature of -78.5o C?

• A liquid• B gas• C solid• D any of the above is possible• E none of the above

Sample Question (vetoed)[by Geller]

• The above graphs all represent experimental values observed over a period of time. The graph that best represents a value that is increasing exponentially over time is which one of the above graphs?

• A Graph A• B Graph B• C Graph C• D Graph D• E None of the above

Time Time Time Time

Sample Question (vetoed)[by Geller]

• Stefan-Boltzmann's Law relates the energy output of an object to its temperature. The energy is proportional to the temperature (in Kelvin) to the fourth power. Therefore, if the temperature (in Kelvin) of an object is doubled, its energy output will increase by how many times?

• A 2• B 4• C 8• D 16• E 32

Sample Questions (affirmed)[Article from Washington Post]

Tumors Prevalent in Anacostia's Fish

Rate Rivals Highest Recorded for U.S. River; Vehicle Pollution Blamed

By Monte Reel

Fish in the Anacostia River have cancerous tumor rates that are as high as ever documented in an American river, and a U.S. government-led study to be published next month links the tumors to pollution caused by vehicle emissions and runoff.

Fifty to 68 percent of mature brown bullhead catfish collected in 2001 from three parts of the river in the city had liver tumors, most of which were cancerous, according to the study led by the U.S. Fish and Wildlife Service. In addition to the liver tumors, 13 to 23 percent of the bullheads had skin tumors, scientists found…

Sample Questions (affirmed)[based upon article from Washington Post]

1. Which one of the following graphs is consistent with the data presented in the article on tumors of Bullhead Catfish at a specific point in the Anacostia river? (for competency 3)

Tumors on Bullhead Catfish in the Anacostia River

010203040506070

1990 1995 2000 2005

Year

% fis

h w

ith

th

ese

tum

ors

liver tumors

skin tumors

Tumors on Bullhead Catfish in the Anacostia River

010203040506070

1990 1995 2000 2005

Year

% f

ish

wit

h t

hese

tum

ors

liver tumors

skin tumors

Tumors on Bullhead Catfish in the Anacostia River

0

10

20

30

40

50

60

70

1990 1995 2000 2005

year

% fis

h w

ith

th

ese tu

mo

rs

skin tumors

liver tumors

Tumors on Bullhead Catfish in the Anacostia River

0

10

20

30

40

50

60

1990 1995 2000 2005

Year

% f

ish

wit

h t

um

ors

liver tumors

skin tumors

Sample Questions (affirmed)[based upon article from Washington Post]

• Which of the following hypotheses are justified given the data in the article? You expect that fish who eat insects in the same river will: (competency 2)

– Have the same level of tumors as brown bullhead catfish because they are living in the same environment.

– Have more tumors because the insects also pick up the air pollution.

– Have fewer tumors than bullhead catfish because they do not eat from the river bottom.

– Can’t be predicted because what the fish eat does not matter.

GMU Analysis and Presentation[by committee]

• Present aggregate data to SCHEV– Percentages of high proficiency– Percentages of acceptable proficiency– Percentages of low proficiency

• Adjust as needed– Determination to be made in pilot testing

Commentary[by Geller]

• How best to test scientific literacy (a.k.a. assessment)– Essays, multiple choice, short answer, mixes

• Problems with need to apply across disciplines– Lowest common denominator– Content independence

• How important is content?

• Comparison with state examinations– Standards of Learning (Virginia)

• Comparison with reading comprehension– New York Times is aimed to be 9th grade reading level– Washington Post aims at 8th grade reading level

Commentary[by Geller]

• Value of tests– Measure of higher education or lower education?– Added burden to all lower level science classes

• What if students take more than one of these introductory science courses?

– Some students take astronomy as general elective.» This can skew statistics.

• Unfunded mandates– More attention to something other than the course

itself?• Why is this a college/university requirement?

– Failure of “lower education”?

Acknowledgements

• GMU Office of Institutional Assessment– Karen Gentemann– Ying Zhou

• Center for Teaching Excellence– Laurie Fathe

• GMU Scientific Literacy Competency Committee– Department representatives

• Faculty volunteers

• Department Colleagues– Robert Ehrlich, Robert Ellsworth, Maria Dworzecka,

Greg Foster (Chemistry), Joe Weingartner