thinking and acting like a scientist: sylvia hurtado ... · contact: sylvia hurtado, 405 hilgard...
TRANSCRIPT
1
Thinking and Acting Like a Scientist:
Investigating the Outcomes of Introductory Science and Math Courses
Sylvia Hurtado, Kevin Eagan, and Jessica Sharkness
University of California, Los Angeles
Contact: Sylvia Hurtado, 405 Hilgard Ave., 3005 Moore Hall, University of California, Los Angeles, CA 90095-1521; Phone: (310) 825-1925.
This study was made possible by the support of the National Institute of General Medical Sciences, NIH Grant Numbers 1 R01 GMO71968-01 and R01 GMO71968-05 as well as the
National Science Foundation, NSF Grant Number 0757076. This independent research and the views expressed here do not indicate endorsement by the sponsors.
2
Introduction
Without significant improvement in scientific training in U.S. postsecondary institutions,
America stands to lose its competitive edge in scientific achievement, innovation, and economic
development. The National Academies’ report, Rising Above the Gathering Storm (2007),
revealed that only 15% of all undergraduates in the U.S. receive their degrees in natural science
or engineering, compared with 67% in Singapore, 50% in China, 47% in France, and 38% in
South Korea. To continue the United States’ level of achievement and innovation in science and
engineering, we must not only improve production of undergraduate science majors but also
include increasing numbers of diverse people for the future scientific workforce. Indeed,
increasing the diversity of scientific talent in science, technology, engineering, and mathematics
(STEM) represents a critical part of maintaining and increasing national prominence in science
and engineering. As the Council of Graduate Schools (2007) wrote in a report on graduate
education and American competitiveness, “it is imperative that the U.S. citizens from all
population groups, including those who traditionally have not been highly represented, such as
minorities and women, pursue STEM,” for only in this way can the United States “maintain its
competitive edge” (p. 15).
Central to improving the production of new scientists is the identification of successful
students who express and maintain an interest in science. Recent trend data show increases
among entering freshmen in terms of interest in science and engineering majors as well as in the
number of years of high school coursework completed in biology, math, chemistry and physics
(Pryor, Hurtado, DeAngelo, Sharkness, Romero, Korn & Tran, 2008). Recent increases in the
enrollment of African American, Latino, and Native American students’ in STEM undergraduate
and graduate programs also provide encouraging signs that the U.S. has made progress in
3
diversifying the STEM workforce and educational pipeline (National Science Board, 2008).
However, the odds of remaining in science until degree completion are still currently very low:
Only 24% of underrepresented racial minority (URM) students who begin college as a science
major complete a bachelor’s degree in science within six years of college entry, as compared to
40% of White students (Center for Institutional Data Exchange and Analysis, 2000).
While there are many individual and institutional factors that account for student attrition
(Espinosa, 2009; Seymour & Hewitt, 1997), we wish to sharpen the focus on introductory
coursework in science and mathematics. Aspiring students encounter significant obstacles in the
form of “gatekeeper” courses almost as soon as they begin their collegiate coursework. These
introductory science and mathematics courses are designed to provide foundational learning for
all further STEM coursework. Unfortunately, rather than providing tools for future study, such
courses tend to discourage students from continuing in the sciences (Seymour & Hewitt, 1997).
Drawing from research on science pedagogy and students’ habits of mind for scientific work, we
explore how student experiences in introductory courses affect students’ academic achievement
and ability to think and act like a scientist. The purpose of this study is to assess the factors that
predict acquisition of these important skills and their relationship with academic achievement
(grades) in introductory courses. We hope to understand more about how we can successfully
identify and affirm students in introductory science and math courses in college who have the
necessary dispositions for science.
Success in introductory coursework represents the necessary first step toward the
completion of a bachelor’s degree in STEM. Unfortunately, instructors of these courses often
grade on a curve (allocating very few A grades) and narrowly assess student performance when
assigning the grades. We question whether course grades actually capture the full set of
4
scientific skills that students acquire throughout their introductory coursework. Indeed,
instructors typically base grades in introductory science courses on students’ ability to acquire
and retain specific content knowledge rather than on their development of critical thinking skills,
the latter of which are equally necessary for future science careers (Gainen, 1995).
Assessing students based primarily on mastery of content knowledge makes these
gatekeeper courses resemble sorting mechanisms that harvest student talent rather than develop
it. Several scholars have concluded that prior academic achievement largely determines
students’ grades in introductory coursework (Gainen, 1995; Kamii, 1990; NCES, 2000; Payzant
& Wolf, 1993; Waits & Demana, 1988); therefore, gatekeeper course grades may be more of a
reflection of students’ prior abilities than the acquisition of knowledge and skills during the
course. Particularly in the case of underrepresented racial minority students (as well as women),
prior achievement, motivation, and socioeconomic status represent key predictors of future
success in college science and engineering courses (NCES, 2000; Payzant & Wolf, 1993; Waits
& Demana, 1988). It is important to attempt to untangle these prior preparation and background
factors from performance assessment in courses and also to identify whether students develop in
introductory courses the higher-order cognitive skills that they need future success in STEM-
related careers. Our goal is to provide research that will improve practice in introductory courses
that will lead to broader ways of identifying talent in science (academic success), affirm
development of student skills in assessing performance, and ultimately increase the pool of
students who will continue toward undergraduate degrees in STEM fields.
5
Literature Review
“Gatekeeper” Courses
Scholars and practitioners generally refer to introductory courses as “gatekeepers”
because they represent the first course in a series of classes where knowledge becomes
cumulative (Tobias, 1990). Success in introductory courses is, theoretically, a function of both
scientific thinking dispositions and content knowledge (Conley, 2005; Hagedorn, Siadat, Fogel,
Noral, & Pascarella, 1999). Unfortunately, in practice, college-level introductory science and
mathematics courses tend to focus too much on the acquisition of content knowledge and too
little on the development of meta-cognitive skills related to critical thinking and scientific
literacy (Handelsman, Ebert-May, Belchner, Bruns, Chang, DeHaan, Gentile, Lauffer, Stewart,
Tilghman, & Wood, 2004; Hurd, 1997; Williams, Papierno, Makel, & Ceci, 2004). As a result,
introductory science courses tend to have relatively high failure and dropout rates (Seymour &
Hewitt, 1997). Failure to succeed in gatekeeper courses can lead to difficulty in future courses
and may prompt students to switch out of science majors (Labov, 2004; Seymour, 2001).
The high failure and attrition rates observed in gatekeeper courses have been attributed to
many sources, including large class sizes, a lack of engaging pedagogy, and high competition
among students (Handelsman et al., 2004; Seymour & Hewitt, 1997; Tobias, 1992). Because of
the volume of students who enroll in introductory science and math courses, class sizes tend to
be large, and instructors tend to rely on lecture to transmit course content. Although lecture may
represent an efficient method for presenting domain-specific information to a large audience, it
tends to encourage passive learning (Seymour & Hewitt, 1997). In introductory science courses,
lectures may fail to engage students in critical thought about the content being presented and
may not provide the stimulation necessary to intellectually engage students in learning science
6
and math (Gainen, 1995). Handelsman et al. (2004) note that “active participation in
lectures…helps students develop the habits of mind that drive science. However, most
introductory courses rely on ‘transmission-of-information’ lectures” that do not encourage active
learning (p. 521). Introductory courses that do not engage students can cause students to feel
bored and disconnected from science; these feelings in turn may lead intended science majors,
high achieving or otherwise, to switch out of the sciences (Seymour & Hewitt, 1997).
In addition to feelings of disengagement in introductory courses, students also often
report experiencing high levels of competition among their peers (Gainen, 1995; Mazur, 1992;
Seymour & Hewitt, 1997; Tobias, 1992). Much of the competitive culture within introductory
science and math courses originates from the grading practices of faculty teaching these courses,
as they tend to grade students on a curve (Seymour & Hewitt, 1997). Grading on a curve
discourages collaboration and cooperation among students because helping one’s peers in class
could disadvantage an individual student in terms of final (e.g., post-curve) test scores (Mazur,
1992). Indeed, this grading technique engenders a “survival of the fittest” mentality among
students in introductory courses (Epstein, 2006). Such competitive mentalities seem to be
strongest among science students. Vahala and Winston (1994), for example, found that students
taking laboratory science classes reported experiencing a more hostile classroom environment
than did their peers in English courses.
Classroom Environments and Instructor Pedagogies
A competitive culture represents one of several climates that students may encounter in
their introductory courses. Classroom environments can have a significant effect—positive or
negative—on student achievement (Fraser & Fisher, 1982; Prenzel, Kramer, & Dreschel, 2002;
Seidel, 2006). Scholars consistently have concluded that competitive classroom climates have
7
negative effects on students’ learning and performance. For example, Walberg (1979) examined
the effect of classroom climate on student achievement and retention and found that students
who experienced more competitive classroom contexts tended to have higher rates of failure and
lower levels of self-confidence than their peers in more cooperative environments. Using data
collected from 1,083 science students across 116 classrooms, Fraser and Fisher (1982) found
positive correlations between students’ adoption of scientific attitudes and their perception that
the course encouraged participation. In general, it seems that science students who perceive that
courses have a sense of cohesion and goal orientation tend to have higher levels of academic
achievement compared to their peers who had negative perceptions of their classroom context
(Haertel, Walberg, & Haertel, 1981). These studies imply that if gatekeeper courses have
competitive learning climates that reduce collaboration by forcing students to contend with one
another for grades, most or all students may be negatively affected.
Because of the potential implications associated with how students perceive their
learning environment, both in terms of engagement in learning and the learning climate, many
scholars have examined how pedagogical strategies can be adjusted to encourage student
engagement and collaboration in class. For example, to examine how the extent of lecture and
group work affected student learning, Knight and Wood (2005) conducted an experiment with
two offerings of an upper-division biology course. In the first term, the course instructor used
traditional lecture methods to teach the course. In the second term, the instructor employed not
only lecture but also a number of cooperative and problem-solving activities. Knight and Wood
(2005) compared students’ scores on pre-and post-tests from the beginning and end of each term,
respectively, and found that students who had exposure to the cooperative classroom
environment experienced greater knowledge gains compared to their peers in the more
8
traditional, lecture-heavy classroom environment. The authors repeated this experiment in a
subsequent term and obtained similar results.
A related study by Armstrong, Chang, and Brickman (2007) examined how students
exposed to cooperative learning environments compared to their peers taking courses with a
traditional lecture format. The experiment included two instructors, each of whom taught two
sections of biology: one section utilized a traditional lecture format while the other section
included more cooperative learning and group activities. Controlling for pre-test scores, students
in the cooperative learning groups showed significantly more improvement in understanding
than did their peers in the lecture condition. The cooperative group also had higher attendance
rates and expressed more enjoyment and connection with the work.
A large number of studies over the last 25 years have concluded that incorporating small-
group and collaborative learning activities in large STEM courses provides students with a
number of benefits. Indeed, a meta-analysis by Springer, Stanne, and Donovan (1999)
demonstrated that students who enrolled in STEM classes with a greater emphasis on small-
group and collaborative learning had higher levels of academic achievement, higher rates of
persistence in STEM courses, and more positive attitudes about academics in general than did
their peers who had taken courses dominated by lecture. The specific type of collaborative
learning does not seem to matter; students have benefited from a wide range of interactive and
active learning activities, including interactive-engagement strategies (Hake, 1998); student-
faculty interactions (Astin, 1993; Smith, Sheppard, Johnson, & Johnson, 2005); course
discussion (Allen & Tanner, 2002, 2005; Tanner & Allen, 2005), student feedback or clicker
systems (Wood, 2004), and problem-based learning exercises (Shipman & Duch, 2001).
9
Pedagogical strategies that encourage engagement in discussion and group activities not
only promote achievement and persistence but they also provide students with opportunities to
think critically about scientific concepts and their applications and thus promote higher-order
thinking skills (Allen, Duch, & Groh, 1996; Freedman, 1994; Sagan, 1996). As Smith,
Sheppard, Johnson, and Johnson (2005) describe, collaborative learning exercises offer an
opportunity for the sharing of multiple perspectives on a problem, which allows students to
evaluate several sources of evidence and rationales before deciding on a path toward resolution.
Encouraging student participation in class discussions has been linked to higher-order critical
thinking skills by many researchers (cf. Tsui, 2002), as has using students’ ideas in class,
providing opportunities for student interactions in class, and encouraging more classroom
involvement in general (Smith, 1977, 1981; Terenzini, Theophilides, & Lorang, 1984). Further,
scholars have linked growth in critical thinking skills to other activities that encourage active and
open-ended learning, including participation in research projects (Astin, 1993; Tsui, 1999),
academic enrichment programs (Summers & Hrabowski, 2006), and taking essay exams (Astin,
1993).
Supportive Learning Environments and the Skills Needed for Scientific Success
As noted by Walberg (1979), Fraser and Fischer (1982), Vahala and Winston (1994), and
others, students’ perceptions of the classroom environment can have a direct effect on their
academic achievement and cognitive development. Hostile or competitive environments tend to
have negative effects on students’ achievement whereas positive, supportive, and collaborative
classroom contexts enhance student learning. Prenzel, Kramer, and Dreschel (2002) identified
six necessary conditions for a supportive learning environment: relevance of content; quality of
instruction; teacher’s interest; social relatedness; support of competence; and support of
10
autonomy. Environments that provide relevant content offer students the opportunity to see the
real-world application of concepts and content knowledge. Quality instructors transmit
information to students in clear, coherent ways that meet the learning needs of the particular
audience in the classroom. Interested instructors demonstrate their commitment to student
learning and an ethic of care in regard to students’ academic problems. Classrooms that contain
elements of social relatedness emphasize attributes of collegiality and cooperation. Instructors
that offer students support of competence provide individualized, constructive feedback on
coursework. Finally, learning environments containing support of autonomy offer students
opportunities for exploration in problem solving, which allows for multiple approaches for
reaching resolutions and conclusions.
Seidel (2006) writes that classrooms with these six elements engender greater self-
motivation among students while allowing them to become more self-directed in their learning.
Students who feel supported by their professors and have opportunities to engage in constructive
ways with their peers likely increase the frequency with which they take risks in the classroom
and offer alternative approaches to solving problems (Cobb, 1994; Driver, Asoko, Leach,
Mortimer, & Scott, 1994). Additionally, participating in an environment that encourages the
application of abstract scientific concepts to real-world experiments and problems provides
students with the opportunity to understand the relevance and importance of the material
presented in class. Exercises that allow for such application enable students to transition from
abstract knowledge to more concrete understanding.
Specifically connected to the transition from the abstract to the concrete is students’
ability to learn to think and act like scientists. The ability to think like a scientist involves asking
questions, identifying problems, evaluating evidence to make appropriate judgments, and finding
11
ways to make scientific findings relevant and accessible to society at large (Williams, Papierno,
Makel, & Ceci, 2004). Koslowski (1996) describes two approaches related to how students learn
to act and think like scientists: domain-specific and domain-general. Instructors who take a
domain-specific approach to teaching science rely heavily on transmitting specific scientific
concepts to students, having students memorize textbook definitions, and teaching algorithmic
methods for problem solving. The domain-specific approach assumes that students will develop
higher-order, critical thinking skills as they apply the scientific concepts and knowledge taught in
the classroom (Barnett & Ceci, 2002).
In contrast to the domain-specific approach to teaching science, instructors who utilize
domain-general pedagogical techniques tend to focus more on the development of reasoning and
critical thinking skills that can be applied to a variety of disciplines (Koslowski, 1996; Resnick,
1987). The development of these higher-order thinking abilities provide students with the
requisite skills to more accurately evaluate evidence and choose multiple paths in problem
solving (Kuhn, Garcia-Mila, Zohar, & Anderson, 1995). Rather than merely memorizing a set of
causal relationships, students who have developed appropriate levels of critical thinking skills
can infer causal relationships from evidence presented, and this ability to reason and think
critically about evidence represents a key difference between the domain-specific and domain-
general approaches to teaching.
Additional influences on student success in introductory STEM courses
Course content, pedagogical practices, and learning environments all significantly affect
students’ achievement and development of scientific and general critical thinking skills in
college classrooms. However, scholars have identified a number of experiences external to the
classroom environment that also significantly contribute to growth in students’ critical thinking
12
abilities. For example, a substantial body of research has examined the benefits of research
participation for undergraduate students, showing that engagement in research projects can help
students tremendously in terms of their ability to develop scientific literacy and critical thinking
skills because these opportunities provide students with hands-on learning experiences in which
they must apply concepts learned in the classroom to real-world scenarios (Barlow & Villarejo,
2004; Foertsch, Alexander, & Penberthy, 1997; Lopatto, 2004; Sabitini, 1997). Sabitini (1997)
tracked the progress of several undergraduate students who conducted research with him, and
found that these students reported substantial gains in their problem-solving abilities. The
students also appreciated the opportunity to engage in a team setting.
Another activity that has been connected to growth in achievement and critical thinking
skills is participation in peer tutoring (Bulte, Betts, Garner, & Durning, 2007; Glynn,
MacFarlane, Kelly, Cantillon, & Murphy, 2006; Lockspeiser, O’Sullivan, Teherani, & Muller ,
2008). Glynn et al. (2006) conducted a qualitative study in which fifth-year medical students
served as peer tutors for groups of second-year medical students for a specific educational
module related to the Early Patient Contact program. The second-year students reported that the
peer tutors found ways to make the information contained in the module relevant to them, which
encouraged a deeper understanding as well increased knowledge retention. Glynn et al. (2006)
concluded that peer tutoring environments provide students with a safe learning environment,
which encourages students to ask questions and discuss content more freely than they would in
the context of a classroom. In a similar study, Lockspeiser et al. (2008) found that medical
students who took advantage of peer-tutoring programs reported that they gained more support
from their tutors than from the classroom context, and this helped to put them at ease about their
ability to learn all of the information required of first- and second-year medical school students.
13
Overall, it seems that receiving tutoring or serving as a tutor can help students make learning
connections they might otherwise not have made, especially if their only other source of course
content has been from textbooks and lecture.
Conceptual Framework
Building on previous research, the hypothesized model for this study takes a broad view
of success that encompasses not only the narrow construct of students’ course grade but also
students’ ability to think and act like scientists. As Figure 1 shows, we hypothesize that student
success will be affected by a variety of factors that can broadly be broken down into several
conceptual areas: students’ experience of the learning environment in their introductory courses;
course pedagogy; students’ out-of-class experiences with research and tutoring; the amount of
effort students expend on the course; and students’ critical thinking dispositions. The model in
Figure 1 proposes that these factors significantly affect student success after taking into account
students’ demographic characteristics, pre-tests of success, predispositions toward scientific
thinking, and pre-college experiences.
As shown in Figure 1, the experience of the course learning environment is multifaceted,
encompassing how students feel about and interpret their instructors’ teaching methods,
coursework, and learning environment. Course pedagogy is conceived of as the kind of
instructional methods that are used, the emphasis of the coursework, and how often feedback
about course progress/performance is given. Student effort, background characteristics, and
initial propensity toward thinking and acting like a scientist are included in the model because
these factors have all been shown to affect student success in coursework; controlling for these
variables allows for the examination of the unique impact of the introductory science course and
other related experiences on student success.
14
Critical thinking dispositions, measured using the California Critical Thinking
Dispositions Inventory (CCTDI), are also included as predictors of student success. Developed
by Facione, Sanchez, and Facione (1993), the CCTDI measures individuals’ critical thinking
skills across seven dimensions: inquisitiveness, open-mindedness, systematicity, analyticity,
truth-seeking, critical thinking self-confidence, and maturity. The CCTDI measures have been
shown to be significantly associated with college student GPAs (Giancarlo & Facione, 2001),
and were developed in part as markers for cognitive skills typically used in the sciences, such as
interpretation, analysis, inference, evaluation, and explanation (Facione, Sanchez & Facione,
1993). Including the CCTDI measures in our model allows us to use a standard measure of
dispositions for critical thinking to assess a relationship with estimations of student gains in
scientific thinking during introductory courses.
In terms of the outcomes in the model, student “success” was assessed using three
independent measures: thinking like a scientist, acting like a scientist, and students’ course grade.
Our intention was to separate students’ thinking preferences and performance as a scientist from
the evaluation students received in the course. As shown in Figure 1, we hypothesized that post-
test scores of thinking and acting like scientists would be significantly predictive of students’
final course grade. We hypothesized the presence of these causal paths for two reasons. First,
students reported their self-ratings on items related to thinking and acting like scientists prior to
the posting of their course grade; thus, the temporal order in which we collected these measures
dictated that students’ end-of-term dispositions toward science may affect their final course
grade. Second, we wanted to assess whether course grades were associated with the skills
necessary to be a successful scientist or whether grades are independent of these skills.
15
Method
Data and Sample
We collected the data for this study in conjunction with a larger national research project
that examines the facilitators of and barriers to URM science students’ progression toward
research careers in the sciences. Funded by the National Institutes of Health and the National
Science Foundation, the national project examines both individual experiences and institutional
contexts that affect students’ development as science majors during and beyond college. For the
present study, we collected data using two student surveys administered in introductory science
and math courses at five colleges and universities across the country. The five campuses
included one historically Black university, two predominantly White institutions (PWI), and two
Hispanic-Serving institutions (HSI). These five institutions were selected because of their
reputations for graduating significant numbers of students in the biomedical and behavioral
sciences, and three of the five had NIH-funded undergraduate research programs. At each
institution, we administered the survey to at least two introductory science or math courses.
Each institution had at least one section of biology and chemistry sampled; some had more than
one section of biology represented, and one included a calculus section. Each of the sampled
courses was a “gatekeeper,” as each represented the first in a series of classes where knowledge
is cumulative. In other words, success in any one of the courses in this study was necessary to
move onto the next course in the sequence.
Students in each introductory course were surveyed at the beginning and end of the
spring 2007 academic term. The “Science Student Experience Pre-Survey” was given at the
beginning of the term, and collected information on students’ self-rated academic and science
abilities, their level of comfort in articulating and applying science concepts in experiments, and
16
demographic information. Many of the items on the pre-survey were based on Conley’s (2005)
work in identifying critical skills and dispositions students should have for success in
introductory science courses. Administration of the pre-survey yielded 583 responses across the
five campuses and 12 classrooms, which corresponded to a 25% response rate.
The “Science Student Experience Post-Survey” was given at the end of the term before
students received their final course grade. This survey followed up with students on many of the
same self-rated abilities covered in the pre-survey and had a particular focus on students’ skills
for and dispositions toward thinking and acting like scientists. The post-survey also included 75
items from the CCTDI (Facione, Sanchez, & Facione, 1993), items covering specific
pedagogical techniques used in the introductory courses, and student experiences within and
outside of the course.
The pre- and post-surveys were administered online during the first three weeks of the
term and during the final three weeks of the term, respectively. To assist with survey
administration, instructors reminded students about the survey at the end of many of the classes
during the administration period. Additionally, we sent students emails during these timeframes
to provide survey reminders. As an incentive, students were given the chance to win one of 20
$50 gift certificates. The surveys yielded 263 longitudinal cases that form the analytic sample
for this study. The longitudinal response rate was 52%.
Because the analyses used listwise deletion of cases with missing data on the
dichotomous variables in the model, the final analytic sample was reduced to 255 cases. To
adjust for non-response bias between the two surveys, we created response rates based on
students’ gender, race, and academic class (e.g., freshman, sophomore, etc.). We compared the
weighted longitudinal sample (N=255) with the unweighted pre-survey dataset (N=583) and
17
determined that the response weights did not significantly alter the means and distributions of
key variables from the pre-survey. Satisfied that the weighted post-survey sample represented
the pre-survey response population, we proceeded with the use of the response weights in the
structural equation model.
The majority of respondents in the final sample were female (70%) and most were either
White (32%) or Asian (43%). Native Americans made up 2% of the sample, African Americans
8%, and Latinos/Chicanos 13%. Just over three-quarters (76%) indicated that English was their
native language. The vast majority of students in the sample (86%) reported that they were
majoring in a STEM field; most of the others (73% of those who had “other” majors or who were
“undecided”) believed it was likely that they would major in science. Sixty-four percent of
students were in their first year of attendance at their institution, 27% were in their second year
of attendance, and the remainder were in their third year of attendance or had been attending
longer than three years. Appendix B provides the means and standard deviations of all of the
variables in the final model.
Measures
This study primarily focuses on three outcome variables, measured at or near the end of
the academic term. Two outcomes are latent constructs that represent students’ self-rated ability
to think and act like scientists, and these same constructs were measured as pre-tests in order to
control for prior abilities and experiences. The third outcome measure is a variable representing
students’ final grade in their introductory course, which we collected from registrar’s offices at
each institution. The two latent constructs, ability to think like a scientist and to act like a
scientist, were composed of a set of indicator variables identified through factor analysis. A
confirmatory factor analytic model was run in EQS 6.1 to confirm that the relationships between
18
the indicator variables and the constructs held up for both the pre-test and the post-test. Table 1
presents the indicator variables explained by the latent constructs, their factor loadings, and the
fit indices for the measurement model.
As shown in Figure 1, the hypothesized model includes variables representing students’
background characteristics, prior academic achievement, exposure to pedagogical techniques,
experiences within the course, college experiences external to the course, and CCTDI sub-scales.
Appendix A lists all of the measured variables and coding schemes.
Analyses
Structural equation modeling (SEM) in EQS 6.1 allows researchers to simultaneously
estimate the relationships among sets of variables and confirm latent constructs (Bentler, 2006;
Bentler & Wu, 2002). Parameter estimates are generated by analyses of estimated covariance
matrices. SEM accounts for measurement error and provides overall goodness of fit indices to
determine the adequacy of the model, both of which represent advantages over traditional path
analysis (Laird, Engberg, & Hurtado, 2005). To assess model fit, we relied on three fit indices:
non-normed fit index (NNFI), comparative fit index (CFI), and root mean square error of
approximation (RMSEA). NNFI and CFI values above 0.90 indicate adequate model fit, while
RMSEA scores below 0.06 indicate an appropriate level of fit (Raykov, Tomer, & Nesselroade,
1991).
Our approach began with a confirmatory factor analysis that tested the adequacy of our
measurement model. The measurement model included the observed variables and their
associated latent constructs for both the pre- and post-surveys. This measurement model
confirmed the factor structure of the two pre-test factors and their associated post-tests.
19
In the second step, we added all of the hypothesized predictors and paths presented in
Figure 1 to test the full structural model. We relied on prior research and theory when
considering the Wald and LaGrange Multiplier tests, which suggested the deletion or addition of
causal paths in the model to improve model fit. As the final model, presented in Figure 2,
illustrates, we removed several paths from measures of course pedagogies to the three outcome
variables. Similarly, we deleted several of the paths that connected measures of the course
environment with the three outcome variables. Finally, we removed several of the CCTDI
measures and their hypothesized relationships with the three outcomes. If all paths from a
variable were removed, we dropped the variable from the analysis..
Limitations
A number of constraints with the methodology and the data may limit the generalizability
of our findings and conclusions. First, we situated our study within 12 classrooms across five
campuses of varying type, size, selectivity, and mission, and this constraint may limit the
generalizability of our findings to other types of institutions and classroom contexts. Second, the
sample size utilized in this study was admittedly small, which limited the number of variables we
could include in the analyses. Third, these data have a nested design in which students are
clustered within classrooms which are clustered within institutions. Because of sample size
considerations within each classroom and within each institution, we did not disaggregate data
by classrooms or by institutions. Additionally, the number of classrooms and institutions
prevented us from exploring multi-level models in SEM. Finally, because this study examined
students’ experiences in a single introductory course, the short timeframe in which we
administered the surveys may have affected the amount of change detected in students’ learning
to think and act like scientists. This study chose to focus on student changes in these items over
20
the span of a single academic quarter; had students been tracked over a longer period of time, we
may have detected more substantial changes in these outcomes.
Results
Table 1 presents the results of the measurement model for the four latent variables in the
study: pre- and post-tests of thinking like a scientist and acting like a scientist. The model
statistics suggest that the model fits the data well. The Satorra-Bentler chi-square statistic is
300.69 (df= 305, N = 255, p = 0.55), and the fit indices are NNFI =0.98, CFI = 0.98, and RMSEA
= 0.03. Table 1 shows the factor loadings of observed variables on each of the four factors; all
variables loaded highly on the relevant factor. Given the good fit of the measurement model, we
proceeded with the estimation of the structural model.
Table 2 shows the unstandardized regression coefficients, the standardized regression
coefficients, the standard errors, and the significance levels for the direct effects in the final
model. Figure 2 presents causal paths for the final structural model. Solid lines indicate
significant effects in the final model whereas dotted lines represent non-significant paths. The
Satorra-Bentler chi-square statistic for the final model was 1,285.69 (df = 1,047, N = 255, p <
0.001), and the fit indices were: NNFI = 0.91, CFI = 0.92, and RMSEA = 0.03. Although the chi-
square statistic was significant, it is highly dependent on sample size and degrees of freedom
(Bentler, 2006). The other indices suggest that the model appropriately represents the
relationships among students’ characteristics at the beginning of the course, their experiences
and exposure to pedagogical strategies within the course, and the three primary outcomes.
Considering the results of the predictors related to students’ high school math and science
GPA, only tutoring another student in high school appeared to have a significant relationship
with this endogenous variable (β = 0.23, p < 0.01). We detected no significant differences
21
between men and women in their math and science high school GPA. Additionally, we did not
find differences in math and science high school GPA between URM students and their White
and Asian American counterparts. Just 6% of the variance in high school math and science GPA
was explained with these predictors.
Examining the results related to the two endogenous pre-tests in the model of thinking
and acting like scientists, the coefficients in Table 2 indicate that students’ scores on the AP
Chemistry test significantly and positively predicted the pre-tests of acting like a scientist (β =
0.18, p < 0.05) and thinking like a scientist (β = 0.20, p < 0.05). Women appeared to rate
themselves lower than their male classmates on the both acting (β = -0.29, p < 0.01) and thinking
(β = -0.36, p < 0.01) like scientists. The frequency with which students spent time tutoring their
peers in high school appeared to be significantly and positively related to students’ self-rated
ability to think like a scientist (β = 0.12, p < 0.05). These predictors accounted for 12% and 18%
of the variance in the pre-tests of acting and thinking like a scientist, respectively.
The most important predictor in students’ self-rated ability to think like a scientist at the
end of the academic term was their self-rated pre-test (β = 0.57, p < 0.001). Three CCTDI
measures significantly predicted students’ post-test scores for thinking like a scientist.
Openmindedness had a significant and negative relationship with the outcome (β = -0.19, p <
0.01) while analyticity (β = 0.14, p < 0.05) and critical thinking self-confidence significantly and
positively predicted post-test scores on thinking like a scientist.
Only one course pedagogical strategy significantly predicted thinking like a scientist at
the end of the academic term. Students who agreed with the statement that the instructor
primarily used lecture to present course material scored themselves higher on their abilities to
think like scientists compared to their peers in less-lecture-intensive environments (β = 0.13, p <
22
0.05). We should note that 96% of students either agreed somewhat or agreed strongly that their
instructor primarily used lecture methods for course instruction. Further inspection of student
responses to the lecture variable by institution and course shows that a very small proportion of
students in each class disagreed that the course was primarily lecture. Additional post-hoc
analyses indicate that virtually all students who disagreed that the course was primarily lecture
also indicated that they “never” attended class or lab. Therefore, the lecture variable is likely
functioning as a proxy for class and lab attendance, and the positive predictive power of lecture
on thinking like a scientist does not mean that lecture actually promotes this particular
disposition but rather that attending lecture (class) encourages students to think more like
scientists.
Feeling overwhelmed negatively contributed to respondents’ ability to think like
scientists (β = -0.14, p < 0.01); however, intending to major in one of the biomedical or
behavioral sciences had a positive association with thinking like a scientist at the end of the
academic term (β = 0.15, p < 0.05). The model accounted for approximately 61% of the variance
in students’ ability to think like scientists.
A number of the same variables emerged as significant predictors of students’ ability to
act like scientists at the end of the academic term. Similar to thinking like a scientist, students’
pre-test represented the most important predictor of their self-rated ability to act like scientists at
the end of the academic term (β = 0.56, p < 0.001). Openmindedness was negatively associated
with acting like a scientist (β = -0.15, p < 0.05) whereas critical thinking self-confidence
positively predicted this disposition (β = 0.40, p < 0.01). Students in courses that relied heavily
on lecture as a pedagogical technique tended to rate themselves higher on their ability to act like
scientists (β = 0.16, p < 0.05). Again, lecture appears to serve as a proxy for course attendance.
23
Additionally, students who felt overwhelmed by course expectations reported lower scores on
the construct acting like a scientist (β = -0.26, p < 0.01). These direct effects accounted for
approximately 60% of the variance in students’ self-rated ability to act like scientists.
The final outcome variable of interest was students’ course grades. We were especially
interested in examining the relationship between students’ scores on thinking and acting like
scientists and their final course grade. As shown in Table 2, we did not detect a significant
relationship between these dispositions and students’ grade. Instead, we found a significant and
positive effect from students’ prior academic achievement (β = 0.19, p < 0.05) and prior
participation in a high school science research program (β = 0.13, p < 0.05). Additionally,
students in classroom environments that encouraged working in small groups earned higher
grades than their peers in learning environments that did not emphasize group work (β = 0.21, p
< 0.05). Cramming for exams (β = 0.19, p < 0.01) and seeking tutoring on campus (β = 0.10, p <
0.05) significantly and positively predicted students’ final course grade. Similar to the findings
related to thinking and acting like scientists, feeling overwhelmed negatively affected students’
academic performance in the introductory course (β = -0.23, p < 0.01). In fact, feeling
overwhelmed represented the strongest predictor of students’ final grade in the course – even
stronger than their average math and science GPA in high school. We did not detect any
significant effects on course grade from students’ feelings of competition, time spent in lab,
participation in a college research program, or academic major. Similarly, we did not identify
any significant paths from the CCTDI measures to students’ final course grades, as indicated by
the dotted lines in Figure 2. The predictors for students’ final course grade accounted for
approximately 22% of the variance in this measure.
24
Table 3 presents the indirect effects for the three outcome measures. Students’ AP
chemistry test scores, operating through each of the respective pre-tests had significant, positive
indirect effects on both thinking (β = 0.11, p < 0.05) and acting (β = 0.10, p < 0.05) like
scientists. In the same way, being female had a significant, negative indirect effect on thinking (β
= -0.21, p < 0.05) and acting (β = -0.16, p < 0.05) like scientists. Finally, tutoring another student
in high school also had a significant, positive indirect effect on thinking like a scientist (β = 0.07,
p < 0.05). Each of these indirect effects operated through the respective pre-tests of these factors.
Table 3 shows one significant indirect effect for students’ final course grades. Tutoring
another student in high school appeared to exert a significant, positive indirect effect on students’
final course grade through high school science GPA (β = 0.06, p < 0.05). The other indirect
effects on students’ final course grade were not significant.
Discussion and Conclusion
One of the primary goals of higher education is to educate cadres of graduates whose
talents can improve society and global economic competitiveness. Young scientists represent a
key part of this future generation, and, as technology increasingly becomes part of everyday life,
new scientific talent becomes increasingly important (Council of Graduate Schools, 2007). The
National Science Foundation’s National Science Board notes that science and engineering serve
as the primary drivers of both economic growth and national security, and that “excellence in
discovery and innovation in science and engineering…derive from an ample and well-educated
workforce” (National Science Board, 2003, p. 7). Improvement in science teaching and
classroom practice is now more critical than ever to maintaining America’s competitive edge
with a more diverse student population that has the potential to become the next generation of
scientists.
25
Although students’ grades in introductory courses may be useful in sorting students, they
do not seem to be useful in capturing gains in dispositions for scientific work or dispositions for
critical thinking. Our model demonstrated that grades were not significantly related to students’
ability to think like a scientist and act like a scientist, nor were they related to any CCTDI sub-
scales. Instead, students’ course grades were related to cramming for exams, tutoring, and feeling
overwhelmed by course expectations. This confirms previous work that found college-level
introductory science and mathematics courses tend to focus too much on the acquisition of
content knowledge and too little on the development of meta-cognitive skills related to critical
thinking and scientific literacy (Handelsman, et. al., 2004; Hurd, 1997; Williams, Papierno,
Makel, & Ceci, 2004). As we expected, students’ course grades were in large part predicted by
high school preparation (grades and research experience).
If grading practices are the primary sorting mechanism colleges employ in science, and if
grades in large introductory are meted out on a curve with relatively few students earning high
marks, this means that talented students who do not out-compete their peers may be weeded out
of science majors very early on in their college career. Students who do not earn top grades do
not necessarily lack the skills needed to be a good scientist; they may simply lack the prior
preparation or study skills needed to perform well in lecture-based classes that reward cramming
for exams. To keep talented students in science majors, we need to broaden performance criteria
and assessment techniques. Grades alone will not identify the nascent scientific talent that exists
among college students.
Interestingly, students earned higher grades when they were involved in courses where
group work was encouraged. This is congruent with the literature that shows the myriad benefits
of group work in terms of student performance (Armstrong, Chang, & Brickman, 2007; Knight
26
& Wood, 2005; Shipman & Duch, 2001; Springer, Stanne, & Donovan, 1999). Providing
students with opportunities to share ideas and evaluate multiple sources of evidence reinforces
content and concepts presented in class (Armstrong, Chang, and Brickman, 2007; Knight &
Wood, 2005). This reinforcement of learning may have longer-term benefits in terms of students’
retention of domain-specific knowledge (Koslowski, 1996; Kuhn, Garcia-Mila, Zohar, &
Anderson, 1995).
Students also appear to experience benefits in their learning from tutoring programs on
campus. Students who sought tutoring on campus for help in their coursework also earned higher
grades. Similarly, having tutored another student in high school provided students with an
indirect benefit on their final course grades. Respondents who tutored their peers in high school
tended to earn higher grades in their high school math and science courses, which then translated
into a higher grade in their introductory science or math course in college. These findings
suggest that it is possible to harness the power of peer learning, model team work, and also to
reward this cooperation in classrooms.
One troubling aspect of the classroom environment that emerged from this study relates
to students’ feelings of being overwhelmed by course expectations. Respondents who indicated
that they agreed with the statement that they felt overwhelmed by course expectations tended to
earn lower course grades and score lower on the constructs of thinking and acting like scientists.
These feelings may represent a proxy for students’ perception regarding their prior preparation
for introductory science and math coursework or a general naïveté about what is expected from
students enrolled in introductory courses. Indeed, Seymour and Hewitt (1997) concluded that
students’ feelings of being overwhelmed with coursework often resulted from poor preparation
27
in high school and unrealistic expectations for the work associated with being a science major,
which eventually led students to leave the sciences.
Although students who felt overwhelmed earned lower course grades and scored lower
on the scientific habits of mind, the findings suggest that three of the critical thinking subscales
had significant associations with thinking and acting like scientists. Interestingly, we found a
negative relationship between openmindedness and students’ dispositions toward science. The
CCTDI openmindedness subscale measures the extent to which individuals show tolerance of
others’ opinions and perspectives. Given this definition, the negative effect of this variable on
students’ self-rated abilities to think and act like scientists is not surprising, as individuals who
have more of a relativistic perspective may be less likely to embrace the objective, empirical
nature of science.
In contrast to openmindedness, students’ critical thinking self-confidence had a
significant, positive relationship with both thinking and acting like a scientist. This finding
corresponds to the higher-order thinking In contrast, the habits of mind for scientific work were
supports earlier studies on the development of higher-order thinking skills (Koslowski, 1996).
These findings indicate that students who rate themselves higher on their meta-cognitive abilities
also tend to be more likely to develop domain-general scientific skills that enable them to infer
causal relationships and more adeptly evaluate evidence when engaging in problem-solving
activities (Kuhn, Garcia-Mila, Zohar, & Anderson, 1995).
Analyticity was the third critical thinking subscale that related to thinking like a scientist.
Analyticity measures the extent to which students can anticipate the multiple outcomes and
consequences in decision-making processes. Students with high scores on the analyticity
28
subscale appear to think more critically and carefully during problem-solving activities and ask
relevant questions so as to gather as much information as possible before proceeding.
Implications
Many NSF projects are specifically devoted to interventions that are designed to improve
the teaching and learning of science, yet there remains incredible resistance to change. The
question that science faculty must confront is whether we can afford to cram content at the
expense of the development of scientific skills and thinking, and continue to let grading practices
reflect previous preparation rather than actual learning in the classroom. With increased interest
in STEM among entering students, the U.S. is at a critical crossroads in an opportunity to
improve the production of science degrees. In order to move forward most productively, faculty
must reexamine current practices.
Institutional researchers may assist in this reexamination by helping science faculty to
broaden their assessment of student performance, employing varied ways of assessing student
skills, and using multiple measures to evaluate existing programs. Currently, the vast majority of
students are still in large lecture venues in introductory science and mathematics courses. Further
research is needed to understand the impact of more varied and engaging pedagogies used by
faculty in science. Investments made in these areas are necessary to open the valve for the
movement of current students who will expand and diversify the scientific workforce.
29
References
Allen, D. E., Duch, B. J., & Groh, S. E. (1996). The power of problem-based learning in teaching introductory science courses. In Wilkerson, L. and Gijselaers (Eds.), New Directions for Teaching and Learning, 68. San Francisco: Jossey-Bass.
Allen, D., and Tanner, K. (2002). Answers worth waiting for: One second is hardly enough. Cell Biology Education, 1(1), 3–5.
Allen, D., & Tanner, K. (2005). Infusing active learning into the large-enrollment biology class: seven strategies, from the simple to complex. Cell Biology Education, 4, 262-268.
Armstrong, N., Chang, S., & Brickman, M. (2007). Cooperative learning in industrial- sized biology classes. Life Sciences Education, 6(2), 163-171.
Astin, A. W. (1993). What matters in college? Four critical years revisited. San Francisco, CA: Jossey-Bass.
Barlow, A. E. L., & Villarejo, M. (2004). Making a difference for minorities: Evaluation of an educational enrichment program. Journal of Research in Science Teaching, 41(9), 861-881.
Barnett, S. M. & Ceci, S. J. (2002). When and where do we apply what we learn? A taxonomy for far transfer. Psychological Bulletin, 128(4), 612-637.
Bentler, P. M. (2006). EQS 6 structural equations program manual. Encino, CA: Multivariate Software, Inc.
Bentler, P. M. & Wu, E. J. C. (2002). EQS 6 for Windows user’s guide. Encino, CA: Multivariate Software, Inc.
Bulte, C., Betts, A., Garner, K., & Durning, S. (2007) Student teaching: Views of student near-peer teachers and learners. Medical Teacher, 29(6), 583-590.
Center for Institutional Data Exchange and Analysis. (2000). 1999-2000 SMET retention report. Norman, OK: University of Oklahoma.
Cobb, P. (1994). Where is the mind: Constructivist and sociocultural perspectives on mathematical development. Educational Researcher, 23(1), 13-19.
Committee on Science, Engineering, and Public Policy. (2007). Rising above the gathering storm: Energizing and employing America for a brighter economic future. Retrieved February 25, 2008, from National Academies Press website, http://www.nap.edu/catalog/11463.html
Conley, D. T. (2005). College knowledge: What it really takes for students to succeed and what we can do to get them ready. San Francisco: Jossey-Bass.
Council of Graduate Schools. (2007). Graduate education: The backbone of American competitiveness and innovation. Washington, DC: Council of Graduate Schools.
Driver, R., Asoko, H., Leach, J., Scott, P., & Mortimer, E. (1994). Constructing scientific knowledge in the classroom. Educational Researcher, 23(7), 5-12.
Epstein, D. (2006). So that's why they're leaving. Inside Higher Education Retrieved February 2, 2007, from http://insidehighered.com/news/2006/07/26/scipipeline
Espinosa, L. (2009). Pipelines and pathways: Women of color in STEM majors and the experiences that shape their persistence. Unpublished doctoral dissertation.
Facione, P. A., Sanchez, C. A., & Facione, N. C. (1993, June). Are college students disposed to think? Paper presented at the Association for the Advancement of Higher Education Assessment Forum, Chicago, IL.
Foertsch, J., Alexander, B. B., & Penberthy, D. (1997). Summer research opportunity
30
programs (SROPs) for minority undergraduates: A longitudinal study of program outcomes, 1986-1996. Madison, WI: The Lead Center, University of Wisconsin-Madison.
Fraser, B. J. & Fisher, D. L. (1982). Predicting students’ outcomes from their perceptions of the classroom psychosocial environment. American Educational Research Journal, 19(4), 498-518.
Freedman, S. W. (1994). Exchanging writing, exchanging cultures: Lessons in school reform from the United States and Great Britain. Cambridge, MA and Urbana, IL: Harvard University Press and National Council of Teachers of English.
Gainen, J. (1995). Barriers to success in quantitative gatekeeper courses. In J. Gainen & E. W. Willemsen, (Eds.), Fostering student success in quantitative gateway courses .New Directions for Teaching and Learning, 61. San Francisco: Jossey-Bass.
Giancarlo, C. A. & Facione, P. A. (2001). A look across four years at the disposition toward critical thinking among undergraduate students. The Journal of General Education, 50(1), 29-55.
Glynn, L. G., MacFarlane, A., Kelly, M., Cantillon, P., & Murphy, A. W. (2006). Helping each other to learn: A process evaluation of peer-assisted learning. BMC Medical Education, 6(18), 1-9.
Hagedorn, L. S., Siadat, M. V., Fogel, S. F., Nora, A., & Pascarella, E. T. (1999). Success in college mathematics: Comparisons between remedial and nonremedial first-year college students. Research in Higher Education, 40(3), 261-284.
Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand- student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66, 64-74.
Haertel, G. D., Walberg, H. J. & Haertel, E. H. (1981). Socio-psychological environments and learning: A quantitative synthesis. British Educational Research Journal, 7, 27-36.
Handelsman, J., Ebert-May, D., Beichner, R., Bruns, P., Chang, A., DeHaan, R., et al. (2004). Policy Forum: Scientific teaching. Science, 304(5670), 521-522.
Hurd, P. D. (1998). Scientific literacy: New minds for a changing world. Science Education, 82(3), 407-416.
Kamii, M. (1990). Opening the algebra gate: Removing obstacles to success in college preparatory mathematics courses. Journal of Negro Education, 59(3), 392-406.
Knight, J. K. & Wood, W. B. (2005). Teaching more by lecturing less. Cell Biology Education, 4(4), 298-310.
Koslowski, B. (1996). Theory and evidence: The development of scientific reasoning. Cambridge, MA: MIT Press.
Kuhn, D., M. Garcia-Mila, A. Zohar, and C. Anderson. 1995. Strategies of Knowledge Acquisition. Chicago: Society for Research in Child Development.
Labov, J. B. (2004). From the National Academies: The challenges and opportunities for improving undergraduate science education through introductory courses. Cell Biology Education, 3(4), 212-214.
Laird, T. F. N., Engberg, M. E., & Hurtado, S. (2005). Modeling accentuation effects: Enrolling in a diversity course and the importance of social action engagement. The Journal of Higher Education, 76(4), 448-476.
Lockspeiser, T. M., O’Sullivan, P., Teherani, A., Muller J. (2008). Understanding the
31
experience of being taught by peers: The value of social and cognitive congruence. Advances in Health Sciences Education, 13(3), 361-372.
Lopatto, D. (2004). Survey of undergraduate research experiences (SURE): First findings. Cell Biology Education, 3(4), 270-277.
Mazur, E. (1992). Qualitative vs. quantitative thinking: Are we teaching the right thing? Optics and Phonics News, February, 38.
National Center for Education Statistics. (2000). Entry and persistence of women and minorities in college science and engineering, NCES 2000-601. Washington, DC: U.S. Department of Education.
National Science Board. (2003). The science and engineering workforce: Realizing America’s potential, NSB 03-69. Arlington, VA: National Science Foundation.
National Science Board. (2008). Digest of key science and engineering indicators 2008, NSB 08-2. Arlington, VA: National Science Foundation.
Payzant, T. W., and Wolf, D. P. (1993). Piloting pacesetter: Helping at-risk students meet high standards. Educational Leadership 50(5): 42-45.
Prenzel, M., Kramer, K., & Drechsel, B. (2002). Self-determined and interested learning in vocational education. In K. Beck (Ed.), Teaching–learning processes in vocational education (pp. 43–68). Frankfurt .: Peter Lang.
Pryor, J. H., Hurtado, S., DeAngelo, L., Sharkness, J., Romero, L. C., Korn, W. S., & Tran, S. (2008). The American freshman: National norms fall 2008. Los Angeles, CA: Higher Education Research Institute, UCLA.
Raykov, T., Tomer, A., & Nesselroade, J. R. (1991). Reporting structural equation modeling results in Psychology and Aging: Some proposed guidelines. Psychology and Aging, 6(4), 499-503.
Resnick, L. B. (1987). Education and learning to think. Washington, DC: National Academy Press.
Sabatini, D. A. (1997). Teaching and research synergism: The undergraduate research experience. Journal of Professional Issues in Engineering Education and Practice, 123(3), 98–102.
Sagan, C. (1996), The Demon-haunted World: Science as a Candle in the Dark. New York: Ballantine Books.
Seidel, T. (2006). The role of student characteristics in studying micro teaching-learning environments. Learning Environments Research, 9(3), 253-271.
Seymour, E. (2001). Tracking the processes of change in US undergraduate education in science, mathematics, engineering, and technology. Science Education, 86(1), 79-105.
Seymour, E., & Hewitt, N. M. (1997). Talking about leaving: Why undergraduates leave the sciences. Boulder, CO: Westview Press.
Shipman, H. L. & Duch, B. J. (2001). Problem-based learning in large and very large classes. In Duch, B. J., Groh, S. E., & Allen, D. E. (Eds.), The power of problem-based learning: A practical ‘how to’ for teaching undergraduate courses in any discipline. Sterling, VA: Stylus Publishing.
Smith, D. G. (1977). College classroom interactions and critical thinking. Journal of Educational Psychology, 69(2), 180-190.
Smith, D. G. (1981, March). Instruction and outcomes in an undergraduate setting. Paper presented at the meeting of the American Educational Research Association, Los Angeles, CA.
32
Smith, K. A., Sheppard, S. D., Johnson, D. W., & Johnson, R. T. (2005). Pedagogies of engagement: Classroom-based practices. Journal of Engineering Education, 94(1), 87-102.
Springer, L., Stanne, M. E., & Donovan, S. S. (1999). Effects of small-group learning on undergraduates in science, mathematics, engineering, and technology: A meta-analysis. Review of Educational Research, 69(1), 21-51.
Summers, M. F., & Hrabowski Iii, F. A. (2006). Diversity enhanced: Preparing minority scientists and engineers. Science, 311(5769), 1870-1871.
Tanner, K. & Allen, D. (2005). Approaches to biology teaching and learning: Understanding the wrong answers—teaching toward conceptual change. Cell Biology Education, 4(2), 112-117.
Terenzini, P. T., Theophilides, C., & Lorang, W. G. (1984). Influences on students’ perceptions of their academic skills development during college. The Journal of Higher Education, 55(5), 621-636.
Tobias, S. (1990). Stemming the science shortfall at college. In S. Tobias (Ed.), They’re not dumb, they’re different. Tucson, AZ: Research Corporation.
Tobias, S. (1992). Science education reform: What's wrong with the process? In S. Tobias (Ed.), Revitalizing undergraduate science: Why some things work and most don't (pp. 11-22). Tucson, AZ: Research Corporation.
Tsui, L. (1999). Courses and instruction affecting critical thinking. Research in Higher Education, 402), 185-200.
Tsui, L. (2002). Fostering critical thinking through effective pedagogy: Evidence from four institutional case studies. The Journal of Higher Education, 73(6), 740-763.
Vahala, M. E. & Winston, Jr., R. B. (1994). College classroom environments: Disciplinary and institutional-type differences and effects on academic achievement in introductory courses. Innovative Higher Education, 19(2), 99-122.
Waits, B. K., and Demana, F. (1988). Relationship between mathematics skills of entering students and their success in college. School Counselor35(4): 307-310.
Walberg, H. J. (1979). Educational environments and effects: Evaluation, policy, and productivity. Berkeley, CA: McCutchan.
Williams, W. M., Papierno, P. B., Makel, M. C., & Ceci, S. J. (2004). Thinking like a scientist about real-world problems: The Cornell Institute for Research on Children science education program. Journal of Applied Developmental Psychology, 25(1), 107-126.
33
Did
not
par
ticip
ate
in
rese
arch
focu
sed
prog
ram
Soug
ht tu
torin
g on
ca
mpu
s
Con
sist
ently
got
su
ppor
t nee
ded
Wor
ked
on a
pr
ofes
sor's
rese
arch
pr
ojec
t
Ext
erna
l Sup
port
/Sc
ienc
e E
xper
ienc
es
Did
not
par
ticip
ate
in
rese
arch
focu
sed
prog
ram
Soug
ht tu
torin
g on
ca
mpu
s
Con
sist
ently
got
su
ppor
t nee
ded
Wor
ked
on a
pr
ofes
sor's
rese
arch
pr
ojec
t
Ext
erna
l Sup
port
/Sc
ienc
e E
xper
ienc
es
Cra
mm
ed fo
r ex
ams
HPW
Eng
age
in
lab
activ
ities
Stud
ent E
ffor
t
Parti
cipa
ted
in a
st
udy
grou
p
Act
ivel
y pa
rtici
pate
d in
cla
ss d
iscu
ssio
ns
HPW
stud
ying
w
ith o
ther
st
uden
ts fr
om th
is
cour
se
Cra
mm
ed fo
r ex
ams
HPW
Eng
age
in
lab
activ
ities
Stud
ent E
ffor
t
Parti
cipa
ted
in a
st
udy
grou
p
Act
ivel
y pa
rtici
pate
d in
cla
ss d
iscu
ssio
ns
HPW
stud
ying
w
ith o
ther
st
uden
ts fr
om th
is
cour
se
Form
at w
as
prim
arily
lect
ure
Inst
ruct
or g
ave
feed
back
on
prog
ress
Form
at w
as
prim
arily
di
scus
sion
Form
at w
as p
rimar
ily
hand
s-on
Cou
rse
empl
oyed
gr
oup
lear
ning
ac
tiviti
es
Cou
rsew
ork
emph
asiz
ed
synt
hesi
zing
in
form
atio
n
Cou
rse
Peda
gogy
Form
at w
as
prim
arily
lect
ure
Inst
ruct
or g
ave
feed
back
on
prog
ress
Form
at w
as
prim
arily
di
scus
sion
Form
at w
as p
rimar
ily
hand
s-on
Cou
rse
empl
oyed
gr
oup
lear
ning
ac
tiviti
es
Cou
rsew
ork
emph
asiz
ed
synt
hesi
zing
in
form
atio
n
Cou
rse
Peda
gogy
Cri
tical
thin
king
disp
ositi
ons
Pret
ests
Ave
rage
H
S G
PA
in S
TEM
co
urse
s
Act
like
a
scie
ntis
tPr
e-te
st
Thin
k lik
e a
scie
ntis
tPr
e-te
st
Pret
ests
Ave
rage
H
S G
PA
in S
TEM
co
urse
s
Act
like
a
scie
ntis
tPr
e-te
st
Thin
k lik
e a
scie
ntis
tPr
e-te
st
Ave
rage
H
S G
PA
in S
TEM
co
urse
s
Act
like
a
scie
ntis
tPr
e-te
st
Thin
k lik
e a
scie
ntis
tPr
e-te
st
Gra
de
in
intro
co
urse
Act
like
a
scie
ntis
tPo
st-te
st
Thin
k lik
e a
scie
ntis
tPo
st-te
st
Act
like
a
scie
ntis
tPo
st-te
st
Thin
k lik
e a
scie
ntis
tPo
st-te
st
Out
com
es
Scie
nce
Maj
or
Fem
ale
Inco
me
UR
M S
tude
nt
Tuto
red
Stud
ent
in H
igh
Scho
ol
AP
Che
mis
try
Scor
e
Dem
ogra
phic
s
Fem
ale
Inco
me
UR
M S
tude
nt
Tuto
red
Stud
ent
in H
igh
Scho
ol
AP
Che
mis
try
Scor
e
Fem
ale
Inco
me
UR
M S
tude
nt
Tuto
red
Stud
ent
in H
igh
Scho
ol
AP
Che
mis
try
Scor
e
Dem
ogra
phic
s
Freq
uenc
y: F
elt
Ove
rwhe
lmed
Felt
com
petit
ion
in c
ours
e
Cou
rse
Lea
rnin
g E
nvir
onm
ent
Inte
ract
ed w
ith
clas
smat
es
Inst
ruct
or
prom
oted
co
oper
atio
n
Inst
ruct
or
enco
urag
ed
aski
ng q
uest
ions
Inst
ruct
ors v
alue
d st
uden
ts’d
iver
se
expe
rienc
es
Inst
ruct
or w
as
open
to v
iew
s be
side
s ow
n
Saw
real
-life
ap
plic
atio
n or
re
leva
nce
of le
arni
ng
Freq
uenc
y: F
elt
Ove
rwhe
lmed
Felt
com
petit
ion
in c
ours
e
Cou
rse
Lea
rnin
g E
nvir
onm
ent
Inte
ract
ed w
ith
clas
smat
es
Inst
ruct
or
prom
oted
co
oper
atio
n
Inst
ruct
or
enco
urag
ed
aski
ng q
uest
ions
Inst
ruct
ors v
alue
d st
uden
ts’d
iver
se
expe
rienc
es
Inst
ruct
or w
as
open
to v
iew
s be
side
s ow
n
Saw
real
-life
ap
plic
atio
n or
re
leva
nce
of le
arni
ng
Freq
uenc
y: F
elt
Ove
rwhe
lmed
Felt
com
petit
ion
in c
ours
e
Cou
rse
Lea
rnin
g E
nvir
onm
ent
Inte
ract
ed w
ith
clas
smat
es
Inst
ruct
or
prom
oted
co
oper
atio
n
Inst
ruct
or
enco
urag
ed
aski
ng q
uest
ions
Inst
ruct
ors v
alue
d st
uden
ts’d
iver
se
expe
rienc
es
Inst
ruct
or w
as
open
to v
iew
s be
side
s ow
n
Saw
real
-life
ap
plic
atio
n or
re
leva
nce
of le
arni
ng
Inqu
isiti
vene
ssSy
stem
atic
ityTr
uth-
seek
ing
Mat
urity
Ope
n-M
inde
dnes
sC
ritic
al T
hink
ing
Con
fiden
ceA
naly
ticity
Inqu
isiti
vene
ssSy
stem
atic
ityTr
uth-
seek
ing
Mat
urity
Ope
n-M
inde
dnes
sC
ritic
al T
hink
ing
Con
fiden
ceA
naly
ticity
Inqu
isiti
vene
ssSy
stem
atic
ityTr
uth-
seek
ing
Mat
urity
Ope
n-M
inde
dnes
sC
ritic
al T
hink
ing
Con
fiden
ceA
naly
ticity
34
-0.04
Fem
ale
Inco
me
UR
M S
tude
nt
Felt
Ove
rwhe
lmed
Tuto
red
Stud
ent
in H
igh
Scho
ol
Ave
rage
HS
GPA
in
STE
M c
ours
es
Felt
com
petit
ion
in c
ours
e
AP
Che
mis
try
Scor
e
Parti
cipa
ted
in h
igh
scho
ol re
sear
ch
prog
ram
BB
S M
ajor
Cra
mm
ed fo
r ex
ams
HPW
Eng
age
in
lab
activ
ities
Soug
ht tu
torin
g on
ca
mpu
s
Act
like
a
scie
ntis
tPr
e-te
st
Thin
k lik
e a
scie
ntis
tPr
e-te
st
Gra
de in
in
tro c
ours
e
Con
sist
ently
got
su
ppor
t nee
ded
HPW
Par
ticip
ated
in
prof
’sre
sear
ch
proj
ect i
n co
llege
Form
at w
as
prim
arily
lect
ure
Act
like
a
scie
ntis
tPo
st-te
st
Thin
k lik
e a
scie
ntis
tPo
st-te
stD
1
D2
D4
D3
Ext
erna
l Sup
port
/Sc
ienc
e E
xper
ienc
es
Stud
ent E
ffor
t
Cou
rse
Lea
rnin
g E
nvir
onm
ent
The
cour
se
empl
oyed
gro
up
activ
ities
Cou
rse
Peda
gogy
Dem
ogra
phic
sPr
etes
ts
Cri
tical
thin
king
disp
ositi
ons
Ope
n-M
inde
dnes
sC
ritic
al T
hink
ing
Con
fiden
ceA
naly
ticity
Figu
re 2
. Fin
al M
odel
.N
ote:
dot
ted
path
s rep
rese
nt p
aths
that
did
not
reac
h st
atis
tical
si
gnifi
canc
e. P
lain
-line
pat
hs w
ith b
olde
dco
effic
ient
s rep
rese
nt
stat
istic
ally
sign
ifica
nt p
aths
.0.05
-0.0
4
-0.05
0.23
-0.2
9
0.18
0.12
0.20-0.36
0.19
0.08
-0.1
4
0.13
0.11
0.02
-0.07
0.03
0.09
0.21
-0.23
0.14
0.19
0.05
0.05 0.57
0.56
-0.19
-0.14
0.33
0.13
-0.01
0.16
-0.24
0.07
0.15
-0.15
0.40-0.03
-0.26
-0.01
0.08
0.12
35
Table 1 Factor Loadings for the latent constructs in the model Factor Items* Loading Thinking like a scientist pre-test See connections between different areas of science and mathematics 0.78 Understand scientific concepts 0.77 Identify what is known and not known in a problem 0.72 Ask relevant questions 0.72 Draw a picture to represent a problem or concept 0.66 Make predictions based on existing knowledge 0.75 Come up with solutions and explain them to others 0.71 Investigate alternative solutions to a problem 0.71 Understand and translate scientific terminology into non-scientific language 0.74 Acting like a scientist pre-test Relate scientific concepts to real-world problems 0.77 Synthesize or comprehend several sources of information 0.85 Conduct an experiment 0.64 Look up scientific research articles and resources 0.60 Memorize large quantities of information 0.61 Thinking like a scientist post-test See connections between different areas of science and mathematics 0.73 Understand scientific concepts 0.80 Identify what is known and not known in a problem 0.76 Ask relevant questions 0.72 Draw a picture to represent a problem or concept 0.58 Make predictions based on existing knowledge 0.82 Come up with solutions and explain them to others 0.81 Investigate alternative solutions to a problem 0.73 Understand and translate scientific terminology into non-scientific language 0.72 Acting like a scientist post-test Relate scientific concepts to real-world problems 0.80 Synthesize or comprehend several sources of information 0.78 Conduct an experiment 0.71 Look up scientific research articles and resources 0.62 Memorize large quantities of information 0.53 Measurement model fit statistics: Χ2=300.69 (305, N=255), NNFI = 0.98, CFI = 0.98, RMSEA = 0.03, reliability coefficient = 0.82.
*Note: All items were asked as part of a questions stem that read, “Rate your ability in the following areas as it pertains to your academic learning in the sciences.” Response options were Major Strength (5), Above Average (4), Average (3), Below Average (2), Major Weakness (1).
36
Table 2 Parameter estimates for direct effects in the structural model Endogenous variable Predictor variables b B S.E. Sig R2 Average high school math/science GPA 0.06 Parental income 0.01 0.05 0.01 Frequency: Tutored another student 0.13 0.23 0.04 ** Gender: Female -0.04 -0.05 0.05 Race: Underrepresented racial minority student -0.04 -0.04 0.05 Acting like a scientist pre-test 0.12 AP Chemistry score 0.07 0.18 0.03 * Gender: Female -0.43 -0.29 0.11 ** Thinking like a scientist pre-test 0.18 AP Chemistry score 0.07 0.20 0.02 * Frequency: Tutored another student 0.11 0.12 0.04 * Gender: Female -0.47 -0.36 0.09 ** Course grade 0.22 Average high school math/science GPA 1.88 0.19 0.62 * Thinking like a scientist post-test 0.58 0.08 1.79 Acting like a scientist post-test -0.94 -0.14 1.74 Participated in a high school research program 1.80 0.13 0.84 * CCTDI Openmindedness 0.01 0.02 0.04 CCTDI Analyticity 0.09 0.11 0.07 CCTDI Critical thinking confidence -0.04 -0.07 0.05 Course pedagogy: primarily lecture 0.18 0.03 0.43 Course pedagogy: group work employed 0.85 0.21 0.30 * Opinion: felt competition among students 0.37 .0.09 0.28 Opinion: felt overwhelmed by course expectations -0.96 -0.23 0.30 ** Activity: Sought tutoring 0.49 0.14 0.23 * Activity: Crammed for exams 0.78 0.19 0.26 ** Hours per week spent in lab 0.10 0.05 0.16 Activity: Participated in college research program 0.07 0.05 0.08 Major: biomedical or behavioral science -0.30 -0.04 0.51 Thinking like a scientist post-test 0.61 Thinking like a scientist pre-test 0.47 0.57 0.06 *** CCTDI Openmindedness -0.02 -0.19 0.01 ** CCTDI Analyticity 0.02 0.14 0.01 * CCTDI Critical thinking confidence 0.02 0.33 0.01 ** Course pedagogy: primarily lecture 0.12 0.13 0.05 * Course pedagogy: group work employed -0.02 -0.03 0.03 Opinion: felt overwhelmed by course expectations -0.14 -0.24 0.03 ** Activity: Participated in college research program 0.01 0.07 0.01 Major: biomedical or behavioral science 0.17 0.15 0.06 *
N=255; χ2 = 1,285.69 (1,050, p< 0.001); NNFI = 0.91, CFI = 0.92, RMSEA = 0.03 Note: * p < 0.05, ** p < 0.01, *** p < 0.001
37
Table 2 (Continued) Parameter estimates for direct effects in the structural model
N=255; χ2 = 1,285.69 (1,050, p< 0.001); NNFI = 0.91, CFI = 0.92, RMSEA = 0.03 Note: * p < 0.05, ** p < 0.01, *** p < 0.001
Endogenous variable Predictor variables b B S.E. Sig R2 Acting like a scientist post-test 0.60 Acting like a scientist pre-test 0.43 0.56 0.06 *** CCTDI Openmindedness -0.01 -0.15 0.01 * CCTDI Critical thinking confidence 0.03 0.40 0.01 ** Course pedagogy: primarily lecture 0.15 0.16 0.06 * Course pedagogy: group work employed -0.01 -0.01 0.04 Opinion: felt overwhelmed by course expectations -0.16 -0.26 0.03 ** Course pedagogy: received the support necessary -0.01 -0.01 0.03 Activity: Participated in college research program 0.02 0.08 0.01 Major: biomedical or behavioral science 0.14 0.12 0.06 *
38
Table 3 Parameter estimates of indirect effects Outcome Predictor variable b B S.E. Sig. Course grade Acting like a scientist pre-test -0.77 -0.14 0.72 Thinking like a scientist pre-test 0.69 0.11 0.79 Parental income 0.01 0.01 0.02 AP chemistry score -0.01 0.00 0.02 CCTDI: Openmindedness 0.00 0.00 0.01 CCTDI: Analyticity 0.02 0.03 0.03 CCTDI: Critical thinking self-confidence -0.02 -0.04 0.02 Activity: Tutored another student in high school 0.31 0.06 0.13 * Course pedagogy: Primarily lecture -0.10 0.02 0.11 Course pedagogy: group work encouraged -0.01 -0.01 0.05 Opinion: felt overwhelmed by course expectations 0.08 0.02 0.09 Course pedagogy: received the support necessary 0.02 0.00 0.06 Activity: Participated in college research program -0.01 -0.01 0.02 Gender: Female -0.07 -0.01 0.16 Major: Biomedical or behavioral science 0.00 0.00 0.12 Race: Underrepresented racial minority -0.06 -0.01 0.10 Thinking like a scientist post-test AP chemistry score 0.03 0.11 0.01 * Activity: Tutored another student in high school 0.05 0.07 0.02 * Gender: Female -0.22 -0.21 0.05 * Acting like a scientist post-test AP chemistry score 0.03 0.10 0.01 * Gender: Female -0.18 -0.16 0.05 *
Note: * p < 0.05, ** p < 0.01, *** p < 0.001
39
Appendix A: Variables in Initial Model and Coding Variable Coding Demographics Income 1=Less than $20K to 8=More than $200K URM Student 1 = Black, Chicano or Native American; 0 = White/Asian Female 1 = Female, 0 = Male AP Chemistry Score 1 to 5 Tutored Another student in High School 1 = Never, 2 = Occasionally, 3 = Frequently BBS Major 1 = Biomedical or Behavioral Science Major, 0 = Other/Undecided Pre-Tests Average HS GPA in STEM Courses 1 = D/F, 2 = C, 3 = B, 4 = A External Support/Science Experiences Participated in a research-focused program during high school 1 = No, 0 = Yes Sought tutoring on campus 1=Never, 2 = At least once, 3 = Occasionally, 4=Almost always Participated in a research project
Working on a professor's research project (Hours per week during term) 1 = Zero hours to 13=More than 10 hours
Consistently received the support needed to do well 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree Student Effort Crammed For Exams 1=Never, 2 = At least once, 3 = Occasionally, 4=Almost always Actively participated in class discussions 1=Never, 2 = At least once, 3 = Occasionally, 4=Almost always Participated in a study group 1=Never, 2 = At least once, 3 = Occasionally, 4=Almost always
Studied with other students from this course (Hours per week during term) 1 = Zero hours to 13=More than 10 hours
Engaged in lab activities (Hours per week during term) 1 = Zero hours to 13=More than 10 hours Course Pedagogy The format of this course was primarily lecture 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree The format of this course was primarily discussion 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree The format of this course was primarily hands-on activity 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree The course employed group activities to foster learning 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree
40
The coursework emphasized synthesizing and organizing ideas and information 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree
Instructor gave students feedback on their performance or progress in the course 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree
Course Learning Environment I frequently experienced a high level of competition among students 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree I felt overwhelmed by what was expected of me for this course 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree Instructor appeared open to viewpoints besides his/her own 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree Instructor valued students' diverse life experiences 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree I saw the real-life application or relevance of what I learned 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree Instructor promoted cooperation among students 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree Instructor encouraged students to ask questions 1 = Strongly disagree, 2 = Disagree, 3 = Agree, 4 = Strongly agree Interacted with classmates 1=Never, 2 = At least once, 3 = Occasionally, 4=Almost always Critical Thinking Dispositions (See Facione, Sanchez, and Facione (1993) for more details) Inquisitiveness 10 to 60 Systematicity 10 to 60 Truth-seeking 10 to 60 Maturity 10 to 60 Open-Mindedness 10 to 60 Critical Thinking Confidence 10 to 60 Analyticity 10 to 60 Outcomes
Grade in intro Course 9 = A+, 8 = A, 7 = A-, 6 = B+, 5 = B, 4 = B-, 3 = C+, 2 = C, 1 = C- or below
41
Appendix B: Descriptive statistics for variables in the model
Mean S.D. Min. Max Parental income 5.03 2.19 1 8 AP chemistry score 2.05 1.83 0 6 Openmindedness 41.29 6.27 22 58 Analyticity 43.00 4.99 28 60 Critical thinking self-confidence 42.43 7.06 20 60 Course grade 6.78 3.80 0 12 Average grade in high school math and science courses 3.71 0.37 1 4 Participated in a pre-college research program 0.08 0.27 0 1 Tutored another stuent in high school 1.98 0.66 1 3 Pedagogy: course was primarily lecture 3.67 0.56 1 4 Pedagogy: course encouraged group activities 1.98 0.93 1 4 Opinion: Competition among students for grades 2.90 0.91 1 4 Opinion: Felt overwhelmed by course expectations 2.40 0.93 1 4 Opinion: Consistently received the support I needed 2.80 0.60 1 4 Sought out tutoring from a campus office or program 3.27 1.06 1 4 Crammed for exams 2.03 0.90 1 4 Engaging in laboratory activities 1.88 1.66 1 8 Participating in a science research program in college 2.06 2.76 1 13 Gender: Female 0.75 0.48 0 1 Major: Biomedical or behavioral science 0.23 0.48 0 1 Race: underrepresented racial minority 0.29 0.45 0 1 Source: Descriptive analysis of pre- and post-survey data.
42
Appendix C (continued): Correlation table for all variables in the model 1 2 3 4 5 6 7 8 9 10 11 12 13
1 See connections between different areas of science and mathematics (pre-test) 1.002 Understand scientific concepts (pre-test) 0.69 1.003 Relate scientific concepts to real-world problems (pre-test) 0.54 0.68 1.004 Synthesize or comprehend several sources of information (pre-test) 0.57 0.61 0.66 1.005 Identify what is known and not known in a problem (pre-test) 0.57 0.58 0.49 0.50 1.006 Ask relevant questions (pre-test) 0.56 0.57 0.52 0.51 0.62 1.007 Draw a picture to represent a problem or concept (pre-test) 0.48 0.46 0.45 0.47 0.49 0.62 1.008 Make predictions based on existing knowledge (pre-test) 0.60 0.58 0.53 0.57 0.53 0.55 0.60 1.009 Conduct an experiment (pre-test) 0.47 0.52 0.48 0.55 0.44 0.42 0.44 0.64 1.00
10 Come up with solutions and explain them to others (pre-test) 0.58 0.50 0.43 0.60 0.44 0.55 0.50 0.54 0.47 1.0011 Investigate alternative solutions to a problem (pre-test) 0.51 0.49 0.52 0.56 0.45 0.55 0.51 0.60 0.50 0.67 1.0012 Look up scientific research articles and resources (pre-test) 0.34 0.36 0.46 0.51 0.31 0.32 0.28 0.38 0.38 0.22 0.45 1.0013 Memorize large quantities of information (pre-test) 0.44 0.36 0.44 0.53 0.35 0.40 0.28 0.43 0.34 0.37 0.33 0.40 1.0014 Understand and translate scientific terminology into non-scientific language (pre-test) 0.58 0.54 0.55 0.61 0.53 0.49 0.49 0.50 0.43 0.57 0.50 0.34 0.4915 Parental income -0.01 -0.05 -0.11 0.04 -0.13 -0.11 -0.15 -0.09 -0.13 -0.02 -0.12 -0.10 -0.0316 AP chemistry score 0.17 0.15 0.08 0.20 0.17 0.12 0.04 0.07 0.13 0.14 0.10 0.02 0.0817 See connections between different areas of science and mathematics (post-test) 0.53 0.55 0.44 0.44 0.47 0.47 0.38 0.45 0.39 0.30 0.35 0.33 0.3418 Understand scientific concepts (post-test) 0.59 0.53 0.50 0.48 0.44 0.49 0.36 0.42 0.37 0.36 0.44 0.30 0.4219 Relate scientific concepts to real-world problems (post-test) 0.45 0.47 0.54 0.49 0.42 0.41 0.31 0.41 0.40 0.32 0.43 0.33 0.4020 Synthesize or comprehend several sources of information (post-test) 0.44 0.43 0.47 0.48 0.40 0.44 0.27 0.40 0.39 0.34 0.43 0.30 0.3521 Identify what is known and not known in a problem (post-test) 0.52 0.46 0.48 0.44 0.45 0.47 0.39 0.42 0.38 0.36 0.39 0.28 0.4122 Ask relevant questions (post-test) 0.40 0.39 0.39 0.40 0.36 0.52 0.34 0.42 0.33 0.33 0.37 0.23 0.3423 Draw a picture to represent a problem or concept (post-test) 0.43 0.29 0.25 0.36 0.30 0.38 0.52 0.32 0.24 0.33 0.25 0.21 0.2724 Make predictions based on existing knowledge (post-test) 0.49 0.46 0.50 0.52 0.35 0.48 0.41 0.53 0.50 0.45 0.48 0.32 0.4325 Conduct an experiment (post-test) 0.43 0.47 0.49 0.46 0.39 0.41 0.38 0.56 0.59 0.39 0.48 0.44 0.3926 Come up with solutions and explain them to others (post-test) 0.51 0.50 0.45 0.46 0.40 0.42 0.39 0.41 0.40 0.47 0.43 0.30 0.2727 Investigate alternative solutions to a problem (post-test) 0.38 0.38 0.40 0.45 0.37 0.44 0.42 0.51 0.43 0.36 0.53 0.39 0.3328 Look up scientific research articles and resources (post-test) 0.31 0.24 0.34 0.33 0.27 0.30 0.26 0.34 0.29 0.09 0.28 0.65 0.3029 Memorize large quantities of information (post-test) 0.33 0.26 0.34 0.30 0.20 0.27 0.19 0.32 0.26 0.25 0.26 0.33 0.6030 Understand and translate scientific terminology into non-scientific language (post-test) 0.42 0.32 0.33 0.37 0.25 0.27 0.27 0.33 0.31 0.31 0.35 0.25 0.3031 Openmindedness -0.06 -0.04 -0.05 0.04 0.00 -0.05 -0.13 -0.02 -0.02 0.02 -0.04 -0.08 -0.1432 Analyticity 0.28 0.32 0.34 0.30 0.25 0.26 0.24 0.34 0.33 0.30 0.32 0.17 0.1233 Critical thinking self-confidence 0.31 0.30 0.44 0.46 0.32 0.34 0.29 0.41 0.40 0.38 0.47 0.29 0.2534 Course grade 0.11 0.11 0.11 0.07 0.18 0.04 -0.07 0.00 -0.01 0.04 0.06 0.01 0.0535 Average grade in high school math and science courses 0.15 0.11 0.01 0.13 0.07 0.01 -0.03 0.01 0.01 0.07 0.07 0.11 0.0736 Participated in a pre-college research program -0.04 -0.08 -0.03 0.08 0.01 -0.07 0.04 -0.05 0.01 -0.03 0.01 0.10 -0.0637 Tutored another stuent in high school 0.15 0.14 0.04 0.15 0.21 0.14 0.19 0.05 0.08 0.18 0.22 0.17 -0.0738 Pedagogy: course was primarily lecture 0.14 0.13 0.07 0.13 0.05 -0.01 0.02 0.05 0.04 0.05 0.07 0.10 0.0939 Pedagogy: course encouraged group activities -0.02 0.01 0.07 -0.07 0.05 0.07 0.05 0.06 0.03 0.01 0.07 0.04 -0.0140 Opinion: Competition among students for grades -0.08 -0.08 -0.08 -0.09 0.02 0.11 0.04 0.03 0.03 0.09 0.05 -0.08 0.0941 Opinion: Felt overwhelmed by course expectations -0.21 -0.17 -0.09 -0.11 -0.19 -0.12 -0.04 -0.09 0.00 -0.04 -0.08 -0.09 -0.0442 Opinion: Consistently received the support I needed 0.18 0.14 0.17 0.12 0.18 0.12 0.11 0.11 0.13 0.13 0.09 0.13 0.1143 Sought out tutoring from a campus office or program 0.15 0.04 -0.08 0.01 0.03 -0.07 -0.05 -0.03 -0.08 -0.06 -0.06 -0.12 0.0044 Crammed for exams 0.08 0.13 0.09 0.09 0.14 0.06 0.05 -0.08 -0.04 0.01 0.04 0.12 0.0545 Engaging in laboratory activities -0.05 -0.01 0.05 0.03 0.01 0.07 0.02 0.12 0.10 -0.06 0.00 0.21 0.1046 Participating in a science research program in college 0.11 0.11 0.12 0.12 0.03 0.08 0.08 0.08 0.15 0.08 0.18 0.27 0.0647 Gender: Female -0.27 -0.22 -0.22 -0.26 -0.17 -0.15 -0.25 -0.35 -0.18 -0.36 -0.35 -0.11 -0.2248 Major: Biomedical or behavioral science -0.12 -0.17 -0.07 -0.05 -0.08 -0.02 -0.11 0.02 -0.02 -0.09 -0.10 0.08 0.1049 Race: underrepresented racial minority 0.01 0.08 -0.03 0.03 0.00 -0.01 0.05 0.03 0.16 -0.02 0.06 -0.01 0.03
Appendix C: Correlation table for all variables in the model
43
14 15 16 17 18 19 20 21 22 23 24 25 261 See connections between different areas of science and mathematics (pre-test)2 Understand scientific concepts (pre-test)3 Relate scientific concepts to real-world problems (pre-test)4 Synthesize or comprehend several sources of information (pre-test)5 Identify what is known and not known in a problem (pre-test)6 Ask relevant questions (pre-test)7 Draw a picture to represent a problem or concept (pre-test)8 Make predictions based on existing knowledge (pre-test)9 Conduct an experiment (pre-test)
10 Come up with solutions and explain them to others (pre-test)11 Investigate alternative solutions to a problem (pre-test)12 Look up scientific research articles and resources (pre-test)13 Memorize large quantities of information (pre-test)14 Understand and translate scientific terminology into non-scientific language (pre-test) 1.0015 Parental income -0.03 1.0016 AP chemistry score 0.14 0.02 1.0017 See connections between different areas of science and mathematics (post-test) 0.40 -0.12 0.11 1.0018 Understand scientific concepts (post-test) 0.43 -0.12 0.12 0.72 1.0019 Relate scientific concepts to real-world problems (post-test) 0.39 -0.03 0.09 0.67 0.71 1.0020 Synthesize or comprehend several sources of information (post-test) 0.36 -0.03 0.07 0.52 0.67 0.66 1.0021 Identify what is known and not known in a problem (post-test) 0.45 -0.10 0.09 0.56 0.60 0.56 0.58 1.0022 Ask relevant questions (post-test) 0.30 -0.07 -0.01 0.51 0.56 0.50 0.52 0.59 1.0023 Draw a picture to represent a problem or concept (post-test) 0.33 -0.07 0.06 0.43 0.48 0.40 0.40 0.45 0.48 1.0024 Make predictions based on existing knowledge (post-test) 0.43 -0.13 0.10 0.55 0.64 0.59 0.65 0.60 0.60 0.50 1.0025 Conduct an experiment (post-test) 0.36 -0.11 -0.08 0.46 0.52 0.57 0.53 0.46 0.43 0.33 0.60 1.0026 Come up with solutions and explain them to others (post-test) 0.43 -0.09 0.08 0.60 0.63 0.57 0.61 0.56 0.58 0.43 0.66 0.59 1.0027 Investigate alternative solutions to a problem (post-test) 0.41 -0.16 0.03 0.52 0.51 0.54 0.53 0.51 0.55 0.32 0.62 0.65 0.7128 Look up scientific research articles and resources (post-test) 0.22 -0.14 -0.05 0.41 0.41 0.51 0.45 0.37 0.40 0.34 0.43 0.55 0.4629 Memorize large quantities of information (post-test) 0.26 -0.11 -0.12 0.39 0.46 0.42 0.38 0.42 0.47 0.35 0.46 0.37 0.3930 Understand and translate scientific terminology into non-scientific language (post-test) 0.53 -0.04 -0.01 0.45 0.56 0.56 0.50 0.58 0.49 0.40 0.49 0.48 0.6031 Openmindedness 0.01 0.14 0.02 -0.15 -0.13 -0.09 0.03 -0.02 0.02 -0.07 -0.11 -0.16 -0.0632 Analyticity 0.23 -0.04 0.05 0.34 0.36 0.34 0.31 0.27 0.38 0.23 0.38 0.25 0.4033 Critical thinking self-confidence 0.34 -0.02 0.03 0.41 0.45 0.46 0.49 0.34 0.41 0.32 0.51 0.42 0.5334 Course grade 0.04 0.10 0.10 0.12 0.10 0.09 0.03 0.08 0.03 -0.02 0.04 -0.06 0.0235 Average grade in high school math and science courses 0.05 0.06 0.22 0.11 0.17 0.10 0.10 0.01 0.11 0.06 0.13 -0.04 0.1136 Participated in a pre-college research program 0.08 0.08 0.15 -0.01 -0.13 0.07 -0.10 -0.05 -0.11 -0.10 -0.07 -0.09 -0.1137 Tutored another stuent in high school 0.21 -0.02 0.07 0.14 0.08 0.07 0.09 0.08 0.12 0.13 0.04 0.05 0.1838 Pedagogy: course was primarily lecture 0.08 0.10 0.01 0.04 0.16 0.17 0.16 0.15 0.08 0.09 0.09 0.12 0.0739 Pedagogy: course encouraged group activities -0.06 -0.17 -0.04 0.15 0.00 0.04 -0.03 0.04 -0.05 -0.10 0.12 0.11 0.0340 Opinion: Competition among students for grades -0.06 0.01 0.00 -0.04 -0.02 0.04 0.02 0.05 0.00 -0.05 0.01 0.04 -0.1041 Opinion: Felt overwhelmed by course expectations -0.14 0.01 -0.07 -0.19 -0.21 -0.20 -0.17 -0.19 -0.25 -0.14 -0.19 -0.11 -0.2142 Opinion: Consistently received the support I needed 0.17 0.00 0.08 0.23 0.22 0.12 0.13 0.19 0.11 0.14 0.16 0.14 0.2143 Sought out tutoring from a campus office or program 0.09 0.14 0.15 0.04 0.07 0.10 0.03 0.15 0.00 0.18 0.03 -0.06 -0.0644 Crammed for exams 0.13 -0.04 -0.05 0.07 0.10 0.11 0.09 0.07 0.07 0.06 -0.04 0.01 0.1045 Engaging in laboratory activities -0.07 -0.16 -0.10 0.15 0.02 0.05 -0.03 0.13 0.09 0.00 0.13 0.20 -0.0146 Participating in a science research program in college -0.01 0.01 0.03 0.20 0.19 0.21 0.17 0.12 0.08 0.16 0.15 0.11 0.1547 Gender: Female -0.23 0.03 0.01 -0.19 -0.19 -0.25 -0.14 -0.18 -0.26 -0.14 -0.28 -0.21 -0.2648 Major: Biomedical or behavioral science -0.09 -0.09 -0.24 -0.08 -0.07 -0.05 0.00 0.04 0.05 0.03 0.02 0.06 -0.0749 Race: underrepresented racial minority 0.04 -0.19 -0.12 0.12 0.07 0.05 -0.10 0.01 0.08 0.04 0.11 0.14 0.08
Appendix C (continued): Correlation table for all variables in the model
44
27 28 29 30 31 32 33 34 35 36 37 38 391 See connections between different areas of science and mathematics (pre-test)2 Understand scientific concepts (pre-test)3 Relate scientific concepts to real-world problems (pre-test)4 Synthesize or comprehend several sources of information (pre-test)5 Identify what is known and not known in a problem (pre-test)6 Ask relevant questions (pre-test)7 Draw a picture to represent a problem or concept (pre-test)8 Make predictions based on existing knowledge (pre-test)9 Conduct an experiment (pre-test)
10 Come up with solutions and explain them to others (pre-test)11 Investigate alternative solutions to a problem (pre-test)12 Look up scientific research articles and resources (pre-test)13 Memorize large quantities of information (pre-test)14 Understand and translate scientific terminology into non-scientific language (pre-test)15 Parental income16 AP chemistry score17 See connections between different areas of science and mathematics (post-test)18 Understand scientific concepts (post-test)19 Relate scientific concepts to real-world problems (post-test)20 Synthesize or comprehend several sources of information (post-test)21 Identify what is known and not known in a problem (post-test)22 Ask relevant questions (post-test)23 Draw a picture to represent a problem or concept (post-test)24 Make predictions based on existing knowledge (post-test)25 Conduct an experiment (post-test)26 Come up with solutions and explain them to others (post-test)27 Investigate alternative solutions to a problem (post-test) 1.0028 Look up scientific research articles and resources (post-test) 0.51 1.0029 Memorize large quantities of information (post-test) 0.47 0.44 1.0030 Understand and translate scientific terminology into non-scientific language (post-test) 0.51 0.44 0.43 1.0031 Openmindedness -0.11 -0.07 -0.12 0.00 1.0032 Analyticity 0.23 0.16 0.18 0.30 0.33 1.0033 Critical thinking self-confidence 0.46 0.32 0.29 0.37 0.07 0.59 1.0034 Course grade 0.06 -0.04 0.14 0.09 0.07 0.10 -0.02 1.0035 Average grade in high school math and science courses 0.16 0.10 0.07 0.10 0.04 0.03 0.02 0.27 1.0036 Participated in a pre-college research program -0.03 0.01 -0.09 0.06 0.08 -0.03 0.03 0.16 0.11 1.0037 Tutored another stuent in high school 0.12 0.09 0.00 0.06 0.14 0.10 0.13 0.02 0.22 -0.05 1.0038 Pedagogy: course was primarily lecture 0.07 0.04 0.17 0.20 0.06 0.02 -0.02 0.01 0.00 -0.06 0.01 1.0039 Pedagogy: course encouraged group activities 0.11 0.02 -0.02 -0.15 -0.30 -0.07 0.14 0.10 -0.05 0.01 0.01 -0.25 1.0040 Opinion: Competition among students for grades 0.01 -0.11 0.00 -0.14 -0.11 -0.03 0.02 -0.09 -0.15 0.11 -0.07 0.09 0.1741 Opinion: Felt overwhelmed by course expectations -0.18 -0.21 -0.16 -0.22 -0.19 -0.17 0.01 -0.29 -0.22 -0.03 -0.16 0.03 0.1942 Opinion: Consistently received the support I needed 0.10 0.15 0.14 0.06 -0.09 0.08 0.17 0.07 0.03 0.04 0.06 -0.08 0.3143 Sought out tutoring from a campus office or program -0.10 -0.05 0.00 0.12 0.14 0.02 -0.12 0.17 0.07 -0.07 -0.07 0.18 -0.2444 Crammed for exams 0.07 0.10 0.02 0.09 0.02 -0.06 -0.05 0.22 0.12 -0.10 0.12 0.01 -0.0245 Engaging in laboratory activities 0.16 0.20 0.18 0.01 -0.19 0.05 0.06 0.11 0.00 -0.07 -0.04 -0.12 0.4646 Participating in a science research program in college 0.14 0.25 0.14 0.13 -0.05 0.01 0.14 0.10 0.18 0.01 0.06 0.09 -0.0247 Gender: Female -0.30 -0.08 -0.25 -0.21 0.08 -0.20 -0.21 0.01 -0.06 -0.06 0.04 -0.05 0.0148 Major: Biomedical or behavioral science 0.08 0.06 0.22 -0.02 -0.07 -0.22 -0.06 -0.12 -0.22 0.11 -0.15 -0.01 0.1849 Race: underrepresented racial minority 0.12 0.07 0.11 0.10 -0.07 0.15 0.09 -0.14 -0.04 0.01 0.07 -0.06 0.10
Appendix C (continued): Correlation table for all variables in the model
45
Appendix C (continued): Correlation table for all variables in the model
40 41 42 43 44 45 46 47 48 491 See connections between different areas of science and mathematics (pre-test)2 Understand scientific concepts (pre-test)3 Relate scientific concepts to real-world problems (pre-test)4 Synthesize or comprehend several sources of information (pre-test)5 Identify what is known and not known in a problem (pre-test)6 Ask relevant questions (pre-test)7 Draw a picture to represent a problem or concept (pre-test)8 Make predictions based on existing knowledge (pre-test)9 Conduct an experiment (pre-test)
10 Come up with solutions and explain them to others (pre-test)11 Investigate alternative solutions to a problem (pre-test)12 Look up scientific research articles and resources (pre-test)13 Memorize large quantities of information (pre-test)14 Understand and translate scientific terminology into non-scientific language (pre-test)15 Parental income16 AP chemistry score17 See connections between different areas of science and mathematics (post-test)18 Understand scientific concepts (post-test)19 Relate scientific concepts to real-world problems (post-test)20 Synthesize or comprehend several sources of information (post-test)21 Identify what is known and not known in a problem (post-test)22 Ask relevant questions (post-test)23 Draw a picture to represent a problem or concept (post-test)24 Make predictions based on existing knowledge (post-test)25 Conduct an experiment (post-test)26 Come up with solutions and explain them to others (post-test)27 Investigate alternative solutions to a problem (post-test)28 Look up scientific research articles and resources (post-test)29 Memorize large quantities of information (post-test)30 Understand and translate scientific terminology into non-scientific language (post-test)31 Openmindedness32 Analyticity33 Critical thinking self-confidence34 Course grade35 Average grade in high school math and science courses36 Participated in a pre-college research program37 Tutored another stuent in high school38 Pedagogy: course was primarily lecture39 Pedagogy: course encouraged group activities40 Opinion: Competition among students for grades 1.0041 Opinion: Felt overwhelmed by course expectations 0.44 1.0042 Opinion: Consistently received the support I needed 0.05 -0.13 1.0043 Sought out tutoring from a campus office or program -0.15 -0.28 -0.11 1.0044 Crammed for exams -0.15 -0.15 0.10 -0.08 1.0045 Engaging in laboratory activities 0.05 -0.02 0.19 -0.17 -0.05 1.0046 Participating in a science research program in college -0.10 -0.09 0.08 0.04 0.08 -0.01 1.0047 Gender: Female 0.06 0.09 0.04 0.05 0.05 0.07 -0.13 1.0048 Major: Biomedical or behavioral science 0.22 0.17 0.05 -0.18 -0.09 0.18 -0.09 0.10 1.0049 Race: underrepresented racial minority -0.12 -0.08 -0.01 0.09 -0.07 0.14 -0.05 -0.04 -0.01 1.00