prevalence of mixed methods research in mathematics education · 2013-08-02 · for research in...

The Mathematics Educator 2012 Vol. 22, No. 1, 84–113

Prevalence of Mixed Methods Research in

Mathematics Education

Amanda Ross and Anthony J. Onwuegbuzie

In wake of federal legislation such as the No Child Left Behind Act of 2001 that have called for “scientifically based research in education,” this study examined the possible trends in mixed methods research articles published in 2 peer-reviewed mathematics education journals (n = 87) from 2002 to 2006. The study also illustrates how the integration of quantitative and qualitative research enhances the findings in mathematics education research. Mixed methods research accounted for 31% of empirical articles published in the 2 journals, with a 10% decrease over the 5-year span. Mixed methods research articles were slightly more qualitatively oriented, with 59% constituting such a design. Topics involving mathematical thought processes, problem solving, mental actions, behaviors, and other occurrences related to mathematical understanding were examined in these studies. Qualitative and quantitative data were used to complement one another and reveal relationships between observations and mathematical achievement.

In recent years there have been renewed calls in the United States for reform in mathematics education research as a result of federal legislation such as the No Child Left Behind (NCLB) Act of 2001 (NCLB, 2001) and the Education Sciences Reform Act (ESRA) of 2002 (ESRA, 2002) that have called for

Amanda A. Ross is an educational consultant and president of A. A. Ross Consulting and Research, LLC. She currently writes and reviews mathematics curriculum and assessment items, creates instructional design components, performs standards alignments, writes preparatory standardized test materials, writes grant proposals, and serves as external

evaluator.

Anthony Onwuegbuzie is a professor in the Department of Educational Leadership and Counseling at Sam Houston State University, where he teaches doctoral-level courses in qualitative research, quantitative research, and mixed research. With a h-index of 47, and writing extensively on qualitative, quantitative, and mixed methodological topics, he has had published more than 340 works, including more than 270 journal articles, 50 book chapters, and 2 books.

Prevalence of Mixed Methods

85

“scientifically based research in education.” In particular, much of the ensuing debate has revolved around whether or not the purpose of research should be to determine what works. Moreover, guidelines and review procedures of the Institute of Education Sciences (U.S. Department of Education) and its influential What Works Clearinghouse (see www.whatworks.ed.gov) have led some researchers and policymakers to imply that randomized controlled trials represent the gold standard for research and that designs associated with qualitative research and mixed methods research are inferior to quantitative research designs in general and experimental research designs in particular (Patton, 2006).

The current debate in the United States regarding the gold standard is in stark contrast to the controversy that prevailed 40 years ago when calls abounded to make mathematics education research more scientific (Lester, 2005). Lester and Lambdin (2003) noted that the use of experimental and quasi-experimental techniques in mathematics education research during that time was criticized as being inappropriate for addressing questions of what works. Advocating the need for a journal devoted solely to mathematics education research, Joe Scandura (1967), a prominent researcher in the United States during the 1960s and 1970s, concluded:

[M]any thoughtful people are critical of the quality of research in mathematics education. They look at tables of statistical data and they say “So what!” They feel that vital questions go unanswered while means, standard deviations, and t-tests pile up. (p. iii)

Over the last several decades, mathematics education researchers and policy makers have struggled to agree upon what represents the most appropriate research approach to use for research in mathematics education, leading to a form of research identity crisis. This struggle has been complicated further by federal legislation such as NCLB and ESRA wherein “scientifically based research in education” has been a contested phrase in many education fields (cf. McLafferty, Slate, & Onwuegbuzie, 2010). Indeed, little is known about the effect of this federal legislation on articles published in mathematics education research journals. In particular, little

Amanda Ross & Anthony J. Onwuegbuzie

86

information appears to exist regarding the extent to which the published research in mathematics education journals includes what is commonly known as mixed methods (or mixed) research (Johnson & Onwuegbuzie, 2004).

For the purposes of this paper, we view qualitative research, quantitative research, and mixed methods research as representing the three major research or methodological paradigms. We define qualitative research as relying on the collection, analysis, and interpretation of non-numeric data that naturally occur (Lincoln & Guba, 1985) from one or more of the sources identified by Leech and Onwuegbuzie (2008): talk, observations, drawing/photographs/videos, and documents. We define quantitative research as involving the collection, analysis, and interpretation of numeric data, with the goals of describing, explaining, and predicting phenomena. We follow Johnson, Onwuegbuzie, and Turner (2007) in their definition of mixed methods research:

Mixed methods research is an intellectual and practical synthesis based on qualitative and quantitative research…. It recognizes the importance of traditional quantitative and qualitative research but also offers a powerful third paradigm choice that often will provide the most informative, complete, balanced, and useful research results. Mixed methods research is the research paradigm that (a) partners with the philosophy of pragmatism in one of its forms (left, right, middle); (b) follows the logic of mixed methods research (including the logic of the fundamental principle and any other useful logics imported from qualitative or quantitative research that are helpful for producing defensible and usable research findings); (c) relies on qualitative and quantitative viewpoints, data collection, analysis, and inference techniques combined according to the logic of mixed methods research to address one’s research question(s); and (d) is cognizant, appreciative, and inclusive of local and broader sociopolitical realities, resources, and needs. (p. 129)

In addition, mixed methods research can be further classified as quantitative-dominant, qualitative-dominant (Johnson et al., 2007), or equal-status mixed methods (where the emphasis between quantitative and qualitative approaches is evenly split), termed by Morse (1991, 2003) as QUAN-Qual, QUAL-Quan, and QUAN-QUAL respectively.


87

Although both quantitative and qualitative research methods have many strengths and, if conducted with rigor, can inform mathematics education policy, they each contain unique weaknesses. Quantitative research is well suited to “answering questions of who, where, how many, how much, and what is the relationship between specific variables” (Adler, 1996, p. 5). However, quantitative research studies typically yield data that do not explain the reasons underlying prevalence rates, relationships, or differences that have been identified by the researcher. That is, quantitative research is not apt for answering questions of why and how. In contrast, the strength of qualitative research lies in its ability to capture the lived experiences of individuals; to understand the meaning of phenomena and relationships among variables as they occur naturally; to understand the role that culture plays in the context of phenomena; and to understand processes that are reflected in language, thoughts, and behaviors from the perspective of the participants. However, as noted by Onwuegbuzie and Johnson (2004), “Qualitative research is typically based on small, nonrandom samples…which means that qualitative research findings are often not very generalizable beyond the local research participants” (p. 410). Thus, because of the strengths and weaknesses inherent in mono-method research, in recent years, an increasing number of researchers from numerous fields have advocated for conducting studies that utilize both quantitative and qualitative research within the same inquiry—namely, mixed methods research.

Collins, Onwuegbuzie, and Sutton (2006) have identified four common rationales for mixing quantitative and qualitative research approaches: participant enrichment, instrument fidelity, treatment integrity, and significance enhancement. According to these methodologists, participant enrichment refers to the combining of quantitative and qualitative approaches for the rationale of optimizing the sample (e.g., increasing the number of participants, improving the suitability of the participants for the research study). Instrument fidelity refers to a combination of quantitative and qualitative procedures used by researchers to maximize the appropriateness and/or utility of the quantitative and/or


88

qualitative instruments used in the study. Treatment integrity pertains to the combining of quantitative and qualitative techniques for the rationale of assessing the fidelity of treatments, programs, or interventions. And, finally, significance enhancement involves the use of qualitative and quantitative approaches to maximize the interpretation of the results.

Each of these four rationales can come before, during, and/or after the study. With respect to participant enrichment, for example, mathematics education researchers could increase both the quantity and quality of their pool of participants of either a quantitative or qualitative study by interviewing participants who already have been selected for the study before the actual investigation begins (i.e., pre-study phase) to ask them to identify potential additional participants and to collect (additional) qualitative and quantitative information that establishes their suitability and willingness to participate in the study (Collins et al., 2006). Alternatively, interviews or other data collection tools (e.g., survey, rating scale, Likert-format scale) could be used during the study (i.e., study phase), for instance, to determine each participant’s suitability to continue in the study or to determine whether any modifications to the design protocol are needed. Further, these tools could be used after the study ends (i.e., post-study phase) as a means of debriefing the participants or to identify any outlying, deviant, or negative cases (Collins et al., 2006). With regard to instrument fidelity, mathematics education researchers might conduct a pilot study either to assess the appropriateness (e.g., score reliability, score validity, clarity, potential to yield rich data) and/or utility (e.g., cost, accessibility) of existing qualitative and/or quantitative instruments with the goal of making modifications, where needed, or developing a new instrument. Alternatively, in studies that involve multiple phases, mathematics education researchers could assess instrument fidelity on an ongoing basis and make modifications, where needed, at one or more phases of the study. In addition, mathematics education researchers could assess the validity/legitimation of the qualitative and/or quantitative information yielded by the instrument(s) in order to place the findings in a more appropriate context.


89

With respect to treatment integrity, mathematics education researchers could assess the intervention used in a study either quantitatively (e.g., obtaining a fidelity score that indicates the percentage of the intervention component that was implemented fully or the degree to which the treatment or program was implemented) or qualitatively (e.g., via interviews, focus groups, and/or observations). The use of both quantitative and qualitative techniques for assessing treatment integrity yields “the greatest insights into treatment integrity” (Collins et al., 2006, p. 82). Finally, with regard to significance enhancement, mathematics education researchers could use qualitative data to complement statistical analyses, quantitative data to complement qualitative analyses, or both. Moreover, using both quantitative and qualitative data analysis techniques either concurrently or sequentially within the same study can fulfill one or more of Greene, Caracelli, and Graham’s (1989) five purposes for integrating quantitative and qualitative approaches: triangulation (i.e., comparing results from quantitative data with qualitative findings to assess levels of convergence), complementarity (i.e., seeking elaboration, illustration, enhancement, and clarification of the findings from one method with results from the other method), initiation (i.e., identifying paradox and contradiction stemming from the quantitative and qualitative findings), development (i.e., using the findings from one method to help inform the other method), or expansion (i.e., expanding the breadth and range of a study by using multiple methods for different study phases). Thus, using mixed methods research approaches to fulfill one or more of these four rationales strengthens the design of some research studies.

Although there is a lack of knowledge about the prevalence of mixed methods research in mathematics education, an increasing number of researchers regard mixed methods research as representing scientifically based research. For example, in response to the narrow guidelines and review procedures of the Institute of Education Sciences, the American Evaluation Association (2003) adopted an official organizational policy response that included the statement, “Actual practice and many published examples demonstrate that alternative and mixed methods are rigorous and scientific.


90

To discourage a repertoire of methods would force evaluators backward” (§ 6). Even members of the National Research Council (NRC), who entered the dispute with a published consensus statement, Scientific Research in Education (NRC, 2002), supported the utilization of mixed methods research. For instance, Eisenhart and Towne (2003) noted that the NRC report supports the inclusion of “a range of research designs (experimental, case study, ethnographic, survey) and mixed methods (qualitative and quantitative) depending on the research questions under investigation” (p. 31).

Mixed methods research, the integration of qualitative and quantitative approaches in research studies, began in the 1960s. Campbell and Fiske (1959) are credited with providing the impetus for mixed methods research by introducing the idea of triangulation, which was extended further by Webb, Campbell, Schwartz, and Sechrest (1966). This research approach quickly is becoming prominent in the field of educational research (e.g., Bazeley, 2009; Denscombe, 2008; Greene, 2007; Happ, DeVito Dabbs, Tate, Hricik, & Erlen, 2006; Jang, McDougall, Pollon, & Russell, 2008; Johnson & Gray, 2010; Johnson & Onwuegbuzie, 2004; Johnson et al., 2007; Leech, Dellinger, Brannagan, & Tanaka, 2010; Molina-Azorín, 2010; O'Cathain, 2010; O'Cathain, Murphy, & Nicholl, 2008; Pluye, Gagnon, Griffiths, & Johnson-Lafleur, 2009; Teddlie & Tashakkori, 2009, 2010).

The prevalence of mixed methods research in other academic fields and disciplines (e.g., school psychology, counseling, special education, stress and coping research) has been investigated (e.g., Alise & Teddlie, 2010; Collins, Onwuegbuzie, & Jiao, 2006, 2007; Collins, Onwuegbuzie, & Sutton, 2007; Fidel, 2008; Hanson, Creswell, Plano Clark, Petska, & Creswell, 2005; Hurmerinta-Peltomaki & Nummela, 2006; Hutchinson & Lovell, 2004; Ivankova & Kawamura, 2010; Niglas, 2004; Onwuegbuzie, Jiao, & Collins, 2007; Powell, Mihalas, Onwuegbuzie, Suldo, & Daley, 2008; Truscott et al., 2010). In particular, with respect to the field of school psychology, Powell et al. (2008) examined empirical studies (n = 438) published in the four leading school psychology journals (i.e., Journal of School Psychology, Psychology in the Schools, School Psychology Quarterly, and


91

School Psychology Review) between 2001 and 2005. These researchers found that 13.7% of these studies were classified as representing mixed methods research. Of these mixed methods studies, 95.65% placed emphasis on the quantitative component (i.e., quantitative-dominant mixed methods research; Johnson et al., 2007), whereas only 4.35% were primarily qualitative in nature (i.e., qualitative-dominant mixed methods research; Johnson et al., 2007). Similarly, with regard to the field of special education, Collins, Onwuegbuzie, and Sutton (2007) undertook a content analysis of empirical studies (n = 131) published in the Journal of Special Education between 2000 and 2005. These researchers reported that 11.5% of these studies were classified as representing mixed methods research. Of these mixed methods investigations, 55.6% represented quantitative-dominant mixed methods research, 22.2% represented qualitative-dominant mixed methods research, and 22.2% represented equal-status mixed methods research (i.e., the emphasis between quantitative and qualitative approaches was approximately evenly split). With respect to the field of counseling, Hanson et al. (2005) searched for mixed methods research studies that had been published in counseling journals prior to May 2002. These researchers identified only 22 such studies that were published in counseling journals, with the majority of these articles (40.9%) being published in the Journal of Counseling Psychology. Building on the work of these researchers, Leech and Onwuegbuzie (2006) investigated the prevalence of mixed methods research published in the Journal of Counseling and Development (JCD) from late 2002 (Volume 80, Issue 3) through 2006 (Volume 84, Issue 4). Of the 99 empirical articles published in JCD during this period, only 2% represented mixed methods research. Finally, Onwuegbuzie et al. (2007) examined the prevalence of mixed methods research related to the area of stress and coping by selecting five major electronic bibliographic databases (i.e., PsycARTICLES[(EbscoHost]; PsycINFO[(EbscoHost]; Wilson Education Full-Text; CSA Illumina-Psychology; Business Source Premier [EbscoHost]) that represented the fields of psychology, education, and business. Using the keywords “stress and coping,” these


92

researchers noted that, of the 288 empirical articles that were identified, 5% represented mixed methods research.

Purpose of this Study Although researchers have documented the prevalence rate

of mixed methods research in other fields, few articles have been published examining the prevalence of mixed methods mathematics education research. Recently, Hart, Smith, Swars, and Smith (2009) examined the prevalence of mixed methods research in mathematics education articles published in six journals from 1995 to 2005. These researchers documented that 29% of the articles used both approaches in some way. Ross and Onwuegbuzie (2010) compared the prevalence of mixed methods in a flagship mathematics education journal, Journal for Research in Mathematics Education (JRME), to the prevalence in an all-discipline flagship education journal, American Educational Research Journal (AERJ), from 1999 to 2008. Mixed methods research accounted for 33% of all articles published in these two journals, whereas mixed methods was found to be more prevalent in JRME. With so few studies of the prevalence of mixed methods research in mathematics education, this study is important because it provides additional information regarding the extent to which mathematics education is keeping abreast of the latest methodological advances in incorporating mixed methods approaches. We focused on mathematics education articles published in JRME and The Mathematics Educator (TME) to (a) determine the prevalence of mixed methods research in mathematics education from 2002-2006, (b) to investigate the context associated with the use of mixed methods in mathematics education, and (c) to document possible reasons for using mixed methods in mathematics education research. This time period was chosen for investigation because it includes articles published after the passage of NCLB and the publication of the classic mixed methods textbooks (i.e., Bryman, 1988; Creswell, 1995; Greene & Caracelli, 1997; Newman & Benz, 1998; Reichardt & Rallis, 1994; Tashakkori & Teddlie, 1998). Additionally, we compared the prevalence of mixed methods articles in mathematics education journals to those in other disciplines. Because previous studies examining


93

the prevalence rates of mixed methods research articles in different disciplines and fields have revealed different distributions according to which component—qualitative or quantitative—was more dominant (e.g., Alise & Teddlie, 2010; Powell et al., 2008), this study also examined whether the articles were QUAN-Qual, QUAL-Quan, or QUAN-QUAL. Finally, to reveal a more complete picture of the research findings, we analyzed an exemplar mixed methods mathematics education article to demonstrate how qualitative and quantitative research approaches complement one another. In particular, we sought to answer the following research questions:

(i) How has the use of mixed methods research in two peer-reviewed mathematics education journals, JRME and TME, changed from 2002 to 2006 and how does the prevalence of mixed methods research in mathematics education compare to the prevalence in other academic disciplines?

(ii) Of the articles that utilize mixed methods research in JRME and TME:

(a) What is the context of the research? (b) What are the reasons cited in the articles for

the utilization of mixed methods? (c) What reasons are cited in the articles for their

particular composition of methods (QUAN-Qual, QUAL-Quan, or Quan-Qual)?

(iii) How can qualitative and quantitative methods complement one another in providing good educational research in mathematics education?

Method

Sample This study examined 87 journal articles published in JRME

(n = 60) or TME (n = 27), two peer-reviewed mathematics education journals. We chose the two journals because of their relatively low acceptance rates (11-20% for JRME and 10-25% for TME). JRME is widely regarded as the premier mathematics education journal in the United States and TME


94

provides a publication venue for research conducted by those new to the field, graduate students and recently minted PhDs. This sample represented all empirical articles published in these two journals between 2002 and 2006. Non-empirical articles (n = 75), such as editorials, reviews of the related literature, and commentaries, were not included in the study. It should be noted that neither journal encouraged the use of mixed methods research in their mission statements. Additionally, a mathematics education article (i.e., Wood, Williams, & McNeal, 2006) that exemplified a mixed methods research design was selected, not only because of the quality of the study but because it has been highly cited (i.e., 55 citations at the time this article took place; cf. Hirsch, 2005).

Data Collection We determined if each of the 87 articles in our sample

included mixed methods research. Articles were identified as using a mixed methods design if both qualitative and quantitative methods were utilized to any meaningful extent. For example, studies had to include one or more quantitative and qualitative data (such as frequency count and quotations) to be considered mixed methods. Attempts to classify actual published studies into distinct categories necessitated the addition of seven categorization rules (see Appendix). For each article, the particular emphasis used (QUAN-Qual, QUAL-Quan, QUAN-QUAL), the reasons for using more than one approach, and the context of the study were recorded. The example mixed methods research article was read closely to determine the qualitative and quantitative approaches utilized and the way that each approach provided a more comprehensive understanding of the results.

Data Analysis After determining the number of articles utilizing mixed

methods research for each journal over the 5-year span, we calculated the annual and total percentages of mixed methods usage for each journal, as well as both journals combined, for the years 2002 to 2006. We used these values to describe how the prevalence of mixed methods research in mathematics education research has changed over time and to compare these


95

rates with those in other academic disciplines. A series of chi-square tests of homogeneity (cf. Leech & Onwuegbuzie, 2002) was used to compare the prevalence rates (i.e., percentages) between the number of mixed methods research articles published in the two mathematics education journals and the number published in other disciplines for which the sample size and group sizes were reported clearly. A 5% level of statistical significance was used. Also, effect sizes, as measured by Cramer’s V, were reported for all statistically significant findings. Also, we computed odds ratios as a second index of effect size.

After each mixed methods research article was coded according to the emphasized research orientation (QUAN-Qual, QUAL-Quan, or QUAN-QUAL), the annual total and percentage for each journal were calculated. In most cases, it was easy to determine which approach was dominant. However, in some cases, we had to re-examine the purpose of the article and research questions to determine the emphasis. Constant comparison analysis (Glaser & Strauss, 1967) was used to determine the reasons for using mixed methods. Specifically, each identified reason was given a code. Also, each reason was compared with previous reasons to ensure that similar reasons were labeled with the same thematic code. Each emergent theme contained one or more reasons that were each linked to a formulated meaning of significant statements. Thus, the themes emerged a posteriori, and, in contrast, classification of the utilization of designs occurred a priori using the predetermined codes, QUAN-Qual, QUAL-Quan, or QUAN-QUAL. Additionally, we determined the contextual frame of each mixed methods article by identifying the topic.

To demonstrate how combining both quantitative and qualitative approaches within one research study can provide more rigorous educational research we chose one mixed methods journal article as an exemplar. We described the results and inferences stemming from the use of each approach and then compared these to the overall results and inferences from combining both approaches.


96

Results and Discussion Mixed methods research constituted approximately one

third (31%) of all empirical articles accepted for publication in JRME and TME from 2002 to 2006; yet the rate of mixed methods research decreased from 2002 to 2006 from 40% to 30% (Table 1). From 2002 to 2006 the percentage of mixed methods research articles published in JRME went from 55% to 23%, with 2006 having the lowest percentage. On the other hand, the percentage of mixed methods articles published in TME increased from 0% in 2002 to 43% by 2006. Over the 5-year period, JRME actually published more than twice the percentage of mixed methods research articles than did TME, with 38% and 15%, respectively. Interestingly, no articles specifically contained the phrase “mixed methods” but two articles did specify the use of both quantitative and qualitative approaches.

Table 1

Percentages of Mixed Methods Research Studies in JRME and TME

Year JRME TME Annual Total

2002 6/11 = 55% 0/4 = 0% 6/15 = 40%

2003 6/13 = 46% 1/7 = 14% 7/20 = 35%

2004 3/11 = 27% 0/4 = 0% 3/15 = 20%

2005 5/12 = 42% 0/5 = 0% 5/17 = 29%

2006 3/13 = 23% 3/7 = 43% 6/20 = 30%

Total 23/60 = 38% 4/27 = 15% 27/87 = 31% The combined 31% prevalence rate found in the current

study for the two selected mathematics education research journals over a 5-year span is similar to the 29% prevalence of mixed methods in mathematics education journal articles documented by Hart et al. (2009) from 1995 to 2005. However, the 31% prevalence rate was much higher than those reported in other academic disciplines (e.g., Collins, Onwuegbuzie, & Sutton, 2007; Hanson et al., 2005; Leech & Onwuegbuzie,


97

2006; Onwuegbuzie et al., 2007; Powell et al., 2008). Lower prevalence rates for other disciplines have been reported for a similar time span, the two highest rates at 13.7% and 11.5% in school psychology journals (Powell et al., 2008), and special education journals, respectively (Collins, Onwuegbuzie, & Sutton, 2007). Both of these rates are less than one half of the rate of mixed methods research identified in JRME and TME. For other disciplines, mixed methods research studies are published with even less frequency, with such studies accounting for only 2% of the published empirical studies in counseling journals (Leech & Onwuegbuzie, 2006) and only 5% in various research journals that publish stress and coping research (Onwuegbuzie et al., 2007).

More specifically, the 31% prevalence rate is statistically significantly higher than the prevalence rate observed by Powell et al. (2008) for leading school psychology journals (Χ2[1] = 10.30, p < .0013, Cramer’s V = .13), the prevalence rate observed by Leech and Onwuegbuzie (2006) for a leading counseling journal (Χ2[1] = 21.62, p < .0001, Cramer’s V = .32), the prevalence rate observed by Collins, Onwuegbuzie, and Sutton (2007) for a leading special education journal (Χ2[1] = 5.97, p < .01, Cramer’s V = .17), and the prevalence rate observed by Onwuegbuzie et al. (2007) for a the field of stress and coping (Χ2[1] = 35.60, p < .0001, Cramer’s V = .29). Moreover, mixed methods research articles were more than twice as likely to be published in the selected mathematics education journals than in the leading school psychology journals (Odds ratio = 2.27, 95% Confidence Interval [CI] = 1.36, 3.77) and a leading special education journal (Odds ratio = 2.69, 95% CI = 1.19, 6.07), more than 6 times as likely to be published in the mathematics education journals than in the field of stress and coping (Odds ratio = 6.88, 95% CI = 3.40, 13.90), and more than 15 times as likely to be published in the mathematics education journals than in a leading counseling journal (Odds ratio = 15.36, 95% CI = 3.55, 66.47).

Of the mixed methods articles in both journals over the 5-year period, 59% were qualitative-dominant, whereas quantitative-dominant articles constituted 33% and equal-status mixed research articles constituted only 7% (Table 2). Given the increase in qualitative approaches used in mathematics


98

education articles over the past 20 years, it is not surprising that a qualitative-dominant approach constituted the highest percentage of articles overall, as well as in each of JRME and TME individually, (cf. Table 3a and 3b). It is also understandable that fewer articles would constitute a balanced quantitative-qualitative design.

Table 2

Percentages of Mixed Methods Research Study Emphasis in JRME and TME Combined

Year n QUAN-Qual QUAL-Quan QUAN-QUAL

2002 6 50% 33% 17%

2003 7 0% 86% 14%

2004 3 67% 33% 0%

2005 5 40% 60% 0%

2006 6 33% 67% 0%

Total 27 33% 59% 7%

Table 3a

Percentages of Mixed Methods Research Study Emphasis in TME


2002 0

2003 1 0% 100% 0%

2004 0

2005 0

2006 3 33% 67% 0%

Total 4 25% 75% 0%


99

Table 3b

Percentages of Mixed Methods Research Study Emphasis in JRME


2002 6 50% 33% 17%

2003 6 0% 83% 17%

2004 3 67% 33% 0%

2005 5 40% 60% 0%

2006 3 33% 67% 0%

Total 23 35% 57% 9% Constant comparison analysis provided interesting

information regarding reasons behind researchers’ use of mixed methods research in mathematics education journals, as well as the emphasis of the mixed methods research designs. Specific reasons documented throughout the mixed methods research articles included examination of relationships, ideas, beliefs, strategies, mental actions, abilities, conceptions, reflections, reasoning development, experiences, self-reports, understanding, behaviors, determination of differences, effects of pictorial representations on success, practices, descriptions of courses, performance as ascertained via a variety of outcomes, and problem solving. All articles involving both qualitative and quantitative research approaches examined actions, behaviors, relationships, ideas, and/or understanding. In other words, ideals and outcomes involving more than mere achievement scores and closed-ended effects required evidence ascertained from both approaches to support one another.

The researchers of these mixed methods articles did not simply examine outcomes of various independent factors on student success measured solely quantitatively. Researchers in these studies also did not simply rely solely on analysis of transcribed or summarized interview data or observations to determine student knowledge and understanding. Examination of these two mathematics education journals revealed that their use of mixed methods research was needed to delve deeper into


100

teachers’ and students’ behaviors, actions, and understandings. Articles utilizing both methods required data that supported ideas that could be understood via description and statistical techniques—whether categorical data, or achievement scores.

The high percentage of mixed methods research published in these journals indicates a growing desire of mathematics education researchers to include thought processes, occurrences, actions, and behaviors as related to student achievement outcomes and successful instruction. No longer are mathematics education researchers only collecting and analyzing either quantitative or qualitative data, they are realizing the value in combining description, narration, summaries, comparisons, patterns, and so on, as they impact mathematical understanding. Noteworthy is the fact that most mixed methods research studies involve mathematical understanding, not simply knowledge or skills. The National Council of Teachers of Mathematics (2000) advocates the combined attainment of conceptual and procedural understanding in mathematics. The findings revealed the importance of mixed methods, qualitative, and quantitative approaches in these two mathematics education journals. Mixed methods research again constituted 31% of all empirical articles, whereas qualitative and quantitative research accounted for 39% and 21%, respectively. It should be noted that qualitative studies accounted for the highest proportion of empirical articles and that qualitative-dominant mixed methods research designs accounted for the highest percentage of mixed methods research. With the movement towards overall mathematical literacy (Hiebert & Carpenter, 1992; Van de Walle, 2001), constructivist approaches (von Glasersfeld, 1997), and standards-based curriculum (National Council of Teachers of Mathematics [NCTM], 2000), the findings might suggest that mathematics education researchers are interested in revealing a big picture associated with mathematics teaching and learning, with high emphasis on thinking patterns, behaviors, understanding, and the relations thereof, providing justification for a higher proportion of qualitative-dominant mixed methods research articles.

The constant comparison analysis revealed reasons behind orientation of mixed methods research articles. Articles labeled


101

as QUAN-Qual were designed to investigate levels of understanding, levels of correctness, classification, correlations, categorization, significance, and accuracy—to name a few research objectives. Articles labeled as QUAL-Quan were designed to depict actions and behaviors via detailed descriptions and pictorial representations of thinking patterns, problem solving, and social discourse. Researchers who used qualitative-dominant studies also sought to examine processes underlying understanding, instead of merely identifying relationships between a priori variables and levels of understanding. Specifically, researchers of qualitative-dominant mixed methods research studies examined mental actions, discourse, verbal justifications, beliefs, correlations between observations and scores, conceptions, social interactions, task descriptions, observed qualities, and problem solving processes. These researchers reported richer data than would have been the case if data from only one strand (e.g., quantitative findings) had been reported. Thus, findings from both the quantitative and qualitative components of quantitative-dominant mixed methods research studies and qualitative-dominant mixed methods research studies provided justification for the use of each approach.

Analysis of the 27 mixed methods research studies revealed the following five major contextual themes in mathematics education research: relationships, thought processes, pedagogy, representations, and understanding (Table 4). Exemplars whose topics of study were these themes included levels of abstraction, levels of representation, understanding of fractions, teachers’ ideas, arithmetic and algebraic problem-solving skills, thinking patterns, and beliefs about fairness—to name a few research areas.


102

Table 4

Contexts Associated with Mathematics Education Mixed Methods Articles

Contextual Themes

Exemplars

Relationships Japanese students’ level of abstraction and level of representation Ethnicity, out-of-school activities, and arithmetical achievements of Latin American and Korean American students Normative patterns of social interaction and children’s mathematical thinking Third graders’ use of reference point and guess-and-check strategies and accuracy

Thought Processes

Math majors’ reflections on proofs Inservice teachers’ conceptions of proof Students’ conception of mathematical definition Preservice teachers’ conceptions of how materials should be used Preservice teachers’ arithmetic and algebraic problem-solving strategies Mental actions involved in covariational reasoning Reasoning development and thinking patterns of middle school students Beliefs about fairness of dice Sixth and seventh graders’ problem- solving strategies Seventh and eighth graders’ problem- solving strategies, specifically in algebra

Pedagogy Third-grade teacher’s efforts to support the development of students’ algebraic skills Formal evaluative events across courses in a range of institutions in South Africa Japanese and U.S. teachers’ ideas on teaching strategies Extent to which teachers implement mathematics education reform Compatibility of teaching practices of fourth-grade teachers with NCTM Standards


103

Representations High school students’ calculus diagrams Use of representations in write-ups

Understanding Fourth and fifth graders’ understanding of fractions Two low-performing first-grade students’ understanding Preservice teachers’ understanding of prime numbers Above-average high school students’ calculus and algebra skills and understanding Middle school students’ understanding of the equal sign and performance in solving algebraic equations Performance of NCTM-oriented students on standardized tests

Sample Mixed Methods Research Article

Qualitative and quantitative research approaches can be used to complement one another in mathematics education research articles. We used a sample mixed methods research article to illustrate how mixed methods techniques can be used in mathematics education, as well as to illustrate the factors influencing such a complementary design. The sample article, entitled, “Children’s Mathematical Thinking in Different Classroom Cultures,” published in 2006, was taken from JRME. This article (Wood et al., 2006) focused on investigating effects of social interaction on children’s mathematical thinking. Using what they referred to as a “quantitative-qualitative research paradigm” (p. 229), they observed 42 classroom lessons, in order to investigate children's mathematical thinking as articulated in class discussions and their interaction patterns. The analysis used two coding schemes—one for interaction patterns and one for mathematical thinking. Classroom observation transcription notes were used to reveal qualitative and quantitative findings, related to both coding schemes. Qualitative research approaches included transcription of classroom dialogue, identification of classroom cultures, identification of consistent


104

patterns of interaction within segments and across lessons, and the provision of examples of interaction patterns described per classroom culture. Quantitative research approaches included calculation of percentages of interaction patterns and mathematical thinking by class culture (conventional textbook, conventional problem solving, strategy reporting, and inquiry/argument) and calculation of percentages of children’s levels of spoken mathematical thinking (via coding of dialogue). Transcripts of dialogue for each mathematical problem were coded as a particular interaction pattern (e.g., a hinted solution, inquiry, exploration of methods). The percentages of occurrences for 17 interaction patterns were calculated for each of the class cultures. Types of mathematical thinking (recognizing, building-with, and constructing) were also examined quantitatively, via calculation of percentages of each type that occurred at the following levels: comprehension, application, analyzing, synthetic-analyzing, evaluative-analyzing, synthesizing, and evaluating.

Many articles necessitate both types of data collection approaches in order to answer the underlying research questions. In this study, simply providing the transcribed dialogue and/or segmenting the dialogue into pieces would not have illustrated the frequency of types of interactions or the level of student understanding. Such data collection called for quantitative coding of the data. In fact, with the ability to segment the classroom cultures, interaction patterns, and mathematical thinking, the study required numerical data to support the qualitative-dominant study. Frequency scores allowed the researchers to explore the relationship among interaction types, expressed mathematical thinking, and classroom culture. Reform-oriented class cultures revealed more student-dominated participation, as well as higher percentages of higher level thinking. The transcribed student solutions showed the exact facets of such higher level thinking and discourse. In summary, both approaches used together revealed that social interaction does, in fact, increase children’s thinking.


105

Conclusion The results of this study have revealed the increasing role

of mixed methods designs in mathematics education research studies. Despite federal legislation such as NCLB and ESRA that have called for scientifically based research in education—wherein randomized controlled trials were deemed to represent the gold standard for research—approximately one third of all empirical articles published in these two mathematics education research journals over a 5-year span represented mixed methods research studies. Bearing in mind the utility of mixed methods research (Collins et al., 2006; Greene, 2007; Johnson & Onwuegbuzie, 2004; Johnson et al., 2007), this finding is very encouraging because the present study provided evidence that mixed methods research is being used by a significant proportion of mathematics education researchers whose articles are published in these journals. Such articles provide in-depth descriptions of tangible and intangible variables, as related to improvement of students’ mathematical understanding. However, it remains for mathematics education researchers to optimize their mixed methods research designs; they will decide how to design and to modify such studies to best meet their needs. This can be accomplished by utilizing frameworks for conducting mixed methods research that have been developed for many disciplines belonging to the health or social and behavioral science fields. For instance, Collins, Onwuegbuzie, and Sutton’s (2006) framework could be used to help researchers determine their rationale for mixing quantitative and qualitative research approaches. These researchers conceptualized that four rationales (participant enrichment, instrument fidelity, treatment integrity, and/or significance enhancement) can be addressed before, during, and/or after the study. For example, in mathematics education a researcher might administer a quantitative measure and conduct interviews or observations to assess the fidelity of an instructional treatment, program, or intervention, thereby using mixed methods to establish treatment fidelity. Using these methods at different points in the research process can support the researchers’ goals in different ways. Assessing treatment fidelity at the outset of the study can help assess the feasibility of the treatment protocol being implemented in a rigorous and


106

comprehensive manner; during the study it can provide formative evaluation of the fidelity of the treatment to determine whether mid-course adjustments are needed; and after the study it can provide summative evaluation of the treatment to determine the extent to which fidelity occurred. This example highlights that using such a framework could help mathematics education researchers view the combining of quantitative and qualitative approaches as a fluid process that can occur at any stage of the research. Perhaps because of the uniqueness of mathematics education, a framework needs to be developed for utilizing mixed methods techniques in mathematics education research. In any case, determining appropriate frameworks for mathematics education researchers should be the subject of future research.

.

References Adler, L. (1996). Qualitative research of legal issues. In D. Schimmel (Ed.),

Research that makes a difference: Complementary methods for examining legal issues in education, NOLPE Monograph Series (No. 56) (pp. 3-31). Topeka, KS: NOLPE.

Alise, M. A., & Teddlie, C. (2010). A continuation of the paradigm wars? Prevalence rates of methodological approaches across the social/behavioral sciences. Journal of Mixed Methods Research, 4, 103–126. doi:10.1177/1558689809360805

American Evaluation Association. (2003). American Evaluation Association Response to U. S. Department of Education Notice of proposed priority, Federal Register RIN 1890-ZA00, November 4, 2003: Scientifically based evaluation methods. Retrieved February 19, 2008, from http://www.eval.org/doestatement.htm

Bazeley, P. (2009). Mixed methods data analysis. In S. Andrew & E. J. Halcomb (Eds.), Mixed methods research for nursing and the health sciences (pp. 84–118). Chichester, UK: Wiley-Blackwell.

Bryman, A. (1988). Quantity and quality in social research. London, England: Unwin Hyman.

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. doi:10.1037/h0046016


107

Collins, K. M. T., Onwuegbuzie, A. J., & Jiao, Q. G. (2006). Prevalence of mixed methods sampling designs in social science research. Evaluation and Research in Education, 19, 83–101. doi:10.2167/eri421.0

Collins, K. M. T., Onwuegbuzie, A. J., & Jiao, Q. G. (2007). A mixed methods investigation of mixed methods sampling designs in social and health science research. Journal of Mixed Methods Research, 1, 267–294. doi:10.1177/1558689807299526

Collins, K. M. T., Onwuegbuzie, A. J., & Sutton, I. L. (2006). A model incorporating the rationale and purpose for conducting mixed methods research in special education and beyond. Learning Disabilities: A Contemporary Journal, 4, 67–100.

Collins, K. M. T., Onwuegbuzie, A. J., & Sutton, I. L. (2007, February). The role of mixed methods in special education. Paper presented at the annual meeting of the Southwest Educational Research Association, San Antonio, TX.

Creswell, J. W. (1995). Research design: Qualitative and quantitative approaches. Thousand Oaks, CA: Sage.

Denscombe, M. (2008). Communities of practice: A research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2, 270–283. doi:10.1177/1558689808316807

Education Sciences Reform Act. (Pub. L. No. 107-279). (2002). Retrieved February 18, 2008, from http://www.ed.gov/policy/rschstat/leg/PL107-279.pdf

Eisenhart, M., & Towne, L. (2003). Contestation and change in national policy on “scientifically based” education research. Educational Researcher, 32(7), 31–38. doi:10.3102/0013189X032007031

Fidel, R. (2008). Are we there yet? Mixed methods research in library and information science. Library &Information Science Research, 30, 265–272.

Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine.

Greene, J. C. (2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass.

Greene, J. C., & Caracelli, V. W. (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Evaluations, 74. San Francisco, CA: Jossey-Bass.

Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11, 255–274.


108

Hanson, W. E., Creswell, J. W., Plano Clark, V. L., Petska, K. S., & Creswell, J. D. (2005). Mixed methods research designs in counseling psychology. Journal of Counseling Psychology, 52, 224–235. doi:10.1037/0022-0167.52.2.224

Happ, M. B., DeVito Dabbs, D. A., Tate, J., Hricik, A., & Erlen, J. (2006). Exemplars of mixed methods data combination and analysis. Nursing Research, 55(2, Supplement 1), S43–S49. doi:10.1097/00006199-200603001-00008

Hart, L. C., Smith, S. Z., Swars, S. L., & Smith, M. E. (2009). An examination of research methods in mathematics education. Journal of Mixed Methods Research, 30, 26–41. doi:10.1177/1558689808325771

Hiebert, J., & Carpenter, T. P. (1992). Learning and teaching with understanding. In D. A Grouws (Ed.), Handbook of research on mathematics teaching and learning (pp. 65–97). New York, NY: Macmillan.

Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences, 102, 16569–16572.

Hurmerinta-Peltomaki, L., & Nummela, N. (2006). Mixed methods in international business research: A value-added perspective. Management International Review, 46, 439–459. doi:10.1007/s11575-006-0100-z

Hutchinson, S. R., & Lovell, C. D. (2004). A review of methodological characteristics of research published in key journals in higher education: Implications for graduate research teaching. Research in Higher Education, 45, 383–403.

Ivankova, N. V., & Kawamura, Y. (2010). Emerging trends in the utilization of integrated designs in the social, behavioral, and health sciences. In A. Tashakkori & C. Teddlie (Eds.), Sage handbook of mixed methods in social and behavioral research (2nd ed., pp. 581–611). Thousand Oaks, CA: Sage.

Jang, E. E., McDougall, D. E., Pollon, D., & Russell, M. (2008). Integrative mixed methods data analytic strategies in research on school success in challenging environments. Journal of Mixed Methods Research, 2, 221–247. doi:10.1177/1558689808315323

Johnson, R. B., & Gray, R. (2010). A history of philosophical and theoretical issues for mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Sage handbook of mixed methods in social and behavioral research (2nd ed., pp. 69–94). Thousand Oaks, CA: Sage.

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26. doi:10.3102/0013189X033007014


109

Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1, 112–133. doi:10.1525/sp.1960.8.2.03a00030

Leech, N. L., Dellinger, A., Brannagan, K. B., & Tanaka, H. (2010). Evaluating mixed research studies: A mixed methods approach. Journal of Mixed Methods Research, 4, 17–31.

Leech, N. L., & Onwuegbuzie, A. J. (2002, November). A call for greater use of nonparametric statistics. Paper presented at the annual meeting of the Mid-South Educational Research Association, Chattanooga, TN.

Leech, N. L., & Onwuegbuzie, A. J. (2006, November). Mixed methods research in counseling research: The past, present, and future. Paper presented at the annual meeting of the Mid-South Educational Research Association, Birmingham, AL.

Leech, N. L., & Onwuegbuzie, A. J. (2008). Qualitative data analysis: A compendium of techniques and a framework for selection for school psychology research and beyond. School Psychology Quarterly, 23, 587–604. doi:10.1037/1045-3830.23.4.587

Lester, F. K. (2005). On the theoretical, conceptual, and philosophical foundations for research in mathematics education. ZDM, 37, 457–467. doi:10.1007/BF02655854

Lester, F. K., & Lambdin, D. V. (2003). From amateur to professional: The emergence and maturation of the U. S. mathematics education research community. In G. M. A. Stanic & J. Kilpatrick (Eds.), A history of school mathematics (pp. 1629–1700). Reston, VA: National Council of Teachers of Mathematics.

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.

McLafferty, C. L., Slate. J. R., & Onwuegbuzie, A. J. (2010). Transcending the quantitative–qualitative divide with mixed methods: A multidimensional framework for understanding congruence, coherence, and completeness in the study of values. Counseling and Values, 55, 46–62.

Molina-Azorín, J. F. (2010). The use and added value of mixed methods in management research. Journal of Mixed Methods Research, 5, 7–24. doi:10.1177/1558689810384490

Morse, J. (1991). Approaches to qualitative-quantitative methodological triangulation. Nursing Research, 40, 120–123. doi:10.1097/00006199-199103000-00014

Morse, J. (2003). Principles of mixed methods and multimethod research design. In A. Tashakkori & C. Teddlie (Eds.) Handbook of mixed methods in social and behavioral Research (pp. 189–208). Thousand Oaks, CA: Sage.


110

National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: Author.

National Research Council. (2002). Scientific research in education. Washington, DC: National Academy Press.

Newman, I., & Benz, C. R. (1998). Qualitative-quantitative research methodology: Exploring the interactive continuum. Carbondale, IL: Southern Illinois University Press.

Niglas, K. (2004). The combined use of qualitative and quantitative methods in educational research. Unpublished doctoral dissertation, Tallinn Pedagogical University, Tallinn, Estonia.

No Child Left Behind Act of 2001. (2001). Pub. L. No. 107-110.

O’Cathain, A. (2010). Assessing the quality of mixed methods research: Towards a comprehensive framework. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed., pp. 531–555).Thousand Oaks, CA: Sage.

O’Cathain, A., Murphy, E., & Nicholl, J. (2008). The quality of mixed methods studies in health services research. Journal of Health ServicesResearch Policy, 13, 92–98. doi:10.1258/jhsrp.2007.007074

Onwuegbuzie, A. J., Jiao, Q. G., & Collins, K. M. T. (2007). Mixed methods research: A new direction for the study of stress and coping. In. G. Gates (Ed.), Emerging thought and research on students, teacher, and administrator stress and coping (Research on Stress and Coping in Education, Vol. 4) (pp. 215-243). Greenway, CT: Information Age.

Onwuegbuzie, A. J., & Johnson, R. B. (2004). Mixed method and mixed model research. In R.B. Johnson & L. B. Christensen (Eds.), Educational research: Quantitative, qualitative, and mixed approaches (pp. 408–431). Needham Heights, MA: Allyn & Bacon.

Patton, M. Q. (2006). Foreword: Trends and issues as context. Research in the Schools, 13(1), i–ii.

Pluye, P., Gagnon, M., Griffiths, F., & Johnson-Lafleur, J. (2009). A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative, and mixed methods primary studies in mixed studies reviews. International Journal of Nursing Studies, 46, 529–546.

Powell, H., Mihalas, S., Onwuegbuzie, A. J., Suldo, S., & Daley, C. E., (2008). Mixed methods research in school psychology: A mixed methods investigation of trends in the literature. Psychology in the Schools, 45, 291–309. doi:10.1002/pits.20296

Reichardt, C. S., & Rallis, S. F. (1994). (Eds.). The qualitative-quantitative debate: New Perspectives. New Directions for Evaluation, 61. San Francisco, CA: Jossey-Bass.


111

Ross, A., & Onwuegbuzie, A. J. (2010). Mixed methods research design: A comparison of prevalence in JRME and AERJ. The International Journal of Multiple Research Approaches, 4, 233–245.

Scandura, J. M. (Ed.). (1967). Research in mathematics education. Washington, DC: National Council of Teachers of Mathematics.

Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Applied Social Research Methods Series (Vol. 46). Thousand Oaks, CA: Sage.

Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative techniques in the social and behavioral sciences. Thousand Oaks, CA: Sage.

Teddlie, C., & Tashakkori, A. (2010). Overview of contemporary issues in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed., pp. 1–41). Thousand Oaks, CA: Sage.

Truscott, D., Swars, S., Smith, S., Thornton-Reid, F., Zhao, Y., & Dooley, C. (2010). A cross-disciplinary examination of the prevalence of mixed methods in educational research: 1995-2005. International Journal of Social Research Methodology, 13, 317–328. doi:10.1080/13645570903097950

Van de Walle, J. A. (2001). Elementary and middle school mathematics: Teaching developmentally (4th ed.). New York, NY: Longman.

von Glasersfeld, E. (1997). Amplification of a constructivist perspective. Issues in Education, 3, 203–211.

Webb, E. J., Campbell, D. T., Schwartz, R. D., & Sechrest, L. (1966). Unobtrusive measures. Chicago, IL: Rand McNally.

Wood, T., Williams, G., & McNeal, B. (2006). Children’s mathematical thinking in different classroom cultures. Journal for Research in Mathematics Education, 37, 222–255.


112

Appendix

Decision Rules for Classifying Articles Published in Selected Mathematics Education Journals

Rule 1. Studies were not coded as representing mixed methods research if the addition of the qualitative information was not systematic and/or planned. For example, reporting spontaneous, anecdotal comments from study participants in the discussion section of a quantitative study did not result in a mixed methods research designation.

Rule 2. Mere use of interview methods during data collection did not automatically result in a mixed methods research designation. Furthermore, structured or semistructured interviews that generated solely or predominantly quantitative data, such as frequency counts or a list of target behaviors, were not considered as being representative of qualitative research.

Rule 3. In studies that used small sample sizes to evaluate quantitatively intervention effectiveness, detailed background information about participant(s) was not coded as representing a qualitative component.

Rule 4. Reporting planned collection of qualitative data for the purpose of assessing or verifying the appropriateness of an intervention resulted in a mixed methods research designation (assuming that quantitative data were collected solely for the purpose of evaluating treatment outcomes). Even intervention studies that reported only quantitative analyses in the results section were still coded as mixed methods research if the brief discussion of treatment integrity included qualitative data.

Rule 5. Mixed methods research studies in which the qualitative component was essential in order for the remainder of the study to be conducted, and those studies that reported and analyzed both qualitative and quantitative data were coded as mixed methods research. For example, studies employing qualitative methods (e.g., focus groups, open-ended questionnaires) in order to develop the measurement tool that was used in the remainder of the study were designated as


113

mixed methods research because the completion of the study was contingent on the creation of the instrument.

Rule 6. Content analyses were coded as quantitative if the results of the content analysis were reported numerically (e.g., this study). If the content analysis yielded themes that were not quantified in any way, the study was coded as representing qualitative research.

Rule 7. Highlighting case examples from a larger quantitative study did not result in a mixed methods research designation unless the case example section was augmented by new qualitative data (as opposed to simply an in-depth examination of the quantitative data yielded from the case examples who were participants in the larger quantitative study).

prevalence of mixed methods research in mathematics education · 2013-08-02 · for research in...

Documents