Is it good to mix methods?

Martyn Hammersley

The Open University

What is mixed methods?

Here is one definition:

‘Mixed methods’ is a term increasingly used in social science to describe ‘the class of research where the researcher mixes or combines quantitative and qualitative research techniques, methods, approaches, concepts or language into a single study’

(Jonson and Onwuegbuzie 2004:17)

Precursors of mixed method• Anthropologists doing ethnography but also

drawing on village surveys, for example to document variation in attitude towards schooling within a community.

• Sociologists doing in-depth case studies but also collecting and analysing quantitative data, for example Hargreaves 1967 and Lacey 1970.

• Survey researchers doing qualitative pilot work, experimentalists using qualitative debriefing interviews.

It is possible to ‘combine methods’ without mixing quantitative with

qualitative

• Using relatively unstructured observational and interview data in the same study.

• Collecting questionnaire data and combining these with data from official statistics.

• Using factor analysis and regression analysis within the same study.

• Collecting data through ethnographic means but subjecting it to discourse analysis

Triangulation

• This term may, but need not, imply ‘mixing methods’.

• Originally, it referred to checking validity through using data having different threats to validity, and relied upon an analogy with navigation and surveying.

• Later ‘triangulation’ came to be used to mean ‘using different kinds of data to provide information about different aspects of the same phenomenon’ (see Hammersley 2008).

The mixed methods movement

• A response to the paradigm wars of the 1980s and 90s: recognition of the value of both qualitative and quantitative methods.

• A pragmatic attitude: the ‘fit for purpose’ and ‘best of both’ rationales.

• Is it a new paradigm? Is this the best way to think of it?

Paradigm or pragmatism, or Pragmatism as a paradigm?

• Maxwell and Mittapalli (2010) recommend realism as the guiding methodological philosophy

• Mertens et al (2010) propose the ‘transformative paradigm’

• Johnson and Onwuegbuzie (2004) and others recommend Pragmatism as an underpinning philosophy for mixed methods research.

A rather different view

‘Mixed methods research is a Trojan Horse for positivism, reinstalling it as the most respected form of social research, while at the same time — through inclusion — neutralizing the oppositional potential of other paradigms and methodologies that more commonly use qualitative methods’ (Giddings and Grant 2006:59).

The value of pragmatism

• Research is a practical activity aimed at a particular goal and product: producing answers to a set of research questions.

• Given this, how researchers go about their work should be determined by what is required to answer the research questions they are addressing.

Not so simple!

• Even the formulation of research questions is based upon assumptions about the nature of the phenomena concerned and how they can be understood.

• There are discrepant assumptions about the social world and how it can be understood built into different data analysis strategies.

• Much discussion of mixing methods underplays these differences in methodological philosophy.

Can we not just discover how the world is through our research?

• To some extent yes.

• But we cannot avoid making epistemological and ontological assumptions, at least for working purposes.

• And when it comes to what are for us deep or central assumptions about the world, there can be no direct test of these.

The unavoidability, but also danger, of relying upon ‘commonsense’

• It is impossible to question or avoid all assumptions. All research relies upon some.

• But we must try to avoid being misled by what seems to us to be ‘commonsense’.

• Research demands that we be critically cautious. But this cannot amount to a thoroughgoing scepticism, otherwise…

• So, what should we assume, and what should we doubt? In part, this is what qualitative and quantitative researchers disagree about.

The case of eliciting accounts

Criticisms of the structured questionnaire by qualitative researchers:

1.Assumptions built into the structure might be false, or might apply to some informants but not to others, introducing systematic error;

2.We will probably fail to recognise when people are uncertain or ambivalent;

3.It is false to assume that ‘standardising the stimuli’ across respondents/informants makes their responses comparable.

A Questionnaire Item

 

•  

•  

Tick ONE BOX ONLY for each question

Is your research:

Positivist? [ ]

Anti-positivist? [ ]

Is it aimed at:

Producing knowledge? [ ]

Improving practice? [ ]

Getting an EdD? [ ]

The ‘radical critique’ of interviews• Interview data are co-constructed through the

interaction between interviewer and interviewee: responses do not simply reflect the attitudes of informants.

• Answers given also depend upon the socio-cultural resources that the informant has available to draw on. Different resources will produce different answers.

• People’s behaviour is not a product of stable beliefs, attitudes etc, or of fixed causes operating upon them, but is context-dependent.

The uncertain relationship between methodology and method

• Use of structured questionnaires may be premissed upon positivist assumptions, but it need not be.

• Use of relatively unstructured interviews may rely upon interpretivist philosophy but need not.

• The radical critique of interviews is based upon constructionism, and this does imply a re-specification of the research goal. However, the points the critique makes can be recognised up to a point by both positivists and interpretivists.

A paradigmatic chasm?

• We don’t have to accept these methodological philosophies (positivism, interpretivism, constructionism) at face value. In fact, each has stronger and weaker versions.

• However, integrating these philosophies in some way, or adopting a different one, as the rationale for using ‘whatever methods are appropriate’ does not overcome the conflicts.

• Some philosophical assumptions have to be made in all research, and it is important to be aware of these.

Should all research mix methods?

• My answer to this question is definitely ‘No’.

• But if you ask me whether research should be designed so as best to answer its research questions I would say yes, and this may well involve combining quantitative and qualitative methods.

• At the same time, it’s necessary to remember that how we frame research questions, what inferences we draw from data, etc can rely upon contentious philosophical assumptions.

A final point

‘The process of mixing requires distinct method elements to mix and so, ironically, the metaphor of mixing actually works to preserve method schisms […]’ (Gorard 2007 p.1).

Do qualitative and quantitative approaches represent two absolutely distinct sets of methods? No, these terms indicate, and at the same time obscure, a much more complex field of decisions as regards research design, data collection, and analysis. But this is another story!

ReferencesGiddings, L. S., & Grant, B. M. (2006). Mixed-methods research, positivism dressed in drag?

Journal of Research in Nursing, 11(3), 195-203.

Gorard, S. (2007). Mixing methods is wrong: An everyday approach to educational justice. Paper presented at the British Educational Research Association

Hammersley, M. (2008) ‘Troubles with triangulation’, in Bergman, M. (ed.) Advances in Mixed Methods Research, London, Sage.

Hargreaves, D. (1967) Social Relations in a Secondary School, London, Routledge and Kegan Paul.

Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed Methods Research: A Research Paradigm Whose Time Has Come. Educational Researcher, 33(7), 14-26.

Lacey, C. Hightown Grammar, Manchester, Manchester University Press

Maxwell, J. and Mittapalli, K. (2010). Realism as a stance for mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of Mixed Methods in Social and Behavioral Research. Second edition. Thousand Oaks, CA: Sage Publications.

Mertens, D., Bledsoe, K., Sullivan, M., and Wilson, A. (2010) ‘Utilization of mixed methods for transformative purposes’. In A. Tashakkori & C. Teddlie (Eds.), Handbook of Mixed Methods in Social and Behavioral Research. Second edition. Thousand Oaks, CA: Sage Publications.

On the radical critique of interviews, see Hammersley, M. (2008) Questioning Qualitative Inquiry, London, Sage, Chapter 5.