qualitative data is information which does not present itself in numerical form and is descriptive,...
TRANSCRIPT
Qualitative Assessment: A Hands-
On Introduction
Miriam Segura-TottenBSP Research Residency
Institute July 23, 2015
Qualitative data is information which does not present itself in numerical form and is descriptive, appearing mostly in conversational or narrative form.
Words, phrases, text…
Definition
Notebooks Open-ended questions Papers Journal entries On-line discussions, blogs Email Twitter/ ‘tweets’ Notes from observations Responses from interviews and focus groups
What are some sources of qualitative data?
Qualitative analysis is the “interplay between researchers and
data.”
Researcher and analysis are “inextricably linked.”
Qualitative Data Analysis
Inductive process◦ Grounded Theory (“Open coding”)
Unsure of what you’re looking for, what you’ll find No assumptions No literature review at the beginning
Deductive process◦ Theory driven
Know the categories or themes using rubric, taxonomy Looking for confirming and disconfirming evidence Question and analysis informed by the literature,
“theory”
Qualitative Data Analysis
Coding process: ◦ Conceptualizing, reducing, elaborating and
relating text– i.e., words, phrases, sentences, paragraphs.
Building themes:◦ Codes are categorized thematically to describe or
explain phenomenon.
Definitions: Coding and Themes
What have students learned from a particular activity?
What is the mental process of a student engaged in a particular intervention?
What are the key transitions that occur as undergraduates acquire metacognitive-regulation skills? (Stanton et al., 2015)
Example Research Questions
Read through the student reflection paper and highlight words, parts of sentences, and/or whole sentences with some “code” attached and identified to those sections.
Let’s Code #1
Read through this reflection paper and code based on this question:
What does the term "scientific research" mean?
Let’s Code #2
Our prompt for students’ reflective papers:
What does the term "scientific research" mean to you? What do you
think doing scientific research entails?
Thematic analysis procedure:
1. Several individuals code material (triangulation).
2. Meet to decide on common set of terms.
3. Code again using common set.
4. Can go through steps 1-3 several times.
5. Analyze codes: qualitatively and/or quantitatively.
Let’s look at the Stanton paper:
What is the research question of this study?
What are the key transitions that occur as undergraduates acquire metacognitive-regulation skills? (Stanton et al., 2015)
Why is qualitative analysis suited for looking at this research question?
Let’s look at the Stanton paper:Background
Self-regulated learning (metacognition imp):• understand what the task involves• ID strengths and weaknesses• Create plan for completing task (planning)• Monitor how well plan is working (monitoring)• Evaluate and adjust plan as needed
(evaluating)
Let’s look at the Stanton paper:
What was the experimental approach of this study?
Self-evaluation assignments after two tests
E1-SEE2-FT
Prompts
The questions attempt to get at whether students are planning, monitoring and evaluating.
Code student answers.
Let’s look at the Stanton paper:How did the authors approach the coding of student answers?
With “metacognitive regulation in mind”.Theory-driven, not open coding.
Let’s look at the Stanton paper:
How did the authors approach the coding of student answers?
E1-SE: They developed a coding system (“sufficient/provides evidence” or “insufficient/provides no evidence”).
Applied this to planning, monitoring, evaluating (self-regulated learning/metacognition).
Let’s look at the Stanton paper:
How did the authors approach the coding of student answers?
E2-FT: Coded for evidence that students followed study plan developed. Coding system (“yes/followed plan” or “no/did not follow plan”).
Applied this to planning, monitoring, evaluating (self-regulated learning/metacognition).
Let’s look at the Stanton paper:
Analysis of student answers led authors to propose a continuum of metacognitive regulation
What does this mean for your study?• Don’t be scared! Most projects do not require this type
of depth of qualitative analysis.• Many studies can benefit from mixed methods (qual &
quant). • If you want to know student opinions, analyze long
answer questions in exams, or essays, you may want to consider qualitative analysis.
Use mixed methods, multiple sources. Triangulate your data whenever possible. Ask others to review your design
methodology, observations, data, analysis, and interpretations (e.g., inter-rater reliability).
Note limitations of your study whenever possible.
Ensuring “validity” and “reliability” in your research
Read the literature: What is accepted in your field?
Read books or articles on qualitative research
Collaborate with a qualitative researcher Or, just pick their brain over coffee
How to get started in qualitative research
• Designing and Conducting Mixed Methods Research, Creswell, J.W., and Plano Clark, V.L., 2006, Sage Publications.
• Discipline-Based Education Research: A Scientist’s Guide, Slater, S.J., Slater, T.F., and Bailey, J.M., 2010, WH Freeman.
• Educational Researchers: Living with a Lesser Form of Knowledge, Labaree, D.L., 1998, Educational Researcher, 27(8), 4-12.
• Processing Field Notes: Coding and Memoing. In Writing Ethnographic Fieldnotes, 1995, Emerson, R.M., Fretz, R.I., and Shaw, L.L. (pp. 142-168). Chicago: The University of Chicago Press.
References