qualitative program evaluation research: employing the complementary analysis research matrix...
TRANSCRIPT
QUALITATIVE PROGRAM EVALUATION RESEARCH: EMPLOYING THE COMPLEMENTARY ANALYSIS RESEARCH MATRIX APPLICATION (CARMA)
LeAnn Grogan Putney, PhDEducational Psychology & Higher Education
Professor and Department ChairUniversity of Nevada, Las Vegas
Joan Wink, PhDProfessor Emerita
California State University, Stanislaus
EQRC – Las Vegas, NVFebruary 11, 2014
YOU DO KNOW THESE HEAPS OF DATA NEED MORE ANALYSIS, RIGHT?
…I TOLD YOU TO USE CARMA TO GET OUT OF THE HEAPS AND OVER TO CONCEPTS!
3
Vygotsky’s Conceptual Development – AKA, THE HEAPS
Heaps Complexes Concepts
Generalized understandingsGroup similar items
Connections among items
4
Critical Action/Complementary Analysis Research Matrix Application (CARMA)
CARMA helps you move from Heaps of Data to Complexes of Relationships toConcepts related to a theoretical
perspective or producing a grounded theory
Putney, L. G., Wink, J & Perkins, P. (2006). Teachers as Researchers: Using the Critical Action Research Matrix Application for Reflexive Classroom Inquiry. Florida Journal of Teacher Education, Vol IX, p. 23 - 35.
NOTE-TAKINGEXPECTATIONSWho are the intended participants?Who is supposed to provide what service?How are participants to be served?What will be produced in the activity?
NOTE-TAKINGEVIDENT IMPLEMENTATIONWho are the actual participants?Who is actually providing what service?How are participants actually served?What is actually being produced?
NOTE-MAKING / ALIGNMENT OR DEPARTURE? Compare/contrast expected with evident using Action Researcher Interpretations For each question do you see alignment or departure?oWhat might be the implications?
NOTE-REMAKING / RECOMMENDATIONS/Modify or maintain program activity?In what ways?For which participants?With what intended outcomes?
6
Complementary Analysis Research Matrix Application [CARMA]
7
Complementary Analysis Research Matrix Application (CARMA)
Content of fieldnotes NoteTaking
Descriptions of setting/people/activities Mapping of physical space Direct quotes or substance of what was said
Note Making Observer comments in margins –
interpretations & questions Note Remaking
What did you learn – maintain or modify? Reflective Cycle – transformative stage
8
Potential Uses for CARMA
Program evaluation Applied classroom research
Self study (action research) or Team approach with classroom teacher
Educational research School wide behavioral practice Single-sex based classes Comparison of several schools – English Language
Learners in high ranking schools for “best practices”Policy analysisCritical incident study
9
Scale of use
Micro
Macro
Lesson Classroom
School District Community
State National
Continental
Global
Unit/Project/Lesson/Guest speaker (Teacher’s, Speaker’s, and Students’ expectations)
Classroom project/Tolerance
School-wide project/Action Research – 111 teachers
District-wide project/Empowerment
Community-wide/Hospice – EPY 716 2008/Critical Incident dissertation
10
Scale of use
State-wide project/Proposition 227 in California- dissertation
National-level project/No child Left Behind policy analysis – dissertation
Continental level/Quality Assurance in European Higher Education Area - policy analysis – dissertation
Global level/recommendations for future research studies
Micro Macro
Lesson Classroom School District Community State National Continental
Global
PARTICIPANT DISCUSSIONFOR YOUR OWN RESEARCH, WHAT WAS YOUR
INTENDED VS ENACTED?
DID YOU ENCOUNTER WHAT YOU EXPECTED?
DO YOU NEED TO COLLECT MORE DATA OR ARE YOU SEEING REPEATING PATTERNS?
11
CARMA APPLIED TO YOU
12
Complementary Analysis Research Matrix Application (CARMA)
WHAT - Tool for critical evaluation, to help the researcher/evaluator to describe and interpret what is being accomplished in the setting, and in what ways the cultural space is being
utilized by the participants, by measuring program expectations, against
evident implementation of practices and outcomes,
from the perspectives of all levels of participants
(Putney, Wink & Perkins, 2006).
13
Complementary Analysis Research Matrix Application (CARMA)
WHY - Roots of CARMA:Vygotskian view of life as a social
construction, steeped in the context of our cultural experiences to evaluate whether the context of a
program is meeting the needs of the constituents, we would have to determine what the constituents and program administrators are socially constructing through the program implementation
.
14
Complementary Analysis Research Matrix Application (CARMA)
WHY - Roots of CARMA:Ethnographic perspective of
learning from people about their cultural patterns of interaction (Agar, SBCDG) people, who work in a common setting
for individual yet related purposes, begin to co-construct norms and expectations for being together in their local time and space
.
15
Complementary Analysis Research Matrix Application (CARMA)
HOW – Observations and Interviews of constituents
Fieldnotes – written account of observations Continuous notes, jottings, head notes,“cooked up”
TEACHER’S REFLECTIONS ON HER OWN PRACTICE
16
Example from action research
Putney, L. G., Wink, J & Perkins, P. (2006). Teachers as Researchers: Using the Critical Action Research Matrix Application for Reflexive Classroom Inquiry. Florida Journal of Teacher Education, Vol IX, p. 23 - 35.
Step 1 – NoteTaking Notations on classroom expectations
Step 2 – Notetaking from observations what is evident
Step 3 – results and conclusions after contrastive analysis
20
Step 4 - Recommendations - decisions