DATA COLLECTION AND ANALYSIS TCUP Fundamentals of Education Research Workshop Mercy Mugo October 31, 2014

Quality Education for Minorities (QEM) Network

The Research Linkages

Identify the Research Question(s)

Determine the Research Methodology

Collect the Data

Data Analysis and Report of Findings

What is Data Collection?

¨  A detailed plan of procedures that aims to gather data for the purpose of answering a research question(s)

Quantitative Data

¨  Collected in standardized manner ¨  Presented in numerical format ¨  Analyzed using statistical techniques ¨  Results more generalizable

Qualitative Data

¨  Collected in natural setting ¨  Presented in narrative format ¨  Thick in detail and description ¨  Analysis often emphasizes understanding

phenomena as they exist

Data Collection Methods

♦ Surveys ♦ Tests ♦ Rubrics ♦ Checklists

♦ Observations ♦ Interviews ♦ Focus Groups ♦ Document Review/Analysis ♦ Case Studies ♦ Photographs, Videos

Qualitative Methods Quantitative Methods

Cultural Responsive Data Collection Methods

¨  Talking Circles ¨  Visiting ¨  Performance-based Assessment ¨  Appreciative Inquiry

Pros and Cons of Quantitative Methods

Method Advantages Disadvantages

Surveys •  Inexpensive •  Good for gathering

descriptive data •  Cover a wide range of

topics •  A variety of software for

analysis

•  Self-report may lead to bias

•  Data lack depth •  No control for

misunderstood questions, missing data

Pros and Cons of Quantitative Methods

Method Advantages Disadvantages

Tests •  Objective information on what the test taker knows and can do

•  Can be constructed to match level of skills

•  Easy to administer/score •  Provide “hard” data •  Accepted by public

•  Time consuming •  Biased against some

groups •  May be subject to

corruption via coaching or cheating

Pros and Cons of Qualitative Methods Method Advantages Disadvantage

Interviews •  Yield richest data, insights

•  Permit face-to-face contact

•  In-depth exploration of topics

•  Allow interview to explain or clarify questions

•  Expensive and time consuming

•  Need well trained interviewers

•  Interviewee may distort information

•  Large volume of information may be difficult to transcribe

Pros and Cons of Qualitative Methods Method Advantages Disadvantage

Focus Groups

•  Useful to gather different viewpoints and new insight

•  Less time required •  Subject matter is not

sensitive

•  Not suitable for generalization

•  Require qualified facilitator

•  Peer pressure may inhibit responses

Pros and Cons of Qualitative Methods Method Advantages Disadvantage

Observations •  Provide direct information

•  Permit observer to enter into and understand context

•  Exist in natural, flexible setting

•  Good for identifying unanticipated outcomes

•  Expensive •  Time consuming •  Need well qualified

observers •  Selective perception of

observer may distort data •  Behavior observed may not

be representative of a group/situation

Pros and Cons of Qualitative Methods Method Advantages Disadvantage

Document Analysis

•  Inexpensive •  Available locally •  Grounded in setting

and language in which they occur

•  Provide information on historical trends

•  May be incomplete •  May be inaccurate •  Challenges locating

suitable documents •  Time consuming

When Choosing Methods, Consider…

¨  Purpose of study (research questions) ¨  Respondents/data sources ¨  Resources available ¨  Type of information needed ¨  Value of using multiple methods ¨  Importance of ensuring cultural appropriateness

Other Considerations………..

¨  Understand the community ¨  Involve community ¨  Allow time to establish relationships ¨  Take care in constructing and asking questions ¨  Respect cultural protocols ¨  Provide incentives

DATA ANALYSIS

What is Data Analysis?

¨  Summarizing data into manageable format to communicate its meaning

¨  Reflecting on the data and searching for patterns

¨  Seeking out the story in the results

Types of Statistics in Quantitative Research

¤  used to organize, describe, and summarize a set of data.

Descriptive Statistics Inferential Statistics

¨  used to draw inferences about characteristics of a population based on what is known about a sample drawn from that population.

Types of Statistical Analyses in Quantitative Research

¨  Measures of central tendency ¤ Mean, median, mode

¨  Measures of variability ¤  Range, variance,

standard deviation

Descriptive Statistics Inferential Statistics

¨  Parametric Tests ¤  t-tests, Analysis of variance

(ANOVA), Regression analysis

¨  Non-parametric Tests ¤  Chi-Square test; the sign

test

Data Analysis in Qualitative Research

¨  Read all data, get a sense of the whole ¨  Code data, tag items with same meaning using

using unique codes ¨  Identify patterns/themes among the codes ¨  Represent themes (writing, visual, etc.) ¨  Interpret and make meaning out of the data

Qualitative Data Analysis

Mixed Methods Designs

¨  Utilizes both quantitative and qualitative data collection methodologies

¨  Major designs: ¤ Convergent Design ¤ Explanatory design ¤ Exploratory design ¤ Embedded Design

Convergent Design

¤ Collect qualitative and quantitative data concurrently

¤ Analyze the two datasets separately ¤ Mix the two databases by merging results

during interpretation

Creswell. J (2012): Borrowed from Abraham S. Fischler, Presentation Mixed Methods

Explanatory Design ¤ Starts by collecting and analyzing quantitative

data ¤ Uses quantitative results to inform subsequent

qualitative inquiry ¤ Uses quantitative results to shape the qualitative

research questions, sampling, and data collection

Creswell. J (2012): Borrowed from Abraham S. Fischler, Presentation Mixed Methods

Exploratory Design ¤ Starts by collecting and analyzing qualitative

data ¤ Utilizes qualitative results to build the quantitative

phase ¤ Connects the phases by using qualitative results to

shape the quantitative research questions, variables, and instrument

Creswell. J (2012): Borrowed from Abraham S. Fischler, Presentation Mixed Methods

Embedded Design

 

Qualitative (or quantitative) Data Collection and Analysis

(before, during, or after)

Quantitative (or Qualitative) Design 

Quantitative (or Qualitative) Data Collection and Analysis Interpretation

Validity of Results ¨  Validity

¤ Most important characteristic of the study/assessment results

¤  Concerned with the appropriateness of the interpretations made from assessment results

¤ Specific to the interpretation being made and to the group being assessed

Reliability of Results

¨ Reliability ¤ Concerned with how well the results can be

replicated ¤ Would a particular technique (or survey) yield

the same results each time? ¤ Reliability does not ensure accuracy

Working with Identifiable Data

¨  Maintaining confidentiality

¨  Presenting accurate information ¨  How can we reconcile these two conflicting

dynamics?

Ways to Address Small Sample Size

¨  Report results in aggregate --- across several samples to maintain confidentiality

¨  Allow access of raw data to user who may need it for e.g. evidence-based decision-making, policy-making, budgeting, proposal development, etc. ¤ IRB approval ¤ Participants permission to release information/data

Example: Impact of Suppression of Small Data Cells

¨  Negatively affect underrepresented groups (loss of information)

¨  Value of data significantly diminished ¨  Prevent access to information essential to

providing highly needed opportunities A report on the series of outreach meetings on the “Impact of the Suppression of Small Data Cells” in the Survey of Earned Doctorates (SED) Report (2009). Prepared by QEM for NSF’s Science Resources Statistics (SRS) Division.

Impact of Suppression Continued…

¨  Harm diversity-focused initiatives and minority-focused programs

¨  Difficult finding role models ¨  Difficult designing intervention strategies

A report on the series of outreach meetings on the “Impact of the Suppression of Small Data Cells” in the Survey of Earned Doctorates (SED) Report (2009). Prepared by QEM for NSF’s Science Resources Statistics (SRS) Division.

Questions

Thank you!