asking the right questions: collecting and validating data – refining, rephrasing, and refocusing...
TRANSCRIPT
PROCEDE 2014
1
Asking the right questions: Collecting and validating data – refining, rephrasing, and refocusing
Dr. Corinne Haigh, The School of Education, Bishop’s University, [email protected]
PROCEDE 2014: People, Passion, Proof – The Data Difference
PROCEDE 2014
2
Session Outline
What makes a good research question? Where do I begin?
What is the danger of a single story?
How can I be sure that the measurements I take match the question I am trying to answer? How do we define what we wish to measure? What are reliability and validity?
What are some ethical issues to consider?
Pollev.com/chaighbu
From any browser
(code or keyword) <your response>
From a text message
Participating with Poll Everywhere: How to vote via the web or text messaging
22333
Participating with Poll EverywhereHow to vote via the web
Pollev.com/chaighbu
From any browser
Participating with Poll EverywhereHow to vote via texting
744402
From a text message
22333
PROCEDE 2014
10
Reflection
What types of questions was I able to ask using the polling software?
What type of information was I able to gather?
How could I analyze this information to answer a specific question?
How might I present it to others?
Take a moment to reflect on these questions individually.
PROCEDE 2014
13
Reflection
What types of questions was I able to ask using the polling software?
What type of information was I able to gather?
How could I analyze this information to answer a specific question?
How might I present it to others?
Take a moment to reflect on these questions individually.
PROCEDE 2014
14
Now, think of a recent “problem” you have encountered in your professional life…
Briefly describe it.
How did/might you approach it?
What did/do you hope to learn as a result?
Take a few minutes to reflect on these questions individually, and then compare notes with a colleague
PROCEDE 2014
15
The danger of a single story
Asking the right questions involves considering multiple perspectives
We will now watch one video that illustrates the danger of being exposed to only a single story And I’ve included a link to another for you to
watch at another time…
PROCEDE 2014
16
Chimamanda Adichie
Novelist Chimamanda Adichie tells the story of how she found her authentic cultural voice -- and warns that if we hear only a single story about another person or country, we risk a critical misunderstanding.
The Danger of a Single Story
PROCEDE 2014
17
Ben Goldacre
Doctor and epidemiologist Ben Goldacre shows us, at high speed, the ways evidence can be distorted, from the blindingly obvious nutrition claims to the very subtle tricks of the pharmaceutical industry.
Battling Bad Science
PROCEDE 2014
18
Checkpoint
Take 2-3 minutes and write about an educational example of the danger of a single story Try to write for the full time
You will then discuss your writing with a partner (or in a trio) and then with the whole group
PROCEDE 2014
19
Asking a question often starts from necessity
How do I know when…?
Think up some endings to my sentence starter.
The Assessment for Learning (AL) Cycle
1. What is known
2. What might be happening
and why
3. What you need to find
out
4. What you think is
happening
5. What you say
6. What you need to do
7. Did what you chose to
do work?
Gathering referral information
Generating hypotheses
Checking your hypotheses by collecting information
Arriving at tentative judgement
Reporting findings
Planning instruction
Evaluating success
Further information
needed?
20
PROCEDE 2014
PROCEDE 2014
21
Refining Questions
Initial questions may be vague, but vague questions can never be answered
Refine questions by: Consulting past research in the area Speaking with colleagues Applying theory Replicating someone else’s study Clarifying contradictory findings
PROCEDE 2014
22
What is a variable?
A variable is something that can be changed, such as a characteristic or value
We are generally looking at whether change in one variable results in change in another variable
Can you generate some examples of variables from your practice? What kind of relationships between these variables might be interesting to investigate?
Learning environment
Academic
Physical
Emotional
Social
Initial Concern
Gaps in learning; opportunity to learn; history of successes/failures; teachers; past and present programs
Vision; hearing; health history
Reading profiles; academic profile; experiences; cognitive development; aptitude (IQ); Language profile
Family; self concept; motivation; security
Culture; friends; maturity
23
PROCEDE 2014
PROCEDE 2014
24
Operationally defining a variable Specifies exactly how to measure and/or
manipulate the variables in a study
Go back to one of the examples you generated for the previous question – how would you define it? Remember, it must be measureable (think
about setting SMART goals for IEPs)
PROCEDE 2014
25
Types of variables
Independent vs. dependent Independent: precedes, influences, or predicts the
dependent variable (e.g., teaching method) Dependent: affected or predicted by the
independent variable (e.g., student engagement)
Extraneous and confounding Extraneous: source of error affecting the results
(e.g., time of day) Confounding: varies systematically with the
independent variable (e.g., the teacher)
PROCEDE 2014
26
How can we make sense of a complex process? We make inferences…
1.
2.
3.
PROCEDE 2014
27
BALLOON
- Recognize the picture as balloons
- Select the appropriate word
- Produce the appropriate articulation
What mental activity needs to occur to identify the picture and say what it is?
How is making an inference about learning different than blowing up a balloon?
- we cant see the “mental activity”
-Therefore, we must make inferences about the mental activity from observable behaviours.
PROCEDE 2014
Qualitative research
A means for exploring and understanding the meaning that individuals or groups ascribe to a social or human problem
Emphasizes natural settings, understanding, verbal narratives, and flexible designs
28
PROCEDE 2014
29
Qualitative techniques and data
Data: Verbal descriptions, field notes, observations, documents, photographs, people’s own words, narrative
Techniques: observation, participant observation, open-ended interviewing, review of documents and artifacts
PROCEDE 2014
Quantitative research
A means for testing objective theories by examining the relationship among variables
Emphasizes numbers, measurements, deductive logic, control, and experiments
30
PROCEDE 2014
31
Quantitative techniques and data
Data: quantiles, counts, numbers, statistics, measures/instruments
Techniques: experiments, quasi-experiments, structured observations, structured interviews, surveys
PROCEDE 2014
32
How do I know which to use?
It is critical to match the reason for the research with the appropriate method Determined by the purpose and the research
question
Using each method will be a challenge Either can be used well, or poorly
Depth of understanding is important, regardless of the design Leads to greater impact
PROCEDE 2014
33
Checkpoint
Think about some of the potential research questions you generated earlier – what sort of approach would be best suited to answering them? Consider the data you’d want to collect (or
existing data you’d like to access) and the techniques you could use to do so
PROCEDE 2014
Reliability
Test-retest reliability: Agreement between scores on the same test,
given to the same people, at different times
Inter-rater reliability Agreement among different raters or observers
Inter-item reliability Are all of the items on the test equally good at
assessing the same underlying ability?34
PROCEDE 2014
Validity
Content validity The range and relevance of the items Can be very difficult to measure for some more
complex aspects of learning
Construct validity Concerns the underlying theoretical construct: is the
test measuring the target skill or behaviour Test this by comparing scores in two different
measures – should be highly related if they’re measuring the same construct
35
PROCEDE 2014
Reliability and validity
36
PROCEDE 2014
37
Some broad ethical questions to consider:
Does answering this question benefit the individuals being studied? Will the answer be meaningful for others?
Are participants being marginalized or disempowered?
Are participants being deceived about the intent of the study?
Are participants at more than minimal risk? Minimal risk: “Research in which the probability and
magnitude of possible harms implied by participation in the research is no greater than those encountered by participants in those aspects of their everyday life that relate to the research.” (TCPS 2, Chapter 2)
PROCEDE 2014
38
Some broad ethical questions to consider:
Did all participants benefit from the “treatment”?
Is there a possibility that participants will disclose harmful and/or intimate information? What will be done with this information?
How will you protect the anonymity of participants?
PROCEDE 2014
39
Checkpoint
Describe an ethical question or concern you’ve had when trying to collect data to answer a specific question (or speculate about a potential ethical issue that may arise in future).
How did/could you address it?
First reflect on this individually and then have a discussion at your table.
PROCEDE 2014
40
Some final points for reflection… Sometimes coming up with a good research problem or
question is the hardest part
The approach you take to answering your question will depend on: The problem you’re trying to address Your worldview or assumptions about research The specific inquiry strategies you wish to use Your personal experience
There are multiple ways to approach data collection and analysis and no one is better than the other as long as you aware of their strengths and limitations
