research design: the backbone of academic inquiry peter neff – doshisha university matthew apple...
TRANSCRIPT
Research design: The backbone of academic inquiry
Peter Neff – Doshisha UniversityMatthew Apple – Nara National College of TechnologyDavid Beglar – Temple University Japan
CUE Forum 2008
Olympic Memorial Youth CenterNovember 1, 2008, 1:15 - 2:50 p.m.
Introduction: The importance of good research design
Approaching the studyDeveloping the study design
Break for Q&A
Designing the right instrumentImplementing your designConclusion
Further Q&A
Overview
Introduction: The importance of good research design
Poor design vs. Good design
Poorly-designed studyHit on an idea, dive right inNo background researchThrow together a survey, give to a
group of unwary participantsCollect data, then ponder how to
analyzeRun to a colleague for helpFish around for most “interesting”
findingsPray to get published
Poor design vs. Good design
Well-designed studyHit on an idea, do background researchFormulate relevant, specific, practical RQsConsider participants, context, data analysis
in advanceDecide/develop instrument; pilot and revise
itDecide on appropriate pre/post-test
instrumentsPlan stages and structure of data collectionPrepare participants adequately…Then carry out the study
The importance of good designA well-designed study provides
many benefits:– Demonstrates researcher
knowledge– Ties the study to an underlying
philosophy– Provides a clear path for the
researcher(s)– Helps avoid mishaps of previous
studies
The importance of good designOther benefits
– Leads to more concrete results, more definitive conclusions
– Improves chances of publication– Raises the status of SLA as a
field of inquiry
A word about mixed methods designsThe great quan-qual debateMixed methods – the “best” of
both worldsAdd a qualitative component to a
quantitatively-oriented study: – Participant interviews– Observational, audiovisual data– Open-ended survey questions
Plan for qualitative analyses (text analysis, response coding)
Part 1Approaching the Study
Approaching the study
Hitting upon research ideas
Review of the literature
Formulating research questions
Hitting upon research ideasIdentify the topic in a few words
Reflect on “doability” of research– Can I research this?– Should I research this?– Am I interested in researching this?
Review of the literature can help redefine and revise ideas
Identifying the topic: Hints for starting to narrowPose a short question using “what”
or “how” Write a short title that consists of
one sentence under 12 wordsAsk a friend or colleague to read
your topic and gauge their reactionsDraft research questions to see if
the topic can be adequately explored
A “researchable” topic
“Can I do this in my current situation?”
“Does this concern people at other institutions?”
“Does this add to the current body of research related to this topic?”
“Does this study contribute something from a unique perspective?”
Filtering “probably not so good” ideas: To boldly go where no research has
gone before… (The “Star Trek” idea)
My theory is clearly better than X (The “Steven Krashen is so wrong” idea)
My classroom is totally unique (The “I don’t need theory” idea)
This is a really cool technology / methodology / text book (The “I am primarily a teacher” idea)
Filtering ideas: A few hintsReview research designs and
statistical techniquesReview teaching methods and
overall SLA research resultsEvaluate access to potential study
participantsPlan time for material creation,
study design, and implementation
Review of the literature
Relate the study to continuing “dialogue” in current research
Finding a “gap” in the literature
Provide a framework for the importance of the study
Review of the Literature:Finding a “gap” in knowledge“We do not enough about X…”
“This way of looking at X has never been done…”
“This way of learning about X has not been duplicated in my context”
“Previous research has inadequately explored X…”
Finding literature: Some hints Google Scholar using key words or
researcher names Scour recent literature review articles Check for “cited” numbers online Get access to university databases Refer to recently published articles
– After the year 2000– During the previous 2 to 3 years
Examine “outside the field” articles
Finding literature: Separating the wheat…“Top tier” journal articlesMost-often-cited articlesRecent articlesResearch articles (not reviews)Books / Edited book-articlesMajor international conference
papersDissertations / dissertation abstracts
…from the chaff
“In-house” journal articlesArticles from “proceedings” booksOnline journal articles with only .html
versionsUnedited books from small publishersNewspaper and magazine articlesWeb pagesAnecdotal evidenceYour own previous papers for an MA
course
Research questions: A few useful guidelinesNaturally flow from the literature
reviewStrongly connected to the topicAt least two or three (not one)……but not five or six or moreAs specific as possibleDirectly concern variables in the studyDo not contain yes/no question words
RQs: What not to ask
“Is X true/false?”“Will X happen if…?”“Does X cause Y?”
“What do participants think of X?”“Why does X happen?”
RQs: What to ask
“What differences exist between…”“Compared to X, how does Y…?”“To what degree do X and Y
differ…?”“When X is controlled for Y…, how
does Z…?”
“What are underlying patterns among…?”
“To what degree does X predict Y?”
Part 2Developing the Study Design
Research design
Cross-sectional design: A design in which data are collected from a sample at only one point in time.
Longitudinal design: A design in which data are collected at more than one point in time.
Randomized Control-Group Pretest-Posttest Design
Experimental Group 1 T1 Xa (Method a) T2
Experimental Group 2 T1 Xb (Method b) T2
Control Group T1 T2
Randomized Control-Group Pretest-Posttest Design
Reasonably strong conclusions can be reached about the effects of the treatments.
Problem 1: Within session variation (e.g., different teachers or room conditions) may intervene.
The solution? Randomly assigning participants, times, and places to the experimental and control conditions.
Problem 2: The pretest may interact with the treatment. This potential problem is dealt with in the next design.
Randomized Solomon Six-Group Design
Pretested (Random assignment) T1 Xa (Method a)T2
Pretested (Random assignment) T1 Xb (Method b)T2
Pretested (Random assignment) T1
T2
Unpretested (Random assignment) Xa (Method a)T2
Unpretested (Random assignment) Xb (Method b)T2
Unpretested (Random assignment)T2
Randomized Solomon Six-Group Design
This design amounts to doing the experiment twice –once with and once without pretesting.
It is possible to know what effects, if any, are associated with pretesting.
If the results of the “two experiments” are consistent, greater confidence can be placed in the findings.
Counterbalanced Design
This design is useful when randomization is not possible and intact groups must be used.
Replication
Xa Xb Xc Xd
1 A B C D
2 B D A A
3 C A D B
4 D C B C
Counterbalanced design
The counterbalanced design rotates out the participants’ differences (e.g., one group has more aptitude or motivation than the other groups) by exposing each group to all variations of the treatment.
Order-of-presentation effects are controlled.
Primary weakness: The possibility of carryover effects from one treatment to the next exists. Allowing time between treatments can alleviate this problem.
Control-Group Time-Series Design
Experimental Group 1 T1 T2 T3 T4 Xa (Method a) T5 T6 T7
T8
Experimental Group 2 T1 T2 T3 T4 Xb (Method b) T5 T6 T7
T8
Control Group T1 T2 T3 T4 T5 T6 T7 T8
Control-Group Time-Series Design
This design allows the researcher to determine growth over time, and the effect of an intervention.
The presence of a control group increases the trustworthiness of the results because the possibility of a contemporary event causing any gains can be determined.
Control-Group Time-Series Design
This design can be extended by exposing the participants to the intervention on multiple occasions.
This approach is more sensitive to partial gains in knowledge and tests the strength of the intervention more than once, thus giving the researcher a more accurate understanding of the effectiveness of the intervention.
Experimental Group 1 T1 T2 Xa T3 T4 Xa T5 T6
Experimental Group 2 T1 T2 Xb T3 T4 Xb T5 T6
Control Group T1 T2 T3 T4 T5 T6
Q&A Break
Part 3Designing the Right Instrument
Instrument Design
Commonly used instruments in SLA research– Scored tests– Rater scores– Surveys– Interviews
Consider your eventual data analysis when developing instruments
Instruments - Scored tests
Pluses Quantitative items
(M/C, Cloze/C-tests)– Simple to score large
# of participants– Easier to analyze
Qualitative items (short answer, timed essays)– Good complement to
quantitative scores– Can provide more in-
depth assessment of participants’ abilities
Minuses Quantitative items
– Limited to one type of data
Qualitative items– Take more
time/effort to score– Rater bias
Instruments – Performance ratingsAn assessment of participants’
performance in an assigned taskTasks may include presentations,
interviews, written essaysPerformances can be scored using a
Likert-scale, a rubric, or holisticallyUsually scored by at least two
“expert” raters; sometimes also by peers
Performance ratings
Rating criteria should be concretely established with little ambiguity
Avoid including too many (or too few) criteria for one performance task
All raters should undergo a “normative” training session prior to assessment
Use models to train ratersAvoid single-score holistic ratings
Instruments - Surveys
Often used for:– Collecting learner history data (L2
study experience, other background info)
– Assessing participants’ attitudes towards a predetermined construct (language learning motivation, anxiety using the L2)
– Determining reactions to an experimental treatment (teaching methods, innovative learning tasks)
Survey makingFor non-advanced learners –
surveys should be in their L1Build in redundancy - Include
multiple questions for each concept area
Questions should be simply worded – avoid negative or confusing wording
Depending on the purpose of the survey, 20-40 items/session is a good range to shoot for
Survey making
Any survey used in a serious study should be piloted in advance
It is acceptable to make adjustments to an existing instrument
Likert-scale items should usually have between 4 and 6 choices
A few qualitative questions can provide a nice complement to quantitative instruments
Instruments - InterviewsInterviews can provide an
excellent qualitative component to a larger study
It is not necessary to interview all participants – a subsample as small as 10-20% can
be acceptableUse your best judgment on
participants’ language ability– For intermediate-and-above learners,
L2 interviews are often fine
Conducting interviews
Inform students they are being interviewed, obtain consent
Record unobtrusively “Warm up” the participants before
getting into the heart of the interview
Collect more data than you need
Validating Instruments
Instrument Validity
The construct = The heart of the matter
What construct do you wish to measure?
How do you define the construct?What are its component parts? Do
they form a unified whole?
Operationalizing the construct: The items Conceptualize the construct as a
continuum: easy—difficult items and less able—more able persons.
How have other researchers measured the construct?
Write original or adapted items. Cover the estimated range of your
participants. Write 50% more items than you intend
to use. This will allow you to “cherry pick” the best items as well as items at various levels of difficulty.
Operationalizing the Construct: The ItemsMore able ｜ More difficultpersons ｜ items ｜ x ｜ item 1 xx ｜ item 2 item 3 xxx ｜ item 4 item 5 xxxx ｜ item 6 item 7 item 8 xxxx ｜ item 9 item 10 item 11 xxx ｜ item 12 item 13 xx ｜ item 14 x ｜ item 15 ｜Less able ｜ Less difficultPersons ｜ items
Operationalizing the Construct: The Items
After piloting the items, statistically analyze the results.
Examine dimensionality, item difficulty, and item content.
Select the best items to make an efficient, highly reliable instrument.
Part 4Implementing Your Design
Implementing the design
Including other researchers
Practical issues
Handling ethical concerns
Including other researchers in the studyThe nature of the researchers
involved– Main researcher plus “helpers”– One researcher plus “other participants”
The nature of the instructors involved– Teaching methods– Students taught– Course goals– University program goals
Working with other researchersWork with people you know and trustEstablish a schedule early Define clear roles for each researcherDecide definite research goals prior
to data collection Keep in regular contactThe “band practice time” principle
Example research roles
Head researcher / contact personData entry specialistStatisticianInterviewerLiterature analystEditor / proofreader
Heading off potential problems Explain study commitments prior to
starting the study Agree on “ownership” prior to data
collection and data entry– Who will keep the data?– Whose name comes first, second, etc.?
Keep everyone aware of deadlines Include everyone in decision-making
processed and data analysis
Things to avoid in group researchPeople you don’t knowResearch groups larger than four
or five membersUsing someone for language skillsInvolving others just to get more
participantsForgetting to thank others for
their assistance
Improving relations with research helpersWrite clear instructions
Thank profusely for their time and effort
Offer to send copies of final research papers and/or results
Offer to assist in future research
Keep in touch after research ends
Practical Issues
Timing of implementation
Learning and research context
Participant consent
Financial considerations
Timing of the implementationBeginning, middle, or end of
semesterDay of the weekTime of dayExams and exam preparation periods“Culture Festivals” or other club-
related events“Open classes” or “parents’ day”
Learning and research contextDiffering course goals (I.e.,
listening class vs. reading class)Different major field of studyGender, age, year in schoolNumber of class meetingsPerception of the value of
research by institution heads
Participant consent
Always allow for “non-participation” choice from potential participants
Write clear instructions for participants asking for their cooperation
Ask co-researchers or helpers to briefly inform participants about their choice
Financial considerations
Copies for questionnaires, exams, etc.
Computer analysis softwareMailing costsInterview travel costsRecording equipment
Reference booksJournal article costs
Heading off lack of cooperation problemsReview requirements of the study
– How many items in the questionnaire? – How many treatments?
Recognize that students are busy, tired, etc.
Plan to get between 30-50% more data than you need
Try to get more data at a later date
Ethical Considerations
Ethical considerations
Students should not be exploited just because they are there
In theory, they should derive benefit (directly or indirectly) from the research
Explain the basic purpose of your research before collecting data– 1-2 sentences should be fine
It is also good to briefly explain this at the beginning of any survey instrument
Ethics - ConsentProvide students with the chance to
opt out of participationVerbal consent is usually acceptable
for surveys, ratings, and test scoresA written consent form may be
necessary for more involved forms of participation (interviews, essay passages)
When in doubt, check recent articles in well-known journals for guidance
Ethics – Other considerationsThe role of this study within your
institutionPotential gender, proficiency or
other issues that may affect your data or conclusions
Have a plan for anonymizing the data (and consider making this conspicuous on your instruments)
Conclusion
In conclusion…There are many factors to consider when
embarking on a serious studySome points to take away…Most studies should fill a place (a “gap”)
within the current academic dialogue Research questions should reflect
continuity with the literature and should be specific
Carefully consider the design setup which will work best with the participant groups and the research and analytical goals
In conclusionFurther points…Different instruments work better in
different circumstances. Choose those which best reflect your aims.
Plan your analyses at the same time you are developing your instruments
Develop and pilot instruments which can cover responses from a range of participants
In conclusionEven more points…Work with others who will be serious
and committed, and then be an organized and conscientious leader
Consider how practical, pedagogical, and institutional concerns may affect your study
Do not forget current ethical guidelines for carrying out participant research
Good luck with your research!
Thank you for listening
Q&A
