texas a&m university epsy 642 fall 2009 victor l. willson, instructor
TRANSCRIPT
Texas A&M University EPSY 642FALL 2009
Victor L. Willson, Instructor
EPSY 642 Meta Analysis of Social Science Research Syllabus FALL 2009 TUESDAYS 4:45 – 7:35 Victor L. Willson, Professor: Office: by appt 704 Harrington 845-0904 /fax: 862-1256 email:v-
[email protected] Course description: Theory and methodology of current state of the
topic of meta-analysis of research documents. Course objectives: Students will learn meta-analysis methods by
conducting a meta-analysis on a circumscribed topic of their choice. Students will be required to present their findings at the end of the course and produce a research paper on their topic. Students will be encouraged to develop a potentially publishable research article from their work. Components for the project will be reported by students during the semester.
Texts: Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-
Analysis. Sage. Cooper, H. (2010). Research Synthesis and Meta-
Analysis. 4th Ed. Sage
Date Topic Reading Assignment due
SEP 1 Introduction: Purposes, Approaches HC 1 Activity: Online Google Scholar search 5 articles: samples description, variable list
8 NO CLASS 15 Problem finding, Study Selection LW2, HC 2-3 22 Study coding, menu elements LW4, 5, HC 4, handout
29 Outcome variable effect computation LW3, HC 6, handouts
OCT 6 Study Design variables HS 2, HC 5 13 Correlational study analysis LW 6-7 Experimental study analysis LW 7 #1 27 Findings cumulation and reporting HC 6-9, LW 8 #2 handouts & examples NOV 3 Advanced Modeling handouts #3 10-17 Individual meetings #4 Nov 10, #5
Nov 17 24 Presentation of student papers using powerpoint DEC 1 continuation of presentations as needed
Grades: Based on completion of intermediate assignments (20%) and quality of paper (80%)
U= < 60% S = 61% - 100% Intermediate assignments: 1 Proposed meta-analysis topic with preliminary list of at least 20 articles 2 Preliminary list of proposed variables to be extracted from articles 3 Proposed Coding menu for variables 4 Preliminary analysis of outcomes using Q method, funnel plot of one
outcome variable 5 Preliminary covariate analysis using QB and QW method (at least 1
covariate) Project and paper: conduct a meta-analysis using the procedures of this course
on a topic of your choice; limit the number of studies to be finally included to no more than about 20 but at least 8 as appropriate to the topic. Produce a finished paper in APA format summarizing the procedures and results; produce a powerpoint of your paper for presentation.
In order to receive a grade a presentation must be made on the dates listed;
otherwise an incomplete will be given and a presentation to me and a group of students must be made before a final grade can be submitted. Final papers may be turned in after the last day to turn in grades for the semester with a grade of I given and no penalty, but after a presentation has been made and all intermediate assignments are completed. After one semester a grade of F will be given if these conditions are not met.
Note: The handouts and web-based files used in this course are copyrighted. By “handouts” I mean all materials generated for this class, which includes but is not limited to syllabi, quizzes, exams, lab problems, in-class materials, review sheets, and additional problem sets, in paper or electronic form. Because these materials are copyrighted, you do not have the right to copy the handouts unless I expressly grant permission.
As commonly defined, plagiarism consists of passing off as one’s own ideas, words, writings, etc. which belong to another. In accordance with this definition, you are committing plagiarism if you copy the work of another person and turn it in as your own, even if you should have the permission of that person. Plagiarism is one of the worst academic sins, for the plagiarist destroys the trust among colleagues, without which research cannot be safely communicated.
If you have any questions regarding plagiarism, please consult the latest issue of the Texas A&M University Student Rules, under the section “Scholastic Dishonesty”. Honor Code and refers the student to the Honor Council Rules and Procedures on the web:
http://www.tamu.edu/aggiehonor.) “An Aggie does not lie, cheat or steal, or tolerate those who do.”
Purposes for Meta-Analysis• Cumulate findings of studies on a particular
topic• Examine homogeneity of outcomes• Estimate effects of independent variables on
outcomes in a standardized format– Evaluate moderator and mediator effects on
outcomes– Differentiate different types or classes of
outcome effects
Historical backgroundCriticism of traditional narrative reviews of
researchExasperation in social sciences with
constructs measured different ways in terms of determining consistencies
Need to formulate theoretical relationships based on many studies
History part 2Early 1970s efforts focused on significance
testing and “vote counts” of significanceGlass (1976) presented a method he called
“meta-anlaysis” in Am. Ed. Research Assn. presidential address
Others proposed related methods, but Glass and colleagues developed the most widely used approach (Glass, McGaw, & Smith, 1981)
Meta-Analysis as Survey Research• Research articles as unit of focus• Population defined
– Conditions for inclusion of articles• Data requirements needed for inclusion• Completeness of data available in article or estimable• Publication sources available, selected
• Sample vs. population acquisition– Availability of publications and cost– Time to acquisition (how long to wait for
retrieval)
Strengths of Meta-AnalysisDefinition of effect and effect size beyond
“significant or not”Focus on selection threats in traditional
reviews (bias in selection of articles for review)
Systematic consideration of potential mediators and moderators of effects
Data organization of articles for public review
Weaknesses of Meta-AnalysisMethodologically sophisticated and expensivePotential ignoring of contextual effects not
easily quantified; eg. historical/environmental placement of research
Potential improper mixing of studiesAverages hiding important subgroupingsImproperly weighting studies with different
methodological strength/rigor
Research focus for meta-analysis
Defining and delineating the constructDetermining a research outletMeta-Analysis as an interactive, developing
process
Recent CriticismSuri & Clarke (2009): Advancements in
Research Synthesis Methods: From a Methodologically Inclusive Perspective (Review of Educational Research, pp. 395-430)
They propose 6 overlapping approaches:1. Statistical research syntheses (eg. meta-
analysis)2. Systematic reviews3. Qualitative research syntheses4. Qualitative syntheses of qualitative and
quantitative research5. Critical impetus in reviewing research6. Exemplary syntheses
Some critical comments on Suri & Clarke (2009) Systematic reviews- original Glass criticisms
hold: what is the basis for inclusion and exclusion; why are certain articles privileged?
Qualitative research syntheses- how can these be done with situated contexts, small samples, environmentally-developed variables, sources, etc.? Will there be a review for every reader, or for every researcher? Same limitation as all qual research
Qual syntheses of quant and qual research- potentially doable, with an alternating order: qual first to focus emphases in the quant analysis, or quant first to be validated with the qual studies of particular environments and populations- do they fit/match in reasonable ways?
Critical impetus- code words for critical theory/Marxist etc. Answer is already known, why do the research?
Exemplary syntheses- what is the purpose?
Defining and Delineating the Research Topic• Outcome construct definition
– Importance to the field to know what has been learned– How big is it? How many potential studies? – Conduct preliminary searches using various databases
• Refining the construct– How much resource is available? Eg. 1000 studies = 2-
3 years work – Are there specific sub-constructs more important than
others? Select them or one of them– Are there time-limitations (no studies before 19xx)– Are there too few studies for the given construct,
should it be broadened? Too few-> less than 10?
Defining and Delineating the Research TopicWhat is the typical research approach for the
topic area?All quantitativeAll qualitativeMixed quantitative and qualitative
Are there sufficient quantitative studies to provide evidence for findings?Can qualitative studies be included as a
separate part of the study? How?
Determining Research Outlet• Does the proposed journal
– publish research on the construct?– Publish reviews or meta-analyses?
• Is there a journal devoted to reviews that your project would fit with?
• Has a recent similar meta-analysis been published? If so, will yours add anything new?– Ex. Allen, et al (under review) evaluated articles on
first grade retention after 1990 focusing on the quality of the research design in each study to determine if the effects were different from a fairly recent meta-analysis by Jimerson (2001)
Meta-Analysis as an interactive, developing process• View meta-analysis as evolutionary
– As studies are reviewed and included, purpose and scope may change
• Assume initial conceptualizations about both outcomes and potential predictors may change over time– Definitions, instruments, coding may all change
as studies are found and included
• Plan for revisions to all aspects of the meta-analysis
ACTIVITYGet into groups of 3 or 4Use GOOGLE SCHOLAR to find research
articles from the following:Group 1: elementary school grade retentionGroup 2: teenage depression treatmentsGroup 3: IQ and socioeconomic statusGroups 4+: one of above
Select 5 articles that have empirical dataFor each article
define the population being samples List with specific names the variables used in the study Discuss within your group the similarities and differences of
the populations and variables; consider difficulties you would have in summarizing research for these
Plan to report to the class what you found