opre2019 methodsmeeting resourcelist · opre2019 methodsmeeting resourcelist opre report #2020-131...
TRANSCRIPT
OPRE 2019 Methods Meeting Resource List
OPRE REPORT #2020-131
D. HANSEN AND R. HOLZWART
OCTOBER 2020
This document provides a list of resources for readers who wish to learn more about open
science methods. It was developed following the 2019 methods meeting, Methods for
Promoting Open Science in Social Policy Research, organized by the Office of Planning,
Research, and Evaluation (OPRE) in the U.S. Department of Health and Human Services’
Administration for Children and Families.1
1 See https:// opremethodsmeeting.org/ for additional information on OPRE’s methods meetings, including agendas and meeting products.
The resources were compiled from the meeting topic
memorandum, speakers’ slides, Q&A sessions at the meeting, and follow-up input from the
working group members who helped plan the event. They are divided into categories by topic
and can be accessed by clicking the blue hyperlink in each citation.
A. BACKGROUND INFORMATION
The resources in this section introduce the Federal initiatives and policies that support open
science, describe the motivation for promoting open science practices in social science
research, and provide an orientation to open science concepts.
Federal Initiatives and Policies Supporting Open Science
1. Evaluation Policy; Cooperative Research or Demonstration Projects, 79 F.R. 51574.
(2014).
2. Foundations of Evidence-based Policymaking Act of 2018, Pub. L. 115-435. (2019).
3. Holdren, J. P. (2013). Memorandum for the heads of executive departments and
agencies: Increasing access to the results of federally funded scientific research (p. 3).
Executive Office of the President, Office of Science and Technology Policy.
4. Vought, R. T. (2020). Memorandum for the heads of executive departments and
agencies: Phase 4 implementation of the Foundations of Evidence-based Policymaking
Act of 2018: Program evaluation standards and practices. Executive Office of the
President, Office of Management and Budget.
OPRE 2019 Methods Meeting Resource List 2
Motivation for Promoting Open Science in Social Science Research
5. Christensen, G. S., & Miguel, E. (2018). Transparency, reproducibility, and the credibility
of economics research. Journal of Economic Literature, 56(3), 920–980.
6. Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries.
Scientometrics, 90(3), 891–904.
7. Funk, C., Hefferon, M., Kennedy, B., & Johnson, C. (2019). Trust and mistrust in
Americans’ views of scientific experts. Pew Research Center.
8. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of
questionable research practices with incentives for truth telling. Psychological Science,
23(5), 524–532.
9. Leek, J. T., & Peng, R. D. (2015). Statistics: P values are just the tip of the iceberg.
Nature, 520(7549), 612–612.
10. Nuijten, M. B., Hartgerink, C. H., van Assen, M. A., Epskamp, S., & Wicherts, J. M.
(2016). The prevalence of statistical reporting errors in psychology (1985–2013).
Behavior Research Methods, 48(4), 1205–1226.
Orientation to Open Science Concepts
11. Bipartisan Policy Center. (2017). The promise of evidence-based policymaking.
12. Fecher, B., & Friesike, S. (2014). Open science: One term, five schools of thought. In S.
Bartling & S. Friesike (Eds.), Opening Science. Springer Nature.
13. National Academies of Sciences, Engineering, and Medicine. (2018). Open science by
design: Realizing a vision for 21st century research. National Academies Press.
14. Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J.,
Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich,
E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M.,
Ishiyama, J., … Yarkoni, T. (2015). Promoting an open research culture: Author
guidelines for journals could help to promote transparency, openness, and reproducibility.
Science, 348(6242), 1422–1425.
15. Winerman, L. (2017). Trends report: Psychologists embrace open science. American
Psychological Association, 48(10), 90.
B. PRE-REGISTRATION
This section provides resources related to pre-registration, or the process of registering study
plans in a repository before beginning a research project. Pre-registering commits researchers
OPRE 2019 Methods Meeting Resource List 3
to a specific design, hypothesis, and/or data analysis plan. The table at the end of the section
contains links to registry websites and other practical tools.
1. Abrams, E., Libgober, J., & List, J. A. (2020). Research registries: Facts, myths, and
possible improvements (NBER Working Paper 27250). National Bureau of Economic
Research.
2. de Groot, A. D. (2014). The meaning of “significance” for different types of research
[translated and annotated by E.-J. Wagenmakers, D. Borsboom, J. Verhagen, R. Kievit,
M. Bakker, A. Cramer, D. Matzke, D. Mellenbergh, & H. LJ. van der Maas]. Acta
Psychologica, 148, 188–194.
3. Duflo, E., Banerjee, A., Finkelstein, A., Katz, L. F., Olken, B. A., & Sautmann, A. (2020). In
praise of moderation: Suggestions for the scope and use of pre-analysis plans for RCTs in
economics (NBER Working Paper 26993). National Bureau of Economic Research.
4. Kupferschmidt, K. (2018). More and more scientists are preregistering their studies.
Should you? Science.
5. Lane, S. P., & Hennes, E. P. (2018). Power struggles: Estimating sample size for
multilevel relationships research. Journal of Social and Personal Relationships, 35(1), 7–
31.
6. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration
revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606.
7. van 't Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology—A
discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12.
8. Weston, S. J., Ritchie, S. J., Rohrer, J. M., & Przybylski, A. K. (2019). Recommendations
for increasing the transparency of analysis of preexisting data sets. Advances in Methods
and Practices in Psychological Science, 2(3), 214–227.
Pre-registration Tools, Websites, and Registries
Organization Name and Link Type Topical Area Overview and Purpose
American Economic
Association (AEA)
AEA randomized
controlled trials
(RCTs) registry
Registry
Economics and
other social
sciences
● Free registry for RCTs
Center for Open
Science Pre-registration
Tools/
Website
General/Cross-
disciplinary
● Website compiles pre-registration
resources and support materials
(e.g., literature, teaching materials,
instructions, FAQs) for researchers
Center for Open
Science
Registered
reports
Tools/
Website
General/Cross-
disciplinary
● Website compiles resources on
Registered Reports, a publishing
format used by over 250 journals
that emphasizes peer review prior
to data collection
OPRE 2019 Methods Meeting Resource List 4
Organization Name and Link Type Topical Area Overview and Purpose
Evidence in
Governance and
Politics (EGAP)
Registration Registry General/Cross-
disciplinary
● Registry for EGAP, a research,
evaluation, and learning network,
hosted in collaboration with the
Center for Open Science to
increase the sharing of accurate
and transparent information on
research projects
National Institutes of
Health, U.S. National
Library of Medicine
ClinicalTrials.gov Registry Medical
● Database of privately and publicly
funded clinical studies conducted
around the world on a wide range
of diseases and conditions
Office of Evaluation
Sciences (OES) Building evidence Registry
General/Cross-
disciplinary
● Searchable registry of more than 70
OES-supported evaluations on
various topics across the Federal
Government
OES OES evaluation
process
Tools/
Website
General/Cross-
disciplinary
● Website explaining OES evaluation
processes and policies and linking
to relevant resources and guidance
papers
Open Science
Framework (OSF)
Registration
forms and
templates
Tools/
Website
General/cross-
disciplinary
● Website providing sample
registration forms and templates for
general use
OSF Template: Pre-
registration-RR.rtf
Tools/
Website
General/cross-
disciplinary
● Guidance document explaining how
to build a registered report
U.S. Department of
Agriculture, Food
and Nutrition Service
(USDA FNS)
Study and
evaluation plans
Tools/
Website
Food and
nutrition
● Provides annual USDA FNS
research plans containing short
descriptions of projects expected to
be launched in each fiscal year
World Health
Organization (WHO)
International
clinical trial
registry platform
Registries Health ● Webpage links to various clinical
trial registries connected to WHO
C. REPRODUCIBILITY AND REPLICABILITY
This section provides resources related to reproducibility and replicability. Reproducibility is
defined as duplicating the results of a prior study using the same materials as were used by
the original investigator. By contrast, replicability is defined as duplicating the results of a prior
study following the same procedures but collecting new data.2
2 Bollen, K., Cacioppo, J. T., Kaplan, R. M., Krosnick, J. A., Olds, J. L., & Dean, H. (2015). Social, behavioral, and economic sciences
perspectives on robust and reliable science. Report of the Subcommittee on Replicability in Science Advisory Committee to the National
Science Foundation Directorate for Social, Behavioral, and Economic Sciences.
The table at the end of the
section contains links to relevant websites.
OPRE 2019 Methods Meeting Resource List 5
1. Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M.,
Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen,
Y., Forsell, E., Gampa, A., Heikensten, E., Hummer, L., Imai, T., Isaksson, S., … Hang, W. (2018). Evaluating the replicability of social science experiments in Nature and
Science between 2010 and 2015. Nature Human Behavior, 2, 637–644.
2. Errington, T. (2016). Reproducibility Project: Cancer biology - Barriers to replicability in
the process of research. OSF.
3. Goodman, S. N., Fanelli, D., & Ioannidis, J. P. (2016). What does research reproducibility
mean? Science Translational Medicine, 8(341), 341ps12-341ps12.
4. Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M.
C., & Lenne, R. L. (2018). Data availability, reusability, and analytic reproducibility:
Evaluating the impact of a mandatory open data policy at the journal Cognition. Royal
Society Open Science, 5(8), 180448.
5. LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A unified
framework to quantify the credibility of scientific findings. Advances in Methods and
Practices in Psychological Science, 1(3), 389–402.
6. National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and
replicability in science. National Academies Press.
7. Nosek, B. A., & Errington, T. M. (2020). What is replication? PLoS Biology, 18(3), 1–8.
8. Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., du Sert,
N. P., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A
manifesto for reproducible science. Nature Human Behavior, 1, 1–9.
9. Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník,
S., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R.,
Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., ... Nosek, B. A.
(2018). Many analysts, one data set: Making transparent how variations in analytic
choices affect results. Advances in Methods and Practices in Psychological Science,
1(3), 337–356.
10. The Open Science Collaboration. (2015). Estimating the reproducibility of psychological
science. Science, 349(6251), aac4716-1-aac4716-8.
Reproducibility and Replicability Tools and Websites
Organization or Name and Link Type Topical Area Overview and Purpose
Author
Association for
Psychological
Science
Registered
Replication
Reports
Tools/
Website
Psychologica
l sciences
● Guidance and resources for
producing and reviewing Registered
Replication Reports, which are
multi-lab, high-quality replications of
psychological science experiments
OPRE 2019 Methods Meeting Resource List 6
Organization or Name and Link Type Topical Area Overview and Purpose
Author
Nuijten, M.
Checking
robustness in four
steps
Tools/
Website
Social
sciences
● Document provides guidance on
assessing and improving robustness
of psychological science (using
minimal resources)
Project Tier
The Documenting
Research in the
Empirical Social
Sciences (DRESS)
Protocol
Tools/
Website
Social
sciences
● Provides guidance and resources
for using the DRESS Protocol, a set
of standards for replication
documentation
Project Tier TIER Protocol
(version 3.0)
Tools/
Website
Social
sciences
● Provides guidance on implementing
the TIER Protocol, a research
project workflow that emphasizes
importance of replication
documentation throughout research
process
D. DATA MANAGEMENT AND SHARING
This section provides resources related to managing and sharing data. Data management
encompasses the organization, documentation, preservation, and accessibility of data. Good
data management is essential for high-quality research (Eynden, 2011), and data sharing is
crucial for reproducibility. The table at the end of the section contains links and tools specific to
codebooks.
1. Eynden, V. V. D., Corti, L., Woollard, L. B., & Horton, L. (2011). Managing and sharing
data: best practice for researchers. UK Data Archive, University of Essex.
2. Gentzkow, M., & Shapiro, J. M. (2014). Code and data for the social sciences: A
practitioner’s guide (Working paper).
3. Hardwicke, T. E., & Ioannidis, J. (2018). Populating the Data Ark: An attempt to retrieve,
preserve, and liberate data from the most highly-cited psychology and psychiatry articles.
PloS One, 13(8), e0201856.
4. Inter-university Consortium for Political and Social Research (ICPSR). (n.d.).
Recommended informed consent language for data sharing.
5. ICPSR. (2012). Guide to social science data preparation and archiving: Best practice
throughout the data life cycle (5th ed.).
6. ICPSR. (2012). Guidelines for effective data management plans.
7. Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L. S., Kennett, C., Slowik, A., Sonnleitner,C., Hess-Holden, C., Errington, T.
M, Fiedler, S., & Nosek, B. A. (2016). Badges to acknowledge open practices: A simple,
low-cost, effective method for increasing transparency. PLoS Biology, 14(5).
OPRE 2019 Methods Meeting Resource List 7
8. Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of
psychological research data for reanalysis. American Psychologist, 61(7), 726–728.
Data Management and Sharing Tools, Websites, and Media
Organization Name and Link Type Topical Area Overview and Purpose
or Author
Amsterdam Public
Health Quality
Handbook
Codebook/Data
dictionary
Tools/
Website
Statistical
analyses for
public health
● Website and linked resources with
guidance on constructing high-
quality codebooks
Institute for Social
Science Research Codebook help
Tools/
Website
Social
sciences
● Website providing overview of
codebook components
Inter-university
Consortium for
Political and Social
Research (ICPSR)
Guide to
codebooks (1st
ed.)
Tools/
Website
Social
sciences
● Report providing guidance and links
to additional resources for
developing codebooks
ICPSR
Intro to data
management
plans
Video file Social
sciences
● Video of recorded presentation on
data management plans
Princeton University
Library
How to use a
codebook
Tools/
Website General
● Website with description and
examples of how to use a codebook,
with links to additional resources
E. SYSTEMATIC REVIEWS
This section provides background on systematic reviews and meta-analyses, including a
historical overview of these methods and information about current reporting standards.
Systematic review is a form of structured literature review that answers a question through the
analysis of all available evidence. It includes applying predetermined inclusion and exclusion
criteria to the literature, critically appraising the relevant literature, and extracting and
synthesizing data from the evidence base to formulate findings.3
3 National Information Center on Health Services Research and Health Care Technology. (2017). HTA 101: Glossary.
The table at the end of the
section contains links to tools and websites on systematic reviews; the text box contains
resources specific to the Institute of Education Sciences’ What Works Clearinghouse.
1. Agency for Healthcare Research and Quality. (2014). Methods guide for effectiveness
and comparative effectiveness reviews (AHRQ Publication 10(14)-EHC063-EF).
2. Appelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M.
(2018). Journal article reporting standards for quantitative research in psychology: The
APA publications and communications board task force report. American Psychologist,
73(1), 3–25.
3. Glass, G. (1976). Primary, secondary, and meta-analysis of research. Educational
Researcher, 5(10), 3–8.
OPRE 2019 Methods Meeting Resource List 8
4. Petrosino, A., Turpin‐Petrosino, C., Hollis‐Peel, M. E., & Lavenberg, J. G. (2013). 'Scared
Straight' and other juvenile awareness programs for preventing juvenile delinquency.
Cochrane Database of Systematic Reviews, 4.
5. Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C., Porter,
A. C., Tugwell, P., Moher, D., & Bouter, L. M. (2007). Development of AMSTAR: A
measurement tool to assess the methodological quality of systematic reviews. BMC
Medical Research Methodology, 7, 10.
6. Whiting, P., Savović, J., Higgins, J. P., Caldwell, D. M., Reeves, B. C., Shea, B., Davies,
P., Kleijnen, J., & Churchill, R. (2016). ROBIS: A new tool to assess risk of bias in
systematic reviews was developed. Journal of Clinical Epidemiology, 69, 225–234.
Systematic Review Tools, Websites, and Registries
Organization or Name and Link Type Topical Area Overview and Purpose
Author
● Cochrane is a journal and
Higgins, J.,
Lasserson, T.,
Chandler, J., Tovey,
D., & Churchill, R.
Methodological
expectations of
Cochrane
Intervention
Reviews
Systematic
review tool Healthcare
database for systematic reviews in
healthcare. This report describes
methodological standards to
which all Cochrane protocols,
reviews, and updates are
expected to adhere
Joanna Briggs
Institute (JBI)
Checklist for
systematic
reviews and
research
syntheses
Systematic
review tool General
● JBI is an international,
membership-based research and
development organization; report
provides tools for adhering to its
systematic reviews
Preferred Reporting
Items for Systematic
Reviews and Meta-
Analyses (PRISMA)
PRISMA Website General
● Website with guidance and tools
for using PRISMA, an evidence-
based minimum set of items for
reporting in systematic reviews
and meta-analyses
Society for Research
on Educational
Effectiveness
Registry of
Efficacy and
Effectiveness
Studies
Registry Education
● Database of causal inference
studies in education and related
fields
National Institute for
Health Research
PROSPERO
international
prospective
register of
systematic
reviews
Registry Health
● International database of
prospectively registered
systematic reviews in range of
fields where there is a health-
related outcome
OPRE 2019 Methods Meeting Resource List 9
_ _
’ ’
-
Resources Specific to the What Works Clearinghouse
The What Works Clearinghouse (WWC) is an initiative of the Institute of Education Sciences
(IES). WWC reviews original research on education interventions. Links to helpful pages on
the WWC website and related resources follow:
Education Resources Information Center (ERIC) IES. (n.d.). Education resources information center. IES ERIC. (n.d.). FAQ - Peer review. IES WWC. (n.d.). Search publications. IES WWC. (n.d.). Find what works. IES WWC. (n.d.). Handbooks and other resources: Procedures and standards handbooks. IES WWC. (n.d.). Handbooks and other resources: Review protocols. IES WWC. (n.d.). Practice guides. IES WWC. (n.d.). WWC help desk. IES WWC. (n.d.). Using the WWC to find ESSA Tiers of evidence. IES. (2020). Standards for excellence in education research. IES. (2019). Request for applications.
Like OPRE
on Facebook
Facebook.com/OPRE.ACF
Follow OPRE
on Instagram
@OPRE ACF
Follow OPRE
on Twitter
@OPRE ACF
Sign up
for the OPRE
Newsletter
This brief was prepared by Insight Policy Research (1901 North Moore Street, Suite 1100, Arlington, VA 22209) under Contract Number HHSP233201500109I. The Administration for Children and Families Contracting Officer s Representatives are Emily Ball Jabbour and Kriti Jain. The Insight Project Director is Rachel Holzwart, and the Deputy Project Director is Hilary Wagner.
This brief is in the public domain. Permission to reproduce is not necessary. Suggested citation:
Hansen, D., & Holzwart, R. (2020). OPRE 2019 methods meeting resource list (OPRE Report 2020 131). Washington, DC: Office of Planning, Researchand Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
This brief and other reports sponsored by the Office of Planning, Research, and Evaluation are available at http://www.acf.hhs.gov/opre.
Disclaimer: The views expressed in this publication do not necessarily reflect the views or policies of the Office of Planning, Research, and Evaluation; the Administration for Children and Families; or the U.S. Department of Health and Human Services.