-
Promoting Evidence-Based IRB Policy and Practice through Empirical
Study
Mohamed AbdirisakMentored by Prof. Holly Fernandez Lynch, JD, MBE
-
Overview
1. What is the goal of an IRB?• To protect the welfare and rights of human research participants
2. What is the problem?• Current research focuses on structure and process, not IRB effectiveness
3. Why is evaluating effectiveness so hard?• We cannot see the bad things IRBs are preventing
4. Approach:• Analyzed real-world IRB evaluation and accreditation tools to identify themes and gaps in
outcomes• Importantly: Do tools look at participant protection outcomes?• Currently in beginning stages of research
5. Findings
PresenterPresentation NotesAn IRB is effective when it’s doing its job of protecting participantsMeasuring effectiveness can be hard because we can’t see the bad things IRBs are preventingImagine a new HIV prevention drug becomes availableIt would be easy to tell if drug is failing if people still contract HIV after taking it.If people stopped getting HIV, we can’t say for certain that it was due to the drug, because other factors, stopped the HIV contraction.
IRB evaluation research focuses on process and structure, because it’s easy to tell if an IRB has delays in processing protocols and other negative outcomesBut it’s hard to see what the IRB is preventing
-
What were IRBs designed to do?
1. Protecting participant welfare and rights:• Autonomy & Consent• Protect vulnerable groups• Avoid unnecessary harm• Maximize benefits
2. Fostering a culture of ethical concern:• Spot ethical issues• Do the right thing, even without
oversight
3. Advancing useful knowledge• Not inhibiting ethical research• Promote social value and scientific
validity
4. Promoting Justice• Fair participant selection
5. Maintaining and promoting trust in the research enterprise
-
Focus on IRB structure and process
• Examples of structure• Resources available to IRB• How much it costs to administer the IRB• IRB Staffing
• Examples of Process• Compliance with federal regulations• Various forms and documents that are used in review, e.g. consent forms, etc.• IRB decision making process• How long it takes to review protocols
• Structure and process are surrogates for what matters: Effectiveness outcomes
PresenterPresentation NotesGive tumor example from Resnik to illustrate surrogate measures.Now let’s dive into the IRB evaluation tools
-
IRB Evaluation Tools
• Many tools exist for IRB evaluation/accreditation • Designed by government, private agencies, and researchers
• Important question:• What do tools evaluate?• What don’t they evaluate?• Themes or overlap between tools?
• Most importantly: Do tools consider IRB effectiveness, or just structure/process?
• This was the focus of our research this summer
PresenterPresentation NotesWe want toEffectiveness: culture
-
Overview of approach
• Performed literature review of IRB evaluation research• Found gaps and preliminary themes in literature
• Collected a range of IRB evaluation tools• Do tools replicate gaps in literature?
• Created a codebook that was the foundation for our analysis• The codebook organized themes into categories called “nodes”
PresenterPresentation NotesLiterature review of previous IRB research to understand challenges in IRB world ; glean themes
Go into lit review
-
Literature review
• Literature review helped to determine the basic buckets of the codebook
• Started with a single evaluation tool (OHRP Self-Assesment Tool) • This helped to refine categories for the codebook
PresenterPresentation NotesTalk about themes from literature review: Studies, as well as tools
-
Codebook
• Codebook is a guide for sorting tools• It consists of nodes, which can be thought of as
buckets • Each node has a precise definition for what items
can be placed into it.• Any piece of data that fit that definition was
placed in that node• An item may fit into multiple nodes
• Used a qualitative analysis program called nvivo
PresenterPresentation NotesEach node is like a laundry bucketBuckets have certain things that go into them like, whites, delicates, etc.
-
Codebook (Cont.)
• The codebook was refined many times as we coded the tools
• Possible modifications to a node:• Broaden/narrow down definition
to capture desired data• Merge highly overlapping nodes• Delete nodes or add new ones
-
Inter-rater reliability
• Inter-rater reliability is a measure of codebook quality
• Coders should code data similarly if definitions/themes are sufficient and clear
• Two people coded the data• IRR was assessed after 1st round of
coding• Any differences in coding were resolved
CodingIndividually
IRR Comparison
Resolving differences
PresenterPresentation NotesIf the definitions are precise enough, and the themes are all there, coders should code the same way
-
AAHRPP Tool• Created by an an IRB accreditation agency
called AAHRPP• Organizations must meet all of the
standards in the tool to become accredited
• Tool is very comprehensive:• Covers virtually every element of IRB
structure and process• Also evaluates broader human research
protection program• Also addresses role of individual investigators
in participant protection• Can be vague at times
PresenterPresentation NotesFocuses heavily on written policies/procedures
-
OHRP Self-Assessment Tool
• Designed by HHS Office for Human Research Protections (OHRP)
• Intended for users to evaluate their own IRB
• Consists of a series of questions• Focuses on general structure and process:
• IRB budgets• Member selection• IRB oversight• Conflicts of interest management• Workload, etc.
-
FDA Tool• Designed to guide institutions and IRBs in
preparing written procedures/policies• Tool refers to specific federal regulations,
but mainly provides non-binding recommendations
• Consists of checklists that focus on:• Initial and Continuing Review of Research• Approval and reporting of changes to a study• Process for suspending/terminating IRB approval
-
Orion Tool• Created by a private Canadian IRB
accreditation agency • Standards apply to IRBs and organizations
as a whole• Focuses on written procedures for eight
areas such as:• IRB leadership• Member qualification• Data confidentiality
• Tool is short and sometimes vague • Says organization should have written
procedures, but doesn't go into detail about what procedures entail
-
IRB Research Assessment Tool (IRB-RAT)• IRB RAT was created by a team
of researchers• Uses a survey design
• Focuses on:• IRB respect towards investigators• IRB Mission, Approach, and
Culture• Has investigators rate how
important various items are to their vision of an ideal IRB
• Investigators also rate their own IRB on those same items
-
Preliminary Findings
• 5 most coded items across all tools:
• Conflicts of interest or Bias• Informed Consent• Written Operating Procedures• Risk to Participants• Investigator responsibility
• 5 least coded items across all tools:
• Handling investigator complaints• Early Termination and Suspension
of studies• Timeliness of IRB review• Protocol submission process• Participant Outcomes (zero items
coded)
-
Lessons learned this summer
• Patience, time management, and spacing out work
• Processing and understanding lots of information that can be dense at times
• Being resourceful after setbacks• Problems with nvivo software crashing
• Communicating my thoughts and questions as clearly as possible
• Balancing between being self-directed and asking questions/seeking consultation when necessary.
-
Thank You’s
• Prof. Holly Fernandez Lynch• Dr. Justin Clapp• Megan Bogia• MEHP faculty and administrative staff
-
Questions?
PresenterPresentation NotesAnother thing that makes it hard to evaluate IRB effectiveness is the fact that just by IRBs existing, investigator behavior is changed.
A solution to see if IRBs are doing a good job is to compare how protocols look when they come into the IRB vs. how they look when they come out Are the changes made by the IRB relevant to participant protection? This is an indicator of an effective IRB.
The weakness of this approach is that the protocol might have already been good before IRB review, and that the IRB didn’t need to change anything. However, even though the IRB didn’t change anything, we can’t say we don’t need IRBs, because maybe people would behave differently if there was no IRB looking over their shoulder.
Promoting Evidence-Based IRB Policy and Practice through Empirical �Study�OverviewWhat were IRBs designed to do?Focus on IRB structure and processIRB Evaluation ToolsOverview of approachLiterature reviewCodebookCodebook (Cont.)Inter-rater reliability AAHRPP ToolOHRP Self-Assessment ToolFDA ToolOrion ToolIRB Research Assessment Tool (IRB-RAT)� Preliminary FindingsLessons learned this summerThank You’sQuestions?