Promoting Evidence-Based IRB Policy and Practice through Empirical

Study

Mohamed AbdirisakMentored by Prof. Holly Fernandez Lynch, JD, MBE

Overview

1. What is the goal of an IRB?• To protect the welfare and rights of human research participants

2. What is the problem?• Current research focuses on structure and process, not IRB effectiveness

3. Why is evaluating effectiveness so hard?• We cannot see the bad things IRBs are preventing

4. Approach:• Analyzed real-world IRB evaluation and accreditation tools to identify themes and gaps in

outcomes• Importantly: Do tools look at participant protection outcomes?• Currently in beginning stages of research

5. Findings

PresenterPresentation NotesAn IRB is effective when it’s doing its job of protecting participantsMeasuring effectiveness can be hard because we can’t see the bad things IRBs are preventingImagine a new HIV prevention drug becomes availableIt would be easy to tell if drug is failing if people still contract HIV after taking it.If people stopped getting HIV, we can’t say for certain that it was due to the drug, because other factors, stopped the HIV contraction.

IRB evaluation research focuses on process and structure, because it’s easy to tell if an IRB has delays in processing protocols and other negative outcomesBut it’s hard to see what the IRB is preventing

What were IRBs designed to do?

1. Protecting participant welfare and rights:• Autonomy & Consent• Protect vulnerable groups• Avoid unnecessary harm• Maximize benefits

2. Fostering a culture of ethical concern:• Spot ethical issues• Do the right thing, even without

oversight

3. Advancing useful knowledge• Not inhibiting ethical research• Promote social value and scientific

validity

4. Promoting Justice• Fair participant selection

5. Maintaining and promoting trust in the research enterprise

Focus on IRB structure and process

• Examples of structure• Resources available to IRB• How much it costs to administer the IRB• IRB Staffing

• Examples of Process• Compliance with federal regulations• Various forms and documents that are used in review, e.g. consent forms, etc.• IRB decision making process• How long it takes to review protocols

• Structure and process are surrogates for what matters: Effectiveness outcomes

PresenterPresentation NotesGive tumor example from Resnik to illustrate surrogate measures.Now let’s dive into the IRB evaluation tools

IRB Evaluation Tools

• Many tools exist for IRB evaluation/accreditation • Designed by government, private agencies, and researchers

• Important question:• What do tools evaluate?• What don’t they evaluate?• Themes or overlap between tools?

• Most importantly: Do tools consider IRB effectiveness, or just structure/process?

• This was the focus of our research this summer

PresenterPresentation NotesWe want toEffectiveness: culture

Overview of approach

• Performed literature review of IRB evaluation research• Found gaps and preliminary themes in literature

• Collected a range of IRB evaluation tools• Do tools replicate gaps in literature?

• Created a codebook that was the foundation for our analysis• The codebook organized themes into categories called “nodes”

PresenterPresentation NotesLiterature review of previous IRB research to understand challenges in IRB world ; glean themes

Go into lit review

Literature review

• Literature review helped to determine the basic buckets of the codebook

• Started with a single evaluation tool (OHRP Self-Assesment Tool) • This helped to refine categories for the codebook

PresenterPresentation NotesTalk about themes from literature review: Studies, as well as tools

Codebook

• Codebook is a guide for sorting tools• It consists of nodes, which can be thought of as

buckets • Each node has a precise definition for what items

can be placed into it.• Any piece of data that fit that definition was

placed in that node• An item may fit into multiple nodes

• Used a qualitative analysis program called nvivo

PresenterPresentation NotesEach node is like a laundry bucketBuckets have certain things that go into them like, whites, delicates, etc.

Codebook (Cont.)

• The codebook was refined many times as we coded the tools

• Possible modifications to a node:• Broaden/narrow down definition

to capture desired data• Merge highly overlapping nodes• Delete nodes or add new ones

Inter-rater reliability

• Inter-rater reliability is a measure of codebook quality

• Coders should code data similarly if definitions/themes are sufficient and clear

• Two people coded the data• IRR was assessed after 1st round of

coding• Any differences in coding were resolved

CodingIndividually

IRR Comparison

Resolving differences

PresenterPresentation NotesIf the definitions are precise enough, and the themes are all there, coders should code the same way

AAHRPP Tool• Created by an an IRB accreditation agency

called AAHRPP• Organizations must meet all of the

standards in the tool to become accredited

• Tool is very comprehensive:• Covers virtually every element of IRB

structure and process• Also evaluates broader human research

protection program• Also addresses role of individual investigators

in participant protection• Can be vague at times

PresenterPresentation NotesFocuses heavily on written policies/procedures

OHRP Self-Assessment Tool

• Designed by HHS Office for Human Research Protections (OHRP)

• Intended for users to evaluate their own IRB

• Consists of a series of questions• Focuses on general structure and process:

• IRB budgets• Member selection• IRB oversight• Conflicts of interest management• Workload, etc.

FDA Tool• Designed to guide institutions and IRBs in

preparing written procedures/policies• Tool refers to specific federal regulations,

but mainly provides non-binding recommendations

• Consists of checklists that focus on:• Initial and Continuing Review of Research• Approval and reporting of changes to a study• Process for suspending/terminating IRB approval

Orion Tool• Created by a private Canadian IRB

accreditation agency • Standards apply to IRBs and organizations

as a whole• Focuses on written procedures for eight

areas such as:• IRB leadership• Member qualification• Data confidentiality

• Tool is short and sometimes vague • Says organization should have written

procedures, but doesn't go into detail about what procedures entail

IRB Research Assessment Tool (IRB-RAT)• IRB RAT was created by a team

of researchers• Uses a survey design

• Focuses on:• IRB respect towards investigators• IRB Mission, Approach, and

Culture• Has investigators rate how

important various items are to their vision of an ideal IRB

• Investigators also rate their own IRB on those same items

Preliminary Findings

• 5 most coded items across all tools:

• Conflicts of interest or Bias• Informed Consent• Written Operating Procedures• Risk to Participants• Investigator responsibility

• 5 least coded items across all tools:

• Handling investigator complaints• Early Termination and Suspension

of studies• Timeliness of IRB review• Protocol submission process• Participant Outcomes (zero items

coded)

Lessons learned this summer

• Patience, time management, and spacing out work

• Processing and understanding lots of information that can be dense at times

• Being resourceful after setbacks• Problems with nvivo software crashing

• Communicating my thoughts and questions as clearly as possible

• Balancing between being self-directed and asking questions/seeking consultation when necessary.

Thank You’s

• Prof. Holly Fernandez Lynch• Dr. Justin Clapp• Megan Bogia• MEHP faculty and administrative staff

Questions?

PresenterPresentation NotesAnother thing that makes it hard to evaluate IRB effectiveness is the fact that just by IRBs existing, investigator behavior is changed.

A solution to see if IRBs are doing a good job is to compare how protocols look when they come into the IRB vs. how they look when they come out Are the changes made by the IRB relevant to participant protection? This is an indicator of an effective IRB.

The weakness of this approach is that the protocol might have already been good before IRB review, and that the IRB didn’t need to change anything. However, even though the IRB didn’t change anything, we can’t say we don’t need IRBs, because maybe people would behave differently if there was no IRB looking over their shoulder.

Promoting Evidence-Based IRB Policy and Practice through Empirical �Study�OverviewWhat were IRBs designed to do?Focus on IRB structure and processIRB Evaluation ToolsOverview of approachLiterature reviewCodebookCodebook (Cont.)Inter-rater reliability AAHRPP ToolOHRP Self-Assessment ToolFDA ToolOrion ToolIRB Research Assessment Tool (IRB-RAT)� Preliminary FindingsLessons learned this summerThank You’sQuestions?