new data disasters and how to prevent them - wordpress.com · 2017. 1. 26. · data disasters and...

Data Disasters and How to Prevent Them

Colleen W. Marano, PhD

Director Clinical Immunology

Janssen R &D, LLC

&

Anne Martin Robinson, Pharm.D.

Group Scientific Director, Immunology

Global Pharmaceutical Research and Development

AbbVie, Inc.

Disclosure slide

• Colleen Marano is an employee of Janssen R & D, LLC

• Anne Robinson is an employee of AbbVie, Inc.

2

What are data disasters...

• Any issue that potentially interferes with study interpretation or validity, for example:

– missing data,

– confounded data (i.e. endpoints compromised by other procedures or biased by surveys administered out of sequence),

– out of window data,

– inaccessible or lost data,

– data you can't use

Why do data issues arise?

• Protocols are unclear/complicated/internally conflicting

• Study systems are confusing or do not communicate with each other or do not require essential entries

• Humans make errors (or intentionally provide misinformation)

• Lab samples get lost/hemolyzed

• Transit agencies go on strike

• Technology fails to work correctly or batteries fail

• Schedules/vacations/holidays interfere with pre-specified visit plans

• Sick people need to be treated outside of protocol-defined parameters

The type of questions you are asking drives the data to be collected....

• Is the study hypothesis generating (exploratory) or hypothesis testing?

• What data is necessary to support the objectives of the study?

– What is essential?

– What is nice-to-have?

• How much, at what times, & for what duration should the data be collected?

Biostatistics Biomarkers Clinical Therapeutic Area Clinical Operations Clinical Pharmacology Data Management Health Economics Medical Writers Patient-reported Outcomes Pharmacovigilance Programming Quality Management and Compliance Regulatory

A Study is a Cross-functional Team Effort

• Protocol • ICF • Case report forms; data

cleaning & monitoring • Patient diaries • Site/vendor selection • Quality Plan • IND/regulatory

submissions • Statistical Analysis Plan • Data Presentation Plan • TLFs • Study reports

Coordinated with vendors, CROs, Ethics Committees , and Health Authorities

Protocol Considerations (I) • Does the data collected support the study objectives?

• Has the protocol & its requirements been reviewed by individuals outside of your study team?

– Obtain input of key investigative sites, including study coordinators, or patient representatives

– Are there any safety considerations requiring unique sample or data collection?

– What are the relevant concomitant medications?

– What medications should be discontinued prior to study drug?

• Consider eligibility requirements and procedures to minimize protocol deviations & missing data

• Are they in line with patient’s perception of burdens of invasive procedures (i.e. endoscopy [number and interval]

• Are screening procedures consistent with local practices/preferences (e.g. TB testing)

Protocol Considerations (II) • Will there be a blinded assessor?

– For example, an endoscopist blinded to treatment & patient-reported data

– Is the study flow (e.g. visit procedures/sequences) logistically feasible?

• How will missing or out of window data be handled?

• Define end of study completion and impact of patient drop out on study validity?

– Will patients who discontinue study prematurely be replaced?

Informed Consent Considerations • Confirm that ICF explains all protocol-specified

procedures and study risks in patient appropriate language/format.

• Does your ICF language allow banking and future analysis of biological specimens? – Is the collection of genetic samples permitted at each

site/country?

• Does your ICF allow follow-up after patients prematurely end study participation? – For example, collection of colectomy or safety follow-

up through planned final study visit.

Data Collection & Integrity • Collect the right data at the right time

• Consider how will data be collected: paper/electronic case report forms or ePRO

– Data review & query process

– Data cleaning & monitoring: frequency

– Handling of missing data

– Treatment failure

• Avoid double data collection: consider data consistency & reconciliation issues

– Concomitant medications affecting treatment response: targeted med review pages linked to concomitant medication CRF

– Pre-programmed real-time edit checks or backend edit checks

• Multiple data sources

– IWRS/CRF/Central Lab or Imaging Vendor/ePRO

A Few Examples

– Disease indices that are incorrectly calculated

– Missing data

• A missing subscore of a multiple component index

• Missing entries for data that relies on multiple days for calculation

• Failure to follow protocol visit specifications

Simple Endoscopic Score (SES-CD) Ileum Right colon Transverse

colon Left colon Rectum

Presence/size of ulcers 0: none 1: < 0.5 cm 2: 0.5-2 cm 3: > 2 cm

Extent of ulcerated surface 0: 0% 1: < 30% 2: 10-30% 3: > 40%

Extent of affected surface 0: 0% 1: < 50% 2: 50-75% 3: > 75%

Narrowings 0: none 1: single, can be passed 2: multiple, can be passed 3: cannot be passed

Daperno, et al. Gastrointest Endosc 2004;60:505-12.

Incorrectly Calculated Disease Activity Index: SES-CD Hypothetical Example

Ileum Right colon Transverse colon

Left colon Rectum


2 1 0 0 0


1 0 0 0 0


0 1 0 0 0


0 3 0 0 0

Scores present for ileum although right colon has score for “impassable stenosis”

Incorrectly Calculated Disease Activity Index: SES-CD Hypothetical Example

Ileum Right colon Transverse colon

Left colon Rectum


2 1 0 0 0


1 0 0 0 0


1 1 0 0 0


0 3 0 0 0

Scores ileum although right colon has score for “impassable stenosis”

Ulcerated surface subscore is 0 although size of ulcer score indicates aphthi are

present

Pediatric Crohn’s Disease Activity Index (PCDAI) Variable Score

Abdominal pain 0, 5, 10

Stool frequency 0, 5, 10

General well-being 0, 5, 10

Hematocrit 0, 2.5, 5

Erythrocyte sedimentation rate 0, 2.5, 5

Albumin 0, 5, 10

Weight 0, 5, 10

Height 0, 5, 10

Abdomen 0, 5, 10

Perirectal disease 0, 5, 10

Extra-intestinal manifestations 0, 5, 10

Total 0-100

Hyams, et al. J Pediatr Gastroenterol Nutr 1991;12:439-47.

Missing PCDAI Subscore Hypothetical Example (Lost Lab Sample)

Variable Score

Abdominal pain 0

Stool frequency 5

General well-being 5

Hematocrit n/a

Erythrocyte sedimentation rate 0

Albumin 0

Weight 0

Height 0

Abdomen 0

Perirectal disease 0

Extra-intestinal manifestations 0

Total 10?

Is the total really 10?

Is the patient in remission?

Is this PCDAI “missing?”

How will you decide?

Missing SES-CD Segment Hypothetical Example (Ileum Not Intubated)

Ileum Right colon

Transverse colon

Left colon

Rectum

Presence/size of ulcers 0: none 1: aphthous < 0.5 cm 2: 0.5-2 cm 3: > 2 cm

n/a 1 0 0 0


n/a 1 0 0 0


n/a 1 0 0 0


n/a 0 0 0 0

Is the total really 3?

Does it matter why the ileum wasn’t

intubated?

Is this SES-CD “missing?”

How will you decide?

Missing Data: PROs Based on Multiple Days of Patient Entries

• Example: CDAI is based upon seven days of patient reported data for stool frequency, abdominal pain, general well-being

• Things to consider • Which seven days? (i.e., from what period can the 7 days be

selected?)

• Do the seven days need to be consecutive?

Day S M T W Th F S

Liquid/Soft Stool frequency

3 2 2 1 1 2 1

Missing CDAI Entries Hypothetical Example

• What if this happens?

• Are four days of entries sufficient?

Day S M T W Th F S

Stool frequency

3 2 2 1

Missing CDAI Entries Hypothetical Example


• Or this?

Day S M T W Th F S

Stool frequency

3 2 2 1

Day S M T W Th F S

Stool frequency

3 12 1 1 1 2 1

Missing CDAI Entries Example


• Or this?

Day S M T W Th F S

Stool frequency

3 2 2 1

Day S M T W Th F S

Stool frequency

3 12 1 1 1 2 1

Endoscopy prep

Endoscopy

Should days around endoscopy preparation and procedure be

excluded (and if so, should they be replaced?)

Missing Data Based on Visit Scheduling: Hypothetical Example

• Subject is randomized, but last minute travel requires patient to leave country at Week 5 for 30 days; subject is willing to undergo endoscopy at Week 5

• Is this acceptable?

• What if the endoscopy is at Week 11, and the subject received “rescue” treatment at Week 10?

0 Week 8

Endoscopy endpoint

Treatment Period (concomitant medications are kept stable) Off Treatment

12

Safety follow up

Missing Data Based on Visit Scheduling: Hypothetical Example

• Subject is randomized, but last minute travel requires patient to leave country at Week 5 for 30 days; subject is willing to undergo endoscopy at Week 5

• Is this acceptable?

• What if the endoscopy is at Week 11, and the subject received “rescue” treatment at Week 10?

• Be sure to pre-specify study visit windows, so data are consistently assigned to the appropriate visit

• Consider clinical relevance of visit windows and potential issue of missing data

0 Week 8

Endoscopy endpoint

Treatment Period (concomitant medications are kept stable) Off Treatment

12

Safety follow up

Closing thoughts

• If something can go wrong, it will

• Anticipate issues and pre-specify how to handle them

• It is not OK to change the rules after the fact without specifying what was done ad-hoc

• Resist the lure of adding too many variables, procedures, and data collection instruments to a study

• Involve your statistician early and often!

• Consider logic checks (e.g., CDEIS should not total > 44) or systems that force data entry

• Have an external study coordinator review your protocol

• Monitor data periodically in in order to identify issues while there is still time to address them (before data are unblinded)

new data disasters and how to prevent them - wordpress.com · 2017. 1. 26. · data disasters and...

Documents