Pilot StudiesPilot StudiesNuts and BoltsNuts and Bolts

Deborah Grady, MD, MPHDeborah Grady, MD, MPH

Professor of MedicineProfessor of Medicine

Associate Dean for Clinical and Associate Dean for Clinical and Translational ResearchTranslational Research

UCSFUCSF

Pilot StudiesPilot Studies

• Small studies aimed at helping you:Small studies aimed at helping you:–plan the best study possibleplan the best study possible–determine that the study can be donedetermine that the study can be done–get funding for your studyget funding for your study

• Distinguish from pre-testing of Distinguish from pre-testing of questionnaires or measurementsquestionnaires or measurements

• Phase I and phase II trialsPhase I and phase II trials–phase I - determine toxicityphase I - determine toxicity–phase II - determine best dosephase II - determine best dose

Why Do a Pilot Study?Why Do a Pilot Study?

• FeasibilityFeasibility–determine if you can do itdetermine if you can do it

• LogisticsLogistics–determine the fastest, easiest, most cost-determine the fastest, easiest, most cost-

effective way to do iteffective way to do it

• Sample size ??Sample size ??–estimate rate in placebo group, effect size, estimate rate in placebo group, effect size,

variancevariance

• To convince funders you can do itTo convince funders you can do it

FeasibilityFeasibility

• Feasibility of recruitmentFeasibility of recruitment– obtain data on numbers eligibleobtain data on numbers eligible– survey centers, participantssurvey centers, participants– measure time, expense of recruitmentmeasure time, expense of recruitment

• Feasibility of interventions, measuresFeasibility of interventions, measures– can overweight/obese persons perform can overweight/obese persons perform

the 8 yoga postures and practice at home?the 8 yoga postures and practice at home?– will there be side effects?will there be side effects?– can participants do 2-h GTT/other tests?can participants do 2-h GTT/other tests?

• Feasibility of adherence and follow-upFeasibility of adherence and follow-up

Logistics and CostLogistics and Cost

• What staff are needed?What staff are needed?

• How long do visits take?How long do visits take?

• Where will participants park?Where will participants park?

• Can participants answer questionnaires Can participants answer questionnaires while completing GGT?while completing GGT?

• How much will the trial cost?How much will the trial cost?

Sample Size EstimatesSample Size Estimates

• To determine sample size for To determine sample size for full-scale trial:full-scale trial:–Rates of outcome in typical participantsRates of outcome in typical participants–Estimate of effect size Estimate of effect size –Variability of the main outcomeVariability of the main outcome

• Some question validity of effect size Some question validity of effect size estimates based on small studiesestimates based on small studies

How Big Should Pilots Be?How Big Should Pilots Be?

• ““It depends”It depends”

• Sample size estimates rarely useful Sample size estimates rarely useful

• Traditions and guessesTraditions and guesses–Cancer phase I trials start with 3Cancer phase I trials start with 3–Pilots for feasibility often 5-20Pilots for feasibility often 5-20–Pilots to estimate effect size can be largePilots to estimate effect size can be large

• Often depends on resources and timeOften depends on resources and time

Do Pilots Need Control Groups?Do Pilots Need Control Groups?

• ““It depends”It depends”

• Phase I trials (maximum tolerable dose)Phase I trials (maximum tolerable dose)– generally uncontrolledgenerally uncontrolled

• Phase II trials (minimum effective dose)Phase II trials (minimum effective dose)– generally controlled generally controlled

• Feasibility and logisticsFeasibility and logistics– controls generally not needed controls generally not needed

• Effect size/sample size estimatesEffect size/sample size estimates– controls helpful controls helpful

Funding for PilotsFunding for Pilots

• Local institutionLocal institution– Research Allocation Program (RAP)Research Allocation Program (RAP)– Residency Training ProgramResidency Training Program– CTSICTSI

• NIH R03, R21, U34, K awardsNIH R03, R21, U34, K awards

• OtherOther– disease foundationsdisease foundations– pharmaceutical companiespharmaceutical companies– mentorsmentors

• Can sometimes be done without fundingCan sometimes be done without funding

IRB Approval for PilotsIRB Approval for Pilots

• Like any study, pilots involving people Like any study, pilots involving people (or medical records) need IRB approval(or medical records) need IRB approval

• Testing questionnaires, developing Testing questionnaires, developing yoga postures, etc. that are devised and yoga postures, etc. that are devised and pre-tested with advisors and staff pre-tested with advisors and staff generally do not need IRB approvalgenerally do not need IRB approval

Nuts and BoltsNuts and Bolts

• ResourcesResources •• Contracts Contracts

• BudgetsBudgets •• Study team Study team

• SpaceSpace •• Recruitment Recruitment

• Start-upStart-up •• Forms Forms

• Adherence and follow-upAdherence and follow-up

• Money MattersMoney Matters •• Closeout Closeout

Funding SourcesFunding Sources

• Small trialsSmall trials–NIH - R03, R21, R01NIH - R03, R21, R01–Specialty organizations/foundationsSpecialty organizations/foundations• AHA, ACS, ACOG, ADA, etcAHA, ACS, ACOG, ADA, etc• FoundationsFoundations

–Pharmaceutical, device companiesPharmaceutical, device companies

• Large trialsLarge trials–NIH - R01, P01, U01NIH - R01, P01, U01–VA, AHRQ, CDC, DODVA, AHRQ, CDC, DOD–Pharmaceutical, device companiesPharmaceutical, device companies

UCSF ResourcesUCSF Resources

• Senior colleagues in your disciplineSenior colleagues in your discipline

• Experienced trialists and trial staffExperienced trialists and trial staff

• CTSI BREADCTSI BREAD–designdesign–ethicsethics–statistical issuesstatistical issues–database development and managementdatabase development and management

ContractsContracts

• Required if trial supported by industryRequired if trial supported by industry

• Must be approved by UCSF Industry Must be approved by UCSF Industry Contracts & company’s legal divisionContracts & company’s legal division– lots of puzzling legaleselots of puzzling legalese–you should understand and revise, esp. you should understand and revise, esp.

scope of workscope of work–best to start with UCSF template, revise best to start with UCSF template, revise

and submit to companyand submit to company

• Between UCSF Regents and companyBetween UCSF Regents and company–signed by PIsigned by PI

Elements of a ContractElements of a Contract

• Scope of workScope of work

• Timetable for deliverablesTimetable for deliverables

• Budget and timing of paymentsBudget and timing of payments–get a bolus up frontget a bolus up front–don’t link payment to deliverables out of don’t link payment to deliverables out of

your controlyour control

• Ownership of data and publishing rightsOwnership of data and publishing rights

• Rules for breaking contractRules for breaking contract

• Confidentiality, indemnificationConfidentiality, indemnification

• Patents and inventionsPatents and inventions

BudgetsBudgets

• NIH-style (NIH, VA, AHRQ, CDC)NIH-style (NIH, VA, AHRQ, CDC)–prepared far in advanceprepared far in advance–governed by strict rulesgoverned by strict rules• funds restricted to categoriesfunds restricted to categories• permission for carry-over requiredpermission for carry-over required• certain expenses not allowedcertain expenses not allowed• subject to government auditsubject to government audit

• Pharmaceutical company budgetsPharmaceutical company budgets–prepared just before trial startsprepared just before trial starts–negotiablenegotiable–much more flexiblemuch more flexible

Money MattersMoney Matters

• Pre-award managerPre-award manager–help prepare budget and “face” pageshelp prepare budget and “face” pages–help with budget justificationhelp with budget justification–make sure you follow rules/timelinesmake sure you follow rules/timelines

• Post-award managerPost-award manager–pay salaries, buy equipment and suppliespay salaries, buy equipment and supplies–monthly report of expendituresmonthly report of expenditures–projections over life of trialprojections over life of trial

Who is the Study Team?Who is the Study Team?

• Principle investigatorPrinciple investigator

• Project director/clinic coordinatorProject director/clinic coordinator

• RecruiterRecruiter

• Data managerData manager

• Programmer/analystProgrammer/analyst

• StatisticianStatistician

• Quality control supervisorQuality control supervisor

• Administrative assistantAdministrative assistant

• Financial/personnel managerFinancial/personnel manager

How to Hire the Study TeamHow to Hire the Study Team

• Work with your department or Work with your department or unit personnel (HR) managerunit personnel (HR) manager–write a job descriptionwrite a job description–decide on job series, step, salarydecide on job series, step, salary–post the job at UCSF and advertisepost the job at UCSF and advertise– review resumesreview resumes– interviewinterview–selectselect

Where to Find the Study TeamWhere to Find the Study Team

• Other studiesOther studies

• UCSF employeesUCSF employees

• Recent graduates, studentsRecent graduates, students

• Friends and colleaguesFriends and colleagues

• Chronicle, webChronicle, web

Training the Study TeamTraining the Study Team

• Describe, emphasize and demonstrate Describe, emphasize and demonstrate the importance of:the importance of:– following the protocol/operations manualfollowing the protocol/operations manual–meeting recruitment goalsmeeting recruitment goals–complete adherence and follow-upcomplete adherence and follow-up– full outcome ascertainmentfull outcome ascertainment–maintaining participant safety and maintaining participant safety and

confidentialityconfidentiality–ethical conduct of researchethical conduct of research

• Formal training and certification of staffFormal training and certification of staff

Operations Manual and Operations Manual and Training MeetingTraining Meeting

• Purpose and design of the trialPurpose and design of the trial

• Each measurement and visit procedureEach measurement and visit procedure–methodsmethods– qualified personnelqualified personnel– calibration and testing calibration and testing

• Completing and altering study formsCompleting and altering study forms

• Data entry or submissionData entry or submission

• Safety and ethical issuesSafety and ethical issues

• Certification and training of new staffCertification and training of new staff

Team LeadershipTeam Leadership

• Management training usefulManagement training useful

• You are responsible and the bossYou are responsible and the boss

• Keep your staff happyKeep your staff happy– involvement in the science involvement in the science –salary, working conditionssalary, working conditions– level of responsibilitylevel of responsibility

• Set an exampleSet an example

• Promote good staff interactionsPromote good staff interactions

• Some staff should be firedSome staff should be fired

SpaceSpace

• ConsiderationsConsiderations–accessibility, parkingaccessibility, parking–design and décordesign and décor–privacyprivacy–clinical needs, special testsclinical needs, special tests–safety measuressafety measures–costcost

• CTSI CRCsCTSI CRCs

• On-campus or medical spaceOn-campus or medical space

• Off campus rentalOff campus rental

RecruitmentRecruitment

• Hire an experienced Hire an experienced recruiterrecruiter

• Provide adequate Provide adequate time and moneytime and money

• Monitor resultsMonitor results

• Make changesMake changes– different approachesdifferent approaches–more centersmore centers– longer recruitmentlonger recruitment– change eligibilitychange eligibility

0

20

40

60

80

100

120

140

160

1 2 3 4 5 6

Month

Goal

Actual

RecruitmentRecruitment

• MailingMailing– Targeted (age, gender, disease, etc)Targeted (age, gender, disease, etc)• UCSF Integrated Data RepositoryUCSF Integrated Data Repository• KaiserKaiser• DMV, HCFA, registriesDMV, HCFA, registries

– MassMass• DMV, Voter Registration, HCFADMV, Voter Registration, HCFA• Private companiesPrivate companies

– EmailEmail

• Follow HIPPA and confidentiality rulesFollow HIPPA and confidentiality rules

RecruimentRecruiment

• Referral from providersReferral from providers

• AdvertiseAdvertise– hospital, clinics, special sites, churcheshospital, clinics, special sites, churches– radio, newspaper, TV, internetradio, newspaper, TV, internet– celebrities, leaders follow HIPPA and celebrities, leaders follow HIPPA and

confidentiality rulesconfidentiality rules

• CTSI planning a recruitment serviceCTSI planning a recruitment service

Start-upStart-up

01020

30405060

708090

Effort Time

Start-up

Trial

• ProtocolProtocol

• Operations ManualOperations Manual

• FormsForms

• IRB approvalIRB approval

• HiringHiring

• TrainingTraining

• Database designDatabase design

• Database validationDatabase validation

Forms and Database DevelopmentForms and Database Development

• Define variables, create data dictionaryDefine variables, create data dictionary

• Determine method of data entryDetermine method of data entry– Humans Humans – Machine readable formsMachine readable forms

• Design database (CTSI Consultation)Design database (CTSI Consultation)

• Use prior forms - forms librariesUse prior forms - forms libraries

• Pretest, validatePretest, validate

Adherence and Follow-upAdherence and Follow-up

• Monitor ratesMonitor rates

• Obtain contact data at baselineObtain contact data at baseline

• Pleasant, professional staffPleasant, professional staff

• Develop personal relationshipsDevelop personal relationships

• Encourage resumption of Encourage resumption of intervention and return to follow-upintervention and return to follow-up

• Cards, small gifts, meetings, partiesCards, small gifts, meetings, parties

Quality ControlQuality Control

• Standard Operating ProceduresStandard Operating Procedures– protocolprotocol– operations manualoperations manual– statistical analysis planstatistical analysis plan– data and safety monitoring plandata and safety monitoring plan

• Training and certification of staffTraining and certification of staff– ethical conduct of researchethical conduct of research– measurements and other visit proceduresmeasurements and other visit procedures– data entrydata entry

• Performance reviewPerformance review

Quality ControlQuality Control

• Special procedures for quality controlSpecial procedures for quality control– study drugstudy drug– laboratory procedureslaboratory procedures– data qualitydata quality• missing, out of range, illogicalmissing, out of range, illogical• variation by site, investigator, staff variation by site, investigator, staff

membermember

• Periodic checks of source dataPeriodic checks of source data

• Periodic reports (missing forms/data, Periodic reports (missing forms/data, queries, differences, drift)queries, differences, drift)

CloseoutCloseout

• Final visit and post-trial plansFinal visit and post-trial plans–make final measurementsmake final measurements–say goodbye and thank yousay goodbye and thank you–inform of test resultsinform of test results–?inform of treatment status?inform of treatment status–?inform of trial results?inform of trial results–?make clinical recommendation?make clinical recommendation

SummarySummary

• Trails are costly and complicatedTrails are costly and complicated–get advice from experienced colleaguesget advice from experienced colleagues–get materials used in prior trialsget materials used in prior trials–use UCSF servicesuse UCSF services• financial, legal, IRB/CHRfinancial, legal, IRB/CHR• CTSI CRCsCTSI CRCs• CTSI Consultation ServiceCTSI Consultation Service–Design, ethics, statistics, database Design, ethics, statistics, database

managementmanagement