questionnaire design, pretesting and quality control in clinical research celia p. kaplan division...
Post on 01-Jan-2016
223 Views
Preview:
TRANSCRIPT
Questionnaire Design, Pretesting and Quality Control in Clinical Research
Celia P. KaplanDivision of General Internal Medicine
August 24th 2009
Good Clinical Practice (GCP)
“A standard for the design, conduct, performance, monitoring, auditing, recording, analysis, and reporting of clinical trials that provides assurances that the data and reported results are credible and accurate, and that the rights, integrity, and confidentiality of trial subjects are protected”
WHO good clinical practice standards
Quality Control
Quality Control
Minimizes the chance of obtaining faulty data and drawing erroneous conclusions
Quality Control
Minimizes the chance of obtaining faulty data and drawing erroneous conclusions
Fulfills administrative and regulatory obligations
Sources of Data Errors Data acquisition
Sources of Data Errors Data acquisition
instrument problems
Sources of Data Errors Data acquisition
problems instrument lost data
Sources of Data Errors Data acquisition
instrument problems lost data falsified data (made up data, changed )
Sources of Data Errors Data acquisition
instrument problems lost data falsified data (made up data, changed)
Data recording
Sources of Data Errors Data acquisition
instrument problems lost data falsified data (made up data or changed)
Data recording Data entry
Sources of Data Errors Data acquisition
instrument problems lost data falsified data (made up data or changed)
Data recording Data entry Data management
Quality Control Stages Pre-implementation/Developmental Stage
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires/interviews and forms
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires/interviews and forms
Pretest all aspects of the project
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires/interviews and forms
Pretest all aspects of the project Develop an Operations Manual
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring
reports
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings Complete interim staff training and
performance reviews
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires/interviews and forms
Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings Complete interim staff training and performance
reviews
Questionnaires/interviews Much of the data in clinical and
epidemiological research is gathered using questionnaires and interviews
Questionnaire Measures
Desired qualities Reliability – the extent to which the
measures give consistent results A reliable measure is consistent, responses do not
change Test-retest reliability Statistical techniques
Validity – the extent to which the measures reflect the “truth”
Survey measures: Validity Face validity: the researcher’s best judgment as
to whether the item is measuring what it is supposed to measure
Predictive validity: the item accurately predicts an outcome (e.g., college entrance exams predicting college performance)
Concurrent validity: the item is validated by comparing it against a known valid measure (e.g., a measure of job satisfaction might be correlated with work performance)
Survey measures: Validity Content validity: the items are a representative
sample of the characteristics that comprise the area they intend to measure. Determination of content validity usually relies on experts Example: Patient decision making style
Construct validity: the item demonstrates that it accurately measures a psychological construct like depression or satisfaction
Questionnaire MeasuresDesired qualities Reliability – the extent to which the measures give consistent resultsValidity – the extent to which the measures reflect the “truth”
Questionnaire MeasuresDesired qualities Reliability – the extent to which the measures give consistent resultsValidity – the extent to which the measures reflect the “truth”
Questionnaire Measures Desired qualities
Accommodate participants’ age, literacy,
cultural, linguistic, and SES characteristics
Yield good variability in answers
Result in low percent of missing data
Questionnaire Development: Pitfalls
Double-barreled questions – Avoid “and/or” statements in questions e.g., “Does your department have a special
recruitment policy for ethnic minorities or women?”
Long questions and/or questionnaires
Use of “other” as a response category
Use of Other as Response Category
Example
10. Are you Latino/a or Hispanic? 1 Yes
0 No
11. What is your race/ethnicity? Please check one answer only. 1 Black or African American 2 Asian, Asian American or Pacific Islander 3 White, European American or Caucasian 4 American Indian or Alaska Native 5 Other Please specify ____________________
Use of Other as Response Category
Example1 Ameriasian 19 INDIAN (EAST) AMERICAN
2 American 20 Italian
3 Arab 21 MIDDLE EAST
4 Ashkenazi 22 Middle Eastern/ Indian Subcontinent
5 Asian and Caucasian 23 Mixed
6 Brazilian 24 NATIVE AMERICAN
7 Cuban 25 Oriental
8 East Indian 26 PERSIAN
9 Egyptian 27 POLISH AMERICAN
10 EGYPTIAN 28 South Asian
11 egyptian american 29 Southern Asian
12 From India 30 West Indian
13 half Latin, half black 31 White, Arab
14 Indain American 32
15 India- Indian 33
16 Indian 34
17 INDIAN 35
18 Indian (east) 36
Questionnaire Development: Order
First question should be a fact
Beginning questions should stimulate interest
Questions should be in logical order and groupings
If ask attitude questions first and then behavior questions, may inflate reports of behavior
Sensitive questions and open-ended questions should be at the end of the survey
e.g., sexual behavior, religion, income
Questionnaire Development: Language Use simple, clear, common, and
natural language Avoid jargon, complex terms, clichés,
colloquialisms, e.g., “down in the dumps” Check number of syllables in words Check reading level: aim for < 8th grade Define ambiguous terms, e.g., “family” or
“neighborhood” Keep sentences short
Questionnaire Development: Recall
Solutions: Narrow the reference period Average the response: ask about a typical
day or in general (e.g., hours of sleep) Use landmark events or milestones to
anchor time frame: “Since you moved to the US,” or use calendar with major holidays to establish major life events
Use cues to generate associations to jog memory: describe procedure, setting
Questionnaire Design
1Select Interview Methodology
Interviewmethodology
Questionnaire Design
2Review existing
questionnaires and measures
Interviewmethodology
Reviewexisting
measures
Questionnaire Design
3Conduct exploratory
qualitative work
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
Questionnaire Design
4Develop a draft survey
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
Draft survey
Questionnaire Design
5Translation and back-translation
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
Draft survey
Translation/back-translation
Questionnaire Design
6Pre-testing
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
Draft survey
Translation/back-translation
Pre-test the survey
Questionnaire Design
6Pre-testing
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
Draft survey
Translation/back-translation
Pre-test the survey
Questionnaire Design
7Finalize the Questionnaire
Interviewmethodology
Reviewexisting
measures
Conductexploratory
work
Draft survey
Translation/back-translation
Pre-test the survey
Finalizedinstrument
Questionnaire design
Selection of interview methodology
Face-to-face or in-person interview
Telephone
Self-administered, paper
Self-administered, computer/web/e-mail
Questionnaire design
Interview methodology: Face-to-Face
Advantages
Enables the interviewer to establish rapport with respondent
Permits more complex questions Permits the use of visual aids Longer interviews are possible
Disadvantages Cost per interview Quality of answers too personal (sensitive
questions)
Questionnaire design
Interview methodology: Telephone Advantages
Faster contact of participants Better to elicit personal/sensitive information Results are available right away if using
computer-assisted interviewing (CAI) Can do random selection of telephone numbers
Disadvantages Certain sectors of the population are switching to
cell phones only More expensive than mail surveys Difficulties in reaching participants during the day
Questionnaire design
Interview methodology: Self-administered and mail surveys (Paper)
Advantages One of the least expensive methodologies Needs addresses of potential participants Allows participants to answer at their leisure Less intrusive Elimination of interviewer bias
Disadvantages Response rate is low Difficulties to develop clear skip patterns It may be answered by a family member Very difficult to implement in low literacy populations
Questionnaire design
Example: Self-administered
& Mail Surveys
Questionnaire design
Interview methodology: Self-administered, web-based surveys (Computer)
Advantages Elimination of data entry and editing costs Get better answers to sensitive questions Elimination of interviewer bias Ensure that skip patterns are accurately followed Allows participants to answer at their leisure
Disadvantages Interviewees must have access to a computer Must possess (or purchase) a list of email addresses Lower response rate People may quit in the middle of a questionnaire
Questionnaire designExample: Web-based survey
Questionnaire design
Survey modalities: Answer types
Numeric open-endedHow old are you? ____
Text open ended How can we improve our services? ______________
Multiple Choice--CategoricalWhat is your marital status
Single 1 Married 2
etc.
Survey modalities: Answer types Multiple choice-- Ordinal
In general, would you say your health is…?1 Excellent2 Very good3 Good4 Fair5 Poor77 DK
Scales On a scale where “10” means you have a great amount
of interest in a topic and “1” means you have none at all, how would you rate your interest in each of the following specialties?
Cardiology ___ Oncology __ Dermatology__
Questionnaire design
Survey modalities: Answer types Multiple choice-- Ordinal
In general, would you say your health is…?1 Excellent2 Very good3 Good4 Fair5 Poor77 DK
Scales On a scale where “10” means you have a great amount
of interest in a topic and “1” means you have none at all, how would you rate your interest in each of the following specialties?
Cardiology ___ Oncology __ Dermatology__
Questionnaire design
Survey modalities: Answer types Likert-Type scales
Used to know respondents' feelings or attitudes The respondent indicates how closely their feelings
match the question or statement on a rating scale. Response categories: “Strongly Disagree" (least
agreement) to or "Strongly Agree." (most
agreement)
Example: In general, to what degree is each of these factors a
barrier for you in referring a patient to a clinical trial? Not a barrier A major barrier
1 □ 2 □ 3 □ 4 □ 5 □
Questionnaire design
Questionnaire design
Review Existing Questionnaires & Measures
Reference databases Medline, Pubmed, Psychinfo, others
Compendia of measures Books that compile various measures
and review their characteristics
Request questionnaires from other investigators
Questionnaire design
Review Existing Questionnaires & Measures
Is there evidence the instrument works in your target population?
Examine validity and reliability of measures when used in populations similar to your study population
When gold standard does not exist, for construct validity, look at relationship of measure to other measures to see if in hypothesized direction
Questionnaire design
Conduct Qualitative Exploratory Work
Focus groups or semi-structured interviews Explores a topic before constructing formal
questions Rationale:
Determine what is important Discover how respondents think about the topic Assess better language/words to use Identify response categories
Questionnaire design
Create a Draft Survey Instrument
Write initial draft Process of revision and refinement through
cycles of translation/back-translation, reviews and pre-tests (Steps 5 and 6)
Questionnaire design
Translation/Back-translation Procedures
Both versions should have the same meaning Translation and back translation Review by a group of native speakers Challenging when more than one language
involved in the study
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings Complete interim staff training and performance
reviews
Quality Control
Pre-testing the Questionnaire: Overview
Refines Instruments and questionnaires
Evaluates Recruitment methods
Interventions
Data entry and management system
Protocols
Assists with training of personnel
Quality Control
Pre-testing the Questionnaire
Good to test: Sensitive questions Complex or poorly defined topics Questions with terms respondents may not
understand Translations Layout and instructions
Quality Control
Pre-testing the Questionnaire:Field Testing Techniques
Expert reviews Cognitive interviews Full pretest
Quality Control
Pre-testing the Questionnaire:Expert Reviews Formal and systematic examination of
a questionnaire by experts in the field Can cover question wording, layout,
and skip patterns Fast method prior to formal testing May not be a complete review
Quality Control
Pre-testing the Questionnaire:Cognitive Interview: Overview Diagnostic tool for pretesting survey
questions derived from social and cognitive psychology
Interview about an “interview” Explores the processes by which
respondents reach answers Based on structured questionnaire and
protocol
Quality ControlPre-testing the Questionnaire:Cognitive Interview: Techniques Concurrent proving
Ask probes immediately after respondent has given answer to survey item
Advantage: information is fresh in respondent’s mind Disadvantage: Disrupts interview flow and relationship
between questions Retrospective probing
Ask probes after entire interview or block of questions Advantage: able to assess standard administration of
items Disadvantage: Participant may not remember thought
process
Quality ControlPre-testing the Questionnaire:Cognitive Interview: Techniques Probing questions Think aloud interviews
Interviewer asks respondent to think aloud as they answer question
Paraphrasing Respondents rephrase the question in their own words
Quality Control
Pre-testing the Questionnaire:Cognitive Interview: Key Features Comprehension of the question
Do participants understand words and phrases as intended by the researchers (meaning)
“How did you arrive at your estimate of risk?” “What did you understand by the word ‘risk’?”
Retrieve of information Identify process respondents use to answer questions Examine strategies used to access memory
Edit responses: Decide what to report Are items unacceptable? Is answer embarrassing,
socially undesirable?
Quality Control
Pre-Test
Pretest with a sample similar to the population to be studied
Simultaneously pretest recruitment procedures and questionnaire implementation
Quality Control
Pre-Test: Behavior Coding
Code interaction between interviewer and respondent Examples
R answers before hearing whole question R looks puzzled R asks for clarification I repeats question
Quality Control
Pre-Test: Respondent Debriefing
Structured follow-up questions at the end of the interview to assess: Whether questions were clear Ease of completing the questionnaires If two versions are tested, which is
preferable Whether an important question was
missed
Quality Control
Pre-Test: Interviewer Debriefing Interviewers implement pretest with
participants Interviewers detect problems with
questions, response categories, and skip patterns
Interviewers report issues to investigators
Quality Control
Pre-Test Questionnaire: Summary
Testing takes time and resources Some procedures take place late in the
developmental cycle, limiting the amount of change possible
Danger of making changes that are not further tested
Not testing leads to questionnaires that are potentially unreliable or will give you invalid results
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring reports Conduct periodic project meetings Complete interim staff training and performance
reviews
Quality Control
Operations Manual Aim is to ensure high quality data Clearly written, detailed written instructions to obtain
uniformity across sites, staff, and procedures Standardizes procedures for all aspects research project.
Defines how to… Recruit participants Consent participants Measure variables Transport clinical samples Enter data Monitor progress Conduct data analysis
Requires updates, clarifications, and recording of decisions made
Quality Control
Data Management System Desirable features
Range and field type checks Ease of screen set up and use Double data entry (if possible)
Security features protection of human subjects’ rights (privacy)
For Web-based systems Virus protection to monitor and eliminate security
threats Database server behind firewall All data backed up regularly
Quality Control
Staff Training Training and certification of the research team members
Rules to conduct interviews Rules for recruiting participants, including how to
obtain consent Methods
“Train the trainer” model Audio-visual techniques
Certification/recertification to maintain skills
Quality Control Stages Pre-implementation/Developmental Stage
Design high-quality questionnaires and forms Pretest all aspects of the project Develop an Operations Manual Design a secure data-management system Conduct staff training and certification
Implementation Make periodic study- and data-monitoring
reports Conduct periodic project meetings Complete interim staff training and
performance reviews
Quality Control
Data Monitoring Reports
Standard procedures for Identifying missing, incomplete data Identifying incorrect skip patterns Identifying inconsistent and erroneous data Verifying correction
Quality Control
Interim Staff Training and Performance Reviews
Provide continuous supervision Hold frequent meetings of the study team Review all aspects related to implementation of
the study, including Interviewers activities
Listen to interviews Compare measures between interviewers Compare refusal rates between interviewers
Good Clinical Practice
Good Instruments Complete and clear protocol manuals Extensive pretesting Ongoing monitoring procedures
top related