Pre testing & analysing pilot dataDr Kirsten Challinor

COMMONWEALTH OF AUSTRALIACopyright Regulations 1969

WARNING

This material has been reproduced and communicated to you by or on behalf of theUniversity of New South Wales pursuant to Part VB of the Copyright Act 1968 (the Act).

The material in this communication may be subject to copyright under the Act. Any further reproduction or communication of this material by you may be the subject of copyright protection

under the Act.

Do not remove this notice.

Pilot studies

• What is a pilot study? Mini version of a full-scale study = 'feasibility' study Specific pre-testing of a particular research instrument

such as a questionnaire or interview schedule.

• Pilot studies are a crucial element of a good study design.

• Conducting a pilot study does not guarantee success in the main study, but it does increase the likelihood.

Image from http://www.eh.uc.edu/cares/learn/pilot_study.html

"Do not take the risk. Pilot test

first.” De Vaus (1993: 54)

Reasons to pilot• Developing and testing adequacy of research instruments • Assessing the feasibility of a (full-scale) study/survey • Designing a research protocol • Assessing whether the research protocol is realistic and workable • Establishing whether the sampling frame and technique are effective • Assessing the likely success of proposed recruitment approaches • Identifying logistical problems which might occur using proposed methods • Estimating variability in outcomes to help determining sample size • Collecting preliminary data • Determining what resources (finance, staff) are needed for a planned study • Assessing the proposed data analysis techniques to uncover potential problems • Developing a research question and research plan • Training a researcher in as many elements of the research process as possible • Convincing funding bodies that the research team is competent and knowledgeable • Convincing funding bodies that the main study is feasible and worth funding • Convincing other stakeholders that the main study is worth supporting.

Pilot studies• Pilot studies can be based on quantitative and/or qualitativemethods.

• Large-scale studies might employ a number of pilot studies before the main survey is conducted. One might:1. Qualitative data collection and analysis on a

relatively unexplored topic,2. Use the results to design a subsequent

quantitative phase of the study.

Image: http://collaborationforgood.org/tag/pilot/

• The NIEHS• Evidence to be used to

plan and prioritise the effective delivery of eye care for Indigenous Australians.

• Assess the prevalence and main causes of vision impairment.

• Assess the utilisation of eye care services, barriers to health and the impact of vision impairment on Indigenous people.

Pilot for The NIEHSPage 13 of National Indigenous Eye Health Survey- 1.4.1 The pilot study A pilot study titled ‘Evaluation of Selected Vision and Eye Conditions in Aboriginal and Torres Strait Islander Communities’, was conducted in Moree NSW.

The purpose of this study was to develop the procedures and conduct the preliminary pilot studies for the National Indigenous Eye Health Survey. This was to ensure that the processes and protocols were both valid and culturally appropriate for Aboriginal Australian people. The ethics for the study had been approved by the Aboriginal Health and Medical Research Council of NSW. The rapid assessment methodology utilised was based on the Vision Initiative in Victoria, and was compared to a gold standard eye examination by an eye care practitioner. The rapid examination was designed to detect the five predominant and significant eye conditions, including diabetic retinopathy, trachoma, glaucoma, refractive error and cataract. It was found that 90.4% of the retinal images obtained with the non-mydriatic camera were gradable, and that the rapid examination was able to detect proliferative diabetic retinopathy and macula oedema with very high sensitivity and specificity. The study found that the rapid examination was highly specific in correctly identifying normal vision in adults. The rapid assessment also showed high sensitivity in detecting cup to disc ratios greater than 0.6. The rapid assessment method used in this study resulted in a validated, rapid examination methodology which was able to detect diabetic retinopathy, risk of glaucoma, trachoma, refractive error and visual impairment in Aboriginal and Torres Strait Islander populations in Australia, when compared to a gold standard eye care practitioner examination, with minimal staff training and equipment (12).

Burnett, A. Adaptation & validation of a vision related quality-of-life tool for use in

indigenous communities

Link to PhD thesis: http://www.unsworks.unsw.edu.au/primo_library/libweb/action/dlDisplay.do?

vid=UNSWORKS&docId=unsworks_8655&fromSitemap=1&afterPDS=true

The steps used to pilot a questionnaire

Pilot study procedures to improve the internal validity of a questionnaire•Administer the questionnaire to pilot subjects in exactly the same way as it will be administered in the main study. •Ask the subjects for feedback to identify ambiguities and difficult questions. •Record the time taken to complete the questionnaire and decide whether it is reasonable. •Discard all unnecessary, difficult or ambiguous questions. •Assess whether each question gives an adequate range of responses. •Establish that replies can be interpreted in terms of the information that is required. •Check that all questions are answered. •Re-word or re-scale any questions that are not answered as expected. •Shorten, revise and, if possible, pilot again.

(Source: Table 3.23 in Peat et al. 2002: 123)

Limitations of pilot testing

• Making inaccurate predictions or assumptions on the basis of pilot data.

• Problems arising from contamination– Pilot data included in main study– Same participants re-tested

• Problems related to funding.

Image: http://www.saynotocrack.com/index.php/2007/01/

Pilot studies“under-discussed,

underused & underreported”

Prescott and Soeken, 1989 p60

Why are pilot studies not reported?• Publication bias - only papers that have

statistically significant results and not to report non-significant effects

• Selective publication of research results• overestimation of the effectiveness of

interventions = exposing patients to useless or harmful treatments.

• overestimation of adverse effects = patients are denied effective forms of care.

• It is important to share lessons learned otherwise patients may be subjected to poorly developed tools or money may be wasted because methods of recruitment failed.

Image: http://socialinfluenceconsultinggroup.com.au/the-pilot-was-an-influence-bungler/

The importance of reporting the Pilot study• When reported, they often only justify the research methods or particular

research tool used.

• Rarely reporting offers little about exactly what was learnt.

• Therefore investigators should be encouraged to report their pilot studies, and in particular to report in more detail the actual improvements made to the study design and the research process.

Image: http://www.mbaskool.com/business-articles/marketing/759-pilot-project-launch-evaluating-the-market.html

ReferencesPilot studiesEdwin R. van Teijlingen and Vanora Hundley. The importance of pilot studies, Issue 35 Winter 2001 Social Research Update is published quarterly by the Department of Sociology, University of Surrey, Guildford GU7 5XH, England. http://sru.soc.surrey.ac.uk/SRU35.html

Gillian A. Lancaster, Susanna Dodd, and Paula R. Williamson.Design and analysis of pilot studies: recommendations for good practiceJournal of Evaluation in Clinical Practice, 10, 2, 307–312

Indigenous eye surveyNational Indigenous Eye Health Survey Team. National Indigenous Eye Health Survey: Minum Barreng (Tracking Eyes) Full Report. Indigenous Eye Health Unit, Melbourne 2009. ISBN 9780734041142

Anthea Burnett PhD thesis (2009)Vision impairment in Aboriginal and Torres Strait Islander peoples: a toolkit to assess prevalence and impact.http://www.unsworks.unsw.edu.au/primo_library/libweb/action/dlDisplay.do?vid=UNSWORKS&docId=unsworks_8655&fromSitemap=1&afterPDS=true

Homework- read this paper