Research Methodology

How to write a research protocol (O’Brien 2002)

Definition

A protocol is a document that explicitly states the reasoning behind and structure of a research project.

Advantages1. It states the question you want to answer.

2. To plan the project in detail, before you start.

3. To see the total process of your project.

4. It acts as a guide for all personnel involved in the project.

5. It enables you to monitor the progress of the project.

6. It is necessary if you need to apply for funding or ethical approval

Components of a research protocol

A. The problem to be investigated

1. Project title; should be short and concise

2. The research problem; eg: In this study I intend to find whether the use of

a fixed functional appliance (the Herbst appliance) will result in greater

skeletal change than a removable functional appliance (the Twin Block).

If I can show that this occurs this will be an important finding for

orthodontic care.

3. Background (including the literature review); The most important feature

of the background to the project is that it should be brief and to the point.

For a research protocol the background should be no longer than two

pages of A4 paper. In this section, you should concisely review the

literature that is relevant to the problem that you are trying to solve. In

this respect, it is probably good practice to limit the number of papers

quoted to less than 20. When you write the review, you should draw

attention to the good points and the deficiencies of the studies quoted.

You should also remember that it does not always mean that if a study

has been published in a journal, it is flawless in its methodology and

conclusion. The literature review should logically lead to the statement of

the aims of the proposed project.

4. The aims; the aims of the project should be explicitly stated. These should

be confined to the intention of the project and they should arise from the

literature review.

5. The hypothesis. It is general practice that hypotheses are stated in the null

form, because they have their basis in inferential statistics. You challenge

the hypothesis of no difference. The result of statistical testing gives the

probability that the hypothesis of no difference is true.

B. Method of investigation

In a study protocol, the method should be stated in the future tense. 

1. Subjects; it includes:

The population the subjects will be drawn from.

The total number and the number in any subgroups within the

investigation.

The inclusion and exclusion criteria for the subjects.

2. Design; which involve mentioning the method of randomization,

patient registration and allocation.

3. Experimental procedure; describe the method of treatment and how to

collect the data and when

4. Materials, measurements, and apparatus used; describe the parameter

that will be measured (like OJ or MR)

5. Sample size calculation;

6. The statistical methods that you are going to use.

7. Project milestones (Deadline) This section is not essential.

Nevertheless, it does provide a guide (and reminder!) for you and your

supervisor to inform if you are ahead or behind schedule with your

project.

8. Method of dissemination of findings (method of publication): Again,

this is not always essential, but it does let the reader know what you

intend to do with the results of the study. It is occasionally possible to

list the potential titles and publication strategy of the investigators.

However, this can sometimes be considered an over optimistic

approach.

9. Resources required: Finally you should make a list of all the resources

that you are likely to require to successfully complete your

investigation. If these resources have cost implications, you should

also note the potential cost of the investigation.

Advantages of RCT7. The RCT is prospective. As a result, the subjects and the data are under

some control by the investigator.

8. The treatment or intervention is randomly allocated. Therefore, the

perceptions of the investigator on the value of a particular treatment

should not influence treatment allocation that could bias the results.

9. The study is planned before the data is collected. This is the important

distinction between the RCT and the retrospective investigation, and

these results in a minimization of bias that is inherent in the retrospective

study.

RCT Study design (O’Brien 2003)1. Select aims and State objectives, objectives should be capable of

statistical description and analysis

2. Null hypothesis (always assumes there no difference)

3. Outcomes: It is important to know before you start how you are going to

measure the effect of each intervention.

4. The study population and site of the study: This is an important step

because it is important that this population is relevant to both the question

that we hope to answer and to the provision of orthodontic treatment.

5. List of inclusion and exclusion criteria for the study.

6. Power calculations

7. Define the measuring techniques and instruments

8. Define the control group

9. Ethical issues, e.g. - welfare of subjects, should time and money be spent

researching this subject, patient consent

10.Patient registration:

Patient requires treatment and is eligible

Agreement to randomize

Patient consent

11.Formal entry: Details of the patient are then entered onto a log sheet of

the trial or, more commonly, onto a computer database.

12.Randomization: the object of randomization is to allocate one or more

interventions (or control), in a manner that ensures that the samples that

you are going to compare, are similar in every respect apart from the

intervention. In most trials, a randomization list has been prepared in

advance using random numbers.

13.Allocation: The next stage is the method by which the operator finds out

which treatment the patient has been assigned to. It is essential that the

operator does not know what the assignment will be in advance and there

are several methods of concealing this. One popular method is to transfer

the randomization list to a series of sealed envelopes each containing the

allocation on a card. The clinician then opens the next envelope in the

series when the patient formally enters the trial. This method is

particularly relevant when the clinician registers his/her own patients.

However, care needs to be taken to ensure that the clinician does not

reseal the envelope having discovered that the allocation was not what

he/she was hoping! The best method of allocation is to make use of a

central registration office. In this method the treatment assignment is read

from a prepared list and given to the investigator while still on the phone,

following the registration of the patient. While this method is more

expensive and requires more preparation than using envelopes it does

provide an almost foolproof method of allocation.

14.Blinding may occur in many ways, for example, blinding the patient, the

operator, the investigator who measures the outcomes and the

statisticians. However, when we consider the nature of orthodontic

treatment it is impossible to blind treatment allocation to both the

operator and patient. As a result, the only type of blinding that we can

practice is blinding of the person who records and analyses the data. This

is important because if, for example, the evaluator knows that a group of

patients have had a new treatment then they may record outcome data in a

favourable manner. Blinding can be done by concealing the identity of

the patient and the treatment allocation using numbers, or by having the

data recorded by a different person from the one who is going to analyse

the data. 

15.Monitoring progress: So now you have set up your trial, and you think

that you can just sit back and the trial will run, and all you have to do is to

collect and analyse the data. Unfortunately, this is not the case! Several

areas should be evaluated as part of this process.

The first of these is protocol compliance. You need to check that the

study protocols are being followed by the operator(s) in the study. The

easiest way of doing this for an orthodontic study is to periodically look

at the records of the patients in the study and check for protocol

deviations that are recorded.

You should also check for adverse effects.

Finally, a careful record of all study withdrawals or drop-outs should be

made, and as much baseline data as possible recorded. This will ensure

that a statistical check can be made to discover whether the drop-outs

were similar to those people who remained in the trial

16.Interim data analysis is useful if:

you need some data to present at a conference,

check that the treatments are not causing harm, which is important for the

ethics of the trial.

17.Stopping rules are defined at the start of the trial to ensure that there is a

‘safety valve’. If, for example, it becomes obvious during a trial that one

or more treatments is significantly worse or better than another, then the

trial should be stopped.

18.Data analysis: Methods of data analysis for RCTs do not markedly differ

from other orthodontic studies. However, it is important to consider the

difficult question of how to handle data from patients who dropped out of

the investigation. When this occurs we are left with several choices. The

first is to report the number of patients who withdrew from the

investigation and emphasize that the two interventions under

investigation had certain success and failure rates. Or the data analysis

should include the results of the treatments on all the patients who entered

the study, regardless of successful compliance or completion of the

treatment. This is termed an intention to treat analysis (ITT analysis).

This type of approach results in a measure of the true effectiveness of the

treatment and should be attempted wherever possible. One possible

drawback of this approach with orthodontic treatment is that we may not

have collected data on the patients who dropped out of the investigation,

as they may not have returned to the clinic. One solution to this is to

statistically impute data to compensate for the lost data. Several statistical

packages have the ability to be programmed to carry out this type of

analysis.

19.Writing up the project by following the Consolidated Standards of

Reporting Trials (CONSORT) guidelines developed to assess reporting of

randomized clinical trials in journals, (Moher 2010 and Altman 2001)

20.In all RCT, a data and monitoring board for the safety (DSMB) of the

trial is important and crucial.

21.Publication

Systematic review study design and protocol planning1. Create a rationale and aims of the study

2. Outcomes: It is important to know before you start how you are going to

measure the effect of each intervention.

3. Set inclusion and exclusion criteria

4. Conduct a search

Electronically via MEDLINE ‘’American version’’, EMBASE

‘’European version’’, Cochrane library ‘’CENTRAL’’

manually

different languages

Gray literatures.

The search can depend on the use of

Single vocabularies (specific word) and free text vocabularies (in the

whole text) to increase search sensitivity

It can be combined with Boolean operators ‘’AND, NOT, OR’’, or

truncation and wildcard by using this symbol ‘‘?’’(child? mean different

extension of the word, eg, childhood, children, child’s), proximity

operators (eg, dental near/6 anxiety, mean any article contains dental and

anxiety within 6 word closeness)

It can be filtered according to the relevant design of study (RCT or case

control)

5. Review the included studies

6. Reject non-relevant studies

7. Assess the methodology of relevant studies

8. Exclude if weak methodology

9. Extract data from the remaining studies

10.Analyse data

11.If the methodology was similar then combine them to increase the sample

size and increase the power of study

12. Summarize and draw conclusion

Classification of studiesIn general they classified into observational and experimental studies.

Research method

Key features

Case Report A small case series describing the outcome of

treatment of a few (less than 5-10) cases or

reporting potential problems with treatment.

Cross-

sectional

Data collected from sample members on one

occasion.

1. Used to study prevalence (called

descriptive cross sectional) and

relationship between variables (analytical

cross sectional). It can be used to create a

new question which need to be answer by

subsequent powerful study. It can be used

to formulate the normative for patient

(like Bolton norms).

2. Advantages: low cost, short time, can

screen large sample,

3. Disadvantages: not strong, weak external

validity,

Longitudinal Data collected from sample members on two or

more occasions.

Cases control Describes what happens to patients without

actively intervening with the treatment they

receive and they are retrospective with separate

control group.

Cohort study Describes what happens to patients without

actively intervening with the treatment they

receive and they are prospective with separate

control group.

The disadvantages are the cost and long

duration with many confounding factors that

could appear.

Clinical trial Assess whether one health care intervention is

better than another, a placebo or no treatment.

Are prospective and controlled. Allocation to

test/control groups is predetermined.

Control gp can be ethically used if the new

intervention is equipoise which means a

presumed equality between new and old

therapy.

Weakness of RCT is the cost and time as well as

weak external validity (which means the result

not represent the large population sample

because most of the sample who recruited are

either sicker or more compliant than an

individual in community).

RCT represent the intervention efficacy under

ideal conditions rather than effectiveness under

real condition.

RCT not always give the final conclusion about

certain treatment and intervention, so the

outcomes could be followed for a period of time

after applying the new intervention in a system

called ‘’post market surveillance programme’’

No control

trials Quasi-experiment or randomization

Literature

control

Comparison made to information on patients in

a published paper or growth study. Prone to

chronological and/or geographical bias.

Historical

control

Comparison made with patients treated

previously in the same unit/place. Prone to

chronological bias.

Matched

control

Comparison made with patients who are similar

in respect to one or two specific criteria. Prone

to allocation bias.

Clinical trial

Review article Traditional Summarizes information from several

review previously published papers on a specific topic.

Systematic

review

Papers are identified, critically appraised and

the results synthesized according to a defined

protocol.

Meta-

analysis

Combines the results from several different

clinical trials to obtain an overall estimate of the

effectiveness of a particular intervention.

The Cochrane

Collaboration

• launched in 1995

• Summarize the evidences and

provides a database of controlled trials

and systemic reviews

• provides guidelines on preparation of

good clinic trials and systemic

reviews

Which study design for which aim?

For a question about.....

....these are the most appropriate research

methods

Example

Therapy Clinical trial Competing interventions,

a placebo, no treatment.

Diagnosis or screening Cross-sectional survey Prevalence of orthodontic

treatment need.

Prognosis Cohort study or

longitudinal survey

Long-term stability of

mandibular incisors.

Causation Case control study or

cohort study

Factors influencing root

resorption following fixed

appliance treatment.

Summary of evidence Systematic review Orthodontic treatment for

posterior crossbites.

Validity: the extent to which the value obtained represents the object of

interest

Reproducibility: the closeness of successive measurements, errors in

measurement can be both systematic and random

Reliability: often used in a broader sense to encompass both

reproducibility and validity

Systematic error: a tendency to overestimate or underestimate a

parameter giving a biased

Random error: this is the variability due to chance. Increasing the number

of observations e.g. repeating measurements and taking an average value

is an important way of reducing random error

There are four main biases that can affect the internal validity of a study.

1. Selection bias

The term selection bias is used in different ways within the medical

literature.

It is often used in relation to bias occurring during the selection of

representative participants or to bias occurring during the selection of

participants to exposures.

2. Performance bias

Performance bias refers to systematic differences in care provided to

participants in a study.

Blinding of study participants and investigators to treatment allocation

helps minimize performance bias.

When a study is described as single blind only the participants are blind

to their group allocation.

When both participants and investigators are blind to group allocation the

study is described as double blind.

3. Measurement bias

Even when blinding to treatment groups cannot be achieved, blinding to

outcome assessment is usually possible.

In orthodontics, this can be achieved by blind assessment of study

models, radiographs and/or photographs.

This can help minimize systematic differences that may occur in how

outcomes are determined from the groups under comparison

(measurement or detection bias).

Error estimation or measurement • Estimation of reproducibility error is an important part of

experimental study

• Error estimation normally involves the replication of a certain number

of observations

• Replicated measurements should be chosen at random and after a

suitable time interval following the initial measurements

• Important to distinguish between systematic and random error

• The difference between the repeated measurements gives a measure of

the systematic error

• The SD of the difference gives a measure of the random error

• Quoting these two values allows differentiation between random and

systematic error

• The error variance should not be greater than 10% of the total variance

4. Attrition bias

It occurs when there are systematic differences between comparison

groups in withdrawals or exclusions of participants from the results of a

study.

For example, patients may drop out of a study because of side effects of

the intervention or difficulty in wearing a particular appliance.

Excluding these patients from the analysis could result in an over-

estimate of the effectiveness of the intervention.

Conversely, participants might drop out of a study due to an

improvement in the symptoms or malocclusion, e.g. overjet or

crowding, resulting in an under-estimate of treatment effect if they are

not included in the analysis.

In order to minimize attrition bias, all study participants should be

accounted for in the analysis and the analysis undertaken on an

intention-to-treat basis i.e. participants are analysed according to the

group to which they were initially allocated, regardless of whether they

dropped-out, fully complied with the treatment or ended up crossing

over to the other treatment group

Types of errors (Macfarlane, 2003)

There are two types of errors that should be taken into account when

designing a study.

A type I error is the error of wrongly rejecting the null hypothesis when

it is true. The level of significance is defined as the probability of

making a type I error and is denoted by α. In order to guard against type

I errors it is usually set to small values such as 0.05.

A type II error is the error of wrongly accepting the null hypothesis

when it is false. The probability of making a type II error is denoted by

β. The power of a hypothesis test is equal to 1 – β, and is often

expressed as a percentage, rather than a proportion. In medical research,

it is frequently set at least to 80%.

Sample size (Macfarlane, 2003)

It depends on:

1. Type I error (alpha error or statistical significance)

2. Type II error

3. Clinical significant (how does one decide what difference is worth

detecting? Sometimes this information comes from clinical experience

or similar studies in other areas, but often a small pilot study is

required).

4. SD

5. The pre-known mean difference of the object to be measured