issues surrounding reliability and validity-1

7/30/2019 Issues Surrounding Reliability and Validity-1

1/38

Schizophrenia: issues surroundingdiagnosis There are several issues surrounding the

diagnosis of Schizophrenia that need to beassessed.

These include addressing issuessurrounding the reliability and validity ofdiagnosis.


2/38

DSM- IV The Diagnostic andStatistical Manual ofMental Disorder(DSM)(Edition 4), was

last published in 1994. The DSM is produced bythe American

Psychiatric Association. It is the most widelyused diagnostic tool inpsychiatric institutionsaround the world.


3/38

ICD - 10 There is also theInternationalStatisticalClassification ofDiseases (known asICD). It is produced by the

World HealthOrganisation (WHO)and is currently in its10th edition.


4/38

Reliability and validity of DSM-IVand ICD-10 Diagnosing a mental disorder is almost always

done using the DSM-IV and the ICD-10; although itcould be argued that the DSM is the most

commonly used. The main issues surrounding the diagnosis of

mental disorders such as Schizophrenia centre onthe reliability and validity of the diagnoses.


5/38

Reliability of Diagnosis Reliability: Q. Can psychiatrists agree on the diagnosis?

A. Generally = NO!

Validity:Q. How accurate and meaningful is the research?

A. Not very!


6/38

Reliability of Diagnosis Reliability refers to the consistency of

measurement in relation to the classification anddiagnosis of Schizophrenia in two ways:

Test-retest reliability: occurs when a practitionermakes the same consistent diagnosis on separateoccasions from the same information.

Inter-rater reliability: occurs when severalpractitioners make identical, independentdiagnosis of the same patient.


7/38

Test-retest reliability: Canpsychiatrists agree on diagnosis? Read et al (2004)

Reported test-retestreliability of schizophreniadiagnosis to have just 37%concordance rate.

They also noted Copelandet als (1974) study where134 American and BritishPsychiatrists were asked todiagnose a patient basedon a case description.


8/38

Test-retest reliability: Canpsychiatrists agree on diagnosis? 69% of American

psychiatrists diagnosedSchizophrenia v just 2% of

British. This suggests that the

diagnosis of Sz has never

been reliable.


9/38

Inter-rater reliability dopsychiatrists agree? Beck et al (1961)

looked at the inter-raterreliability between 2psychiatrists whenconsidering the casesof 154 patients.

The concordance rate(reliability)was only54% - meaning theyonly agreed on adiagnoses for 54% of

the 154 patients!

I wonder what

the other blokethinks?


10/38

However.Soderberg et al (2005) Reported aconcordance rate of 81% using thelatest version of the DSM, (IV).Q. What does this suggest?Though efforts have been made tobring the two classification systemsinto line to increase reliabilityNilsson et al (2000)found only a60% concordance rate betweenpractitioners using just the ICDclassification system, suggestingthat the DSM IV is more reliable.

I wonder what

the other blokethinks?


11/38

Contradictory evidence?

Though these findings were contradicted byJakobsen et al (2005). When testing thereliability of the ICD-10 classification

system in Denmark, a concordance rate of98% was attained.

Q. What does this demonstrate?


12/38

Alternative Tools

Though attempts have been made to bring thesediagnostic tools in line with each other; thedifferences are still causing problems this led

to the development of several other diagnostictools e.g. St Louis Criteria and PSE.

This means that different clinicians using thesame criteria arrive at the same diagnosis.


13/38

Excellent news!?

Not really. The problem is that as differentcriteria are still being used to diagnose Sz, itis difficult to research studies.

In studies of treatment outcome e.g. it isdifficult to compare data based on individuals

who have been diagnosed with Sz based ondifferent criteria.


14/38

Excellent news!?

It also highlights the difficulties clinicianshave when deciding exactly what they mean by

the diagnosis of Sz.

If the categories are poorly designed andarbitrary, consistent (reliable) diagnosis is

likely to be low.


15/38

Further.

As all criteria are fairly arbitrary, oftenmodified and superceded even the updatedversions are likely to be low in validity even if

more reliable than before.


16/38

As a result..

Seto (2004) reports that in Japan, the term Szhas been replaced by ntegration disorder due

to the difficulty of attaining reliable

diagnosis.

Q. What does this suggest?


17/38

Totally Useless?

Kendell & Jablensky (2007) In response to theargument that Sz should be abolished as aconcept because it is scientifically

meaningless, state that diagnostic categoriesare justifiable concepts, as they provide a usefulframework for organising and explaining thecomplexity of clinical experience, allowing usto derive inferences about outcome and to guidedecisions about treatment.


18/38

Inter-rater reliability dopsychiatrists agree? Even with physical

medical disorders,diagnosis are not alwaysreliable and makingreliable diagnosis of Sz iseven more problematic as a

practitioner has no

physical signs onlysymptoms (what thepatient reports) to basetheir diagnosis on.

I really hope Iagree with thatother bloke!


19/38

Inter-rater reliability dopsychiatrists agree? A true diagnosis cannot be

made until a patient isclinically interviewed.

Psychiatrists are relying onretrospective data, given by aperson whose ability to recallmuch relevant information is

unpredictable. Q. Why? Some may be exaggerating

the truth or blatantlylying!

I really hope Iagree with

that otherbloke!


20/38

Reliability of DSM and ICD

It was originally hoped that the use ofdiagnostic tools could provide a standardisedmethod of recognising mental disorders.

However clear the diagnostic tool, thebehaviour of an individual is always open tosome interpretation. The process is subjective.

The most famous study testing thesubjectivity, reliability and validity ofdiagnostic tools was Rosenhan et al (1972).


21/38

On Being Sane in Insane Places

Rosenhan recruited 8 people (he worked with them orknew him in some capacity).

Each of the 8 people went to a psychiatric hospital

and reported only 1 symptom. That a voice saidonly single words, like thud, empty or hollow. When admitted, they began to act normally. All

were diagnosed with suffering from schizophrenia

(apart from 1). The individuals stayed in the institutions forbetween 7 to 52 days.


22/38

On being sanefollow up

Rosenhan told the institutions about his results, andwarned the hospital that they could expect other individualsto try & get themselves admitted.

41 patients were suspected of being fakes, and 19 of these

individuals had been diagnosed by 2 members of staff. In fact, Rosenhan send no-one at all! All were genuine

patients, suggesting the reliability of SZ diagnosis to bepoor.A good film to watch: One Flew Over the Cuckoos Nest (isJack Nicholsons character mentally ill? Is he mad, bad or sad?You decide!


23/38

Evaluation

- The DSM classification system is often regarded as morereliable than ICD due to degree of specificity in thesymptoms outlined for each category. However, evidencesuggests that both are unreliable tools.

+ Even if reliability of diagnosis based on

classification systems is poor, they do at least allowpractitioners to have a common language,permitting communication of research ideas andfindings, which may ultimately lead to a better

understanding of the disorder, and effectivetreatments.

+ Evidence generally suggests that the reliability of diagnosishas improved as classification systems have been updated.


24/38

Now you have an idea of just how unreliable thediagnosis is how can we treat that which we do notunderstand?

Take a second to think about those being treated withmedication for Sz. Q. how can we be sure they havebeen correctly diagnosed?

What about those being denied treatment that they needin order to maintain some Kind of normality in theirlives?

Think..!!


25/38

Soif we cannot agree on what Sz is =point e.g.inconsistent diagnosis

How do we treat it = issue e.g. labelling/social stigma

Validity of Diagnosis


26/38

What psychiatrists dont understand

It is tempting to label a person as asufferer of schizophrenia, without reallyknowing the extent to which they aresuffering.

The beliefs and biases of some mightmean the unnecessary labelling ofmillions of people as sufferers of a mentaldisorder which could then become a self-fulfilling prophecy(Scheff, 1966).

Sometimes a disorder must reach aparticular level of severity before it can berecognised with confidence as a mentalhealth issue. It is vital we get this right aslabels tend to stick


27/38

Validity of DiagnosisFor diagnosis to be valid..

Aetiological Validity: all patients diagnosed as Sz should havethe samecause for their disorder.Descriptive Validity:patients diagnosed with differentdisorders should actually differ from each other. Reduced bycomorbidity (two or more disorders simultaneously)suggesting they are not discreet.Predictive Validity: it should lead to successful treatment.


28/38

So..

Does the system of classification and diagnosisreflect the true nature of the problems the patient issuffering; the prognosis (the course that thedisorder is expected to take); and how great apositive effect the proposed treatment will actuallyhave.

Many individuals do not neatly fit into categoriesthat have been created. Instead of acknowledgingthis, clinicians tend to diagnose 2 separatedisorders.


29/38

Aetiological ValidityHeather (1976):Argues that only very few causes of mentaldisorders are known and there is a 50% chance of predictingwhat treatment a patient will receive based on diagnosis,suggesting that diagnosis of Sz has low validity.Indeed, in this course we will be covering two causes:Biological and Psychological.Bentall (2003): says the diagnosis of Sz tells us nothingabout the cause of the disorder, impying diagnosis to betherefore invalid.


30/38

Cultural Relativism

Davison & Neale (1994) explain that inAsian cultures, a person experiencingsome emotional turmoil is praised &rewarded if they show no expression of

their emotions. In certain Arabic cultures however, the

outpouring of public emotion isunderstood and often encouraged.

Without this knowledge, an individualdisplaying overt emotional behaviourmay be regarded as abnormal, when itfact it is not.


31/38

Cultural Bias?

Whaley (2004): believes that cultural bias is themain reason that the incidence of Sz is greateramong black Americans than white Americans,as ethnic differences in symptom expression are

overlooked or misinterpreted by practitioners. Thissuggests a lack of validity in diagnosing Szcross-culturally

Cochrane (1977):Afro-Caribbean people living inUK 7 X more likely to be diagnosed with Sz.

Q. why?


32/38

Language difficulties

The clinician might not speak the same languageas the person they are attempting to diagnose.

Certain things can be lost in translation

Due to misinterpretation, this could lead toinappropriate treatment or no treatment at all.


33/38

Predictive ValidityThere is a marked variability in response to treatment (andsymptoms) which has led to the development of subtypes of Sz.However, subtypes have been questioned in terms of validitye.g. how many when diagnosed with undifferentiated Sz will

be recategorised later when new symptoms appear?

Kraeplin (1998): Saw SZ as a chronic deteriorating conditionin all cases. This is not true, with many outcomes possible,

from complete recovery to chronic suffering, again suggestingdiagnosis to be low in validity.


34/38

Schneider (1959) Proposed a different approach to the diagnosis ofSz. He argued that the nature of the symptom that

would determine whether a person was

schizophrenic. He arrived at a number of first rank symptoms,

these included thought insertion and thoughtbroadcast, hearing voices and delusional

perceptions. This approach as been criticised as too stringent

and did littleto improve the predictive validity ofSz.


35/38

Descriptive ValidityAllardyce et al (2006): report that the symptoms used tocharacterize Sz do not define a specific syndrome rather, anumber of different combinations and permutations of thedefining symptoms are possible, suggesting that Sz is not a

separate disorder and that therefore diagnosis of the disorder isinvalid.In clinical practice it is often difficult to determine the

boundaries between Sz and other disorders e.g. mood disorder,personality disorders and other developmental disorders suchas Autism.


36/38

Descriptive ValidityIt is sometimes possible to use additional tests to make thedistinction; however, this is extremely difficult as e.g.depression is comorbid with Sz.

The ICD and DSM have tried to address the problem ofsymptom overlap by proposing mixed disorder categorys e.g.schizo-affective disorder but the validity has been questioned!


37/38

Labelling Someone who has suffered a

mental disorder has to disclose thatinformation in situations such asjob interviews, or they could faceformal action.

Unlike influenza, the label ofschizophrenic stay with a person it sticks!

Schizophrenics risk carrying the

stigma of their condition for therest of their lives.

YET such diagnosis maybe madewith very little evidence of validity

in terms of the conditioningexisting as a separate one.


38/38

How to revise this topic:

DSM IV written by APA last published in 1994. ICD 10 written by WHO. Reliability Beck (1961) 54% agreement Rosenhan study subjectivity

Issues with severity unnecessary labelling. Validity ps dont fit into categories Labelling/Stigma Cultural relativism Davison & Neale (1994)

Schneider (1959) 1st rank symptoms (toostringent).

Other things can produce schizophrenic-likesymptoms.

issues surrounding reliability and validity-1

Documents