COMMUNITY HEALTH STUDIES VOLUME X , NUMBER 2, 1986

MEASURING OUTCOMES OF HEALTH SERVICES: A REVIEW OF SOME AVAILABLE MEASURES

Jane Hall * and Gregory Masterst

* Department of Community Medicine, Westmead Hospital 7 Departmen: of Community Medicine, Royal Prince Al fed Hospital

Abstract There are few comparative reviews to guide

the choice of a health status measure. The appropriate measure in any study depends on the health outcome being considered. Outside of the context of a study the only criteria on which an instrument can be judged as ‘good’ or ‘poor’ are validity and reliability. Other characteristics which should be considered in choosing a measure of health status are: purpose; conceptual focus; operat ional approach; sensitivity; utility weighting; a n d amenability t o quant i ta t ive manipulation.

This paper compares and contrasts five general measures of health status so as to enable researchers to choose among them: the measures are the Sickness Impact Profile, the Rand HIS measures, the Nottingham Health Profile, the Duke-UNC Profile and the Index of Well-being.

Introduction Heal th care evaluat ion is increasingly

required to assess health services which aim to improve the quality of life rather than extend its quantity. The need to measure “health” in a way which goes beyond mortality is essential but difficult. Recent work in the field of health status measurement has been directed to the development of measures which apply across age, sex and disease groups. Such instruments attempt to encompass physical, mental and, to some extent, social aspects of health. In spite of the extensive literature, there are few comparisons of measures.’ This review provides a guide to the selection of the most appropriate measures.

“Health” has a myriad of definitions that can vary according to the wider context and the focus of the study. Therefore the appropriateness of a health status measure depends on the health program being evaluated and those aspects of health it is expected to influence. Using a n instrument which does not focus on those aspects of health relevant to the particular study is like

HALL & MASTERS 147

using scales to measure a person’s height. No matter how good the instrument, it is not ‘right’ in the context in which it is being used. In addition to focus, there are important considerations which can guide the researcher in making a choice. These include validity (that is, the extent to which the instrument really measures health), and reliability (that is, how well it minimises error in repeated measurements). Instruments may be categorised as “good” or “poor” depending on the extent to which validity and reliability have been demonstrated. In choosing a measure the first step is selecting one with an appropriate focus and the second step is to investigate evidence for the validity and reliability of that measurement focus. These two steps are discussed below and lead to a third step based on additional characteristics of these measures - as defined by the authors.

Focus The choice of health status measure must be

begun by identifying the specific aspects of health that an instrument is required to encompass. The conceptual focus of the available instruments can then be compared with the focus required. The focus of an instrument is not always indicated by its title and in some cases names are quite misleading. The Index of Well-Being discussed here does not cover general well-being but rather the departure from normal physical functioning.2 Another example is the General Heal th Questionnaire which is, in fact, a screen to detect non-psychotic phychiatric distrubance.3

The authors’ own checklist for considering focus comprises ability/ performance, mental health, behavioursIfeeling-states, and symptoms.

There is a distinction between ability and actual performance. For example, an individual may be capable of dressing independently but refuse to d o so at home where there are sympathetic others to assist. It is essential to be quite clear about which aspect, ability or performance, is to be measured and then to make

COMMUNITY HEALTH STUDIES

sure that the instrument will measure that. A second point t o consider is whether the

intended measure should encompass behaviours or feeling-states or both. Mental health encompasses many aspects and it is necessary to define which are t o be measured if this is included. Similarly it is necessary t o decide whether the measure is sensitive t o t ransient dis turbances a n d / o r underlying chronic disturbance. Finally the relevance of including signs and symptoms must be determined. Few of the measures reviewed here include symptom status.

Validity The validity of a health status measure is the

extent to which it does actually measure health. Thus it is the key characteristic of any instrument. Health is a multi-dimensional concept which is difficult to define precisely; therefore there is no one accepted measure of health to serve as a gold standard against which other instruments can be assessed. This difficulty with criterion validation has led to the development of a number of approaches to testing for validity.

Content validity is the extent to which the items of an instrument cover a representative range of the construct t o be measured. The domain of the construct should be defined in advance, not after the instrument has been developed.* Content validity should be distinguished clearly from face validity, which refers t o whether a n instrument looks valid; the latter is not a sufficient test of validity but is a useful first line of defence against selecting an inappropriate instrument.

In construct validity two types of evidence are sought. Convergent validity is suggested when the proposed measure (or part of it) is highly correlated with another measure of the same construct. Discriminant validity is suggested when the new measure correlates less well with other measures from which it was intended to differ.2,5,6

Reliability Low reliability in a health status instrument

means that scores may not represent accurately changes in health but merely random fluctuations. Reliability is a critical aspect of a health status measure yet it has been little tested or reported.’ Three techniques for testing reliability are test/ re- test analysis, alternate forms analysis and internal consistency analysis.

Test / re-test analysis involves administering the same instrument to the same people on two different occasions. The agreement between the two scores represents the reliability of the instrument. However, any change in scores may

HALL & MASTERS 148

relate to changes in the underlying attribute rather than poor instrument reliability. The length of time between tests must be short enough to expect the underlying attribute to remain stable.

In alternate forms reliability testing, each person is required to complete alternative forms of the instrument t o be tested. The problem with this type of reliability testing is, ensuring that the two forms of the instrument are in fact measuring exactly the same thing.

Inte.rna1 consistency differs in that the instrument is only administered once and involves an analysis of the extent t o which parts of the instrument conform to the expected relationships among the items. For example, in split-half reliability testing the instrument is divided into two comparable halves and the scores for each half compared.

Other Characteristics Other characteristics can be used with validity

and reliability to guide the “consumer”in choosing from the array of health status measures available.

The purpose of the instrument should be considered. Measures designed for a different purpose than the measurement of health outcomes may not be appropriate.

How the data are collected can be described as the operational approach. For example, is the instrument designed to be self-administered or interviewer administered?

The sensitivity of the instrument refers to its ability to register the changes that are expected to occur. For example, it is inappropriate t o choose a n instrument which can only measure crude changes in activities of daily living when the program being evaluated is attempting to restore fine eye-hand co-ordination.

The way in which the various items are combined and weighted is the utility weighting of the instrument. In many instruments the notion of utility weighting has not been explicitly addressed. The utility weighting has an important bearing on the quantitative manipulation that is appropriate. Does the instrument produce a single score and what statistical methods are valid?

The Five Measures The Sickness Impact Profile (SIP), the Rand

Health Insurance Study measures (Rand), the Nottingham Health Profile (NHP), the Duke- UNC Profile (DUHP) and the Index of Well-Being (IWB) each represent a notable contribution t o the literature of the last decade-and-a half.*-12 These

COMMUNITY HEAL TH STUDIES

five measures are btoader than the traditional focus on mortality and symptoms and they are not disease specific. They contain both physical and mental constructs of health. By being sensitive to low levels of morbidity, they aim to discriminate in ostensibly healthy populations. In addition they satisfy the requirement that their validity and reliability have been investigated.

To some extent, the five instruments have the same starting point. They build on, in some cases explicitly, the WHO definition of health as not merely the absence of disease but physical, mental and social well-being. In spite of this similarity, the instruments differ in their practical application of this definition.

The Sickness Impact Profile The SIP was developed specifically for use as

an outcome measure in health services evaluation.* While the S I P focuses exclusively on the sickness/ dysfunction end of the health spectrum it attempts to be sensitive to minimal levels of sickness-related behaviours. T h e S I P is constructed so as t o measure changes in a n individual’s behaviour that are due to illness. The categories, as shown in Table 1, encompass p h y s i c a l a n d p y s c h o - s o c i a l d i m e n s i o n s .

Respondents indicate behaviours that apply to them on that day. It covers low levels of sickness to chronic illness. By taking a behavioural base, it excludes feelings-states.

The instrument has been validated for three types of administration; self-administered with interviewer delivery, interviewer administered, and mail delivered. Reliability was compared across the three administration modes. Test-retest reliability and internal consistency were both found to be high.

The high weighting established in the development of the S I P enables a weight t o be assigned to each item and, accordingly, scores can be simply calculated, by addition, for each of the 12 categories, for the overall SIP and for physical and psychosocial dimensions which are combinations of various categories. The interval scale properties of the instrument mean that the scores can be subjected to sophisticated statistical manipulation.

The SIP is a well-validated and reliable measure which will give a single summary score of health status. It is appropriate in many contexts, b u t m a y be of l i m i t e d s e n s i t i v i t y i n “hea1thy”populations due t o its focus o n dysfunction related to sickness.

TABLE 1

Categories and Dimensions of the SIP

1. Sleep and rest 2. Eating 3. Work 4. Home Management 5. Recreation and pastimes

6. Ambulation 7. Mobility 8. Body Care and Movement

9. Social interaction 10. Alertness Behaviour 11. Emotional Behaviour 12. Communication

HALL & MASTERS 149

Independent categories

Physical dimension

Psychosocial dimension

COMMUNITY HEALTH STUDIES

The Rand HIS Measures The Rand Corporation’s Health Insurance

Study (HIS) is a social experiment involving nearly 8,000 people which was designed to assess the effects of different health insurance schemes on use of services and health status. In order t o d o this the Rand researchers aimed to develop “reliable and valid health status measures that could be used to detect small but important changes in health status of enrollees sampled from general populations”.9 The Rand researchers acknowledge that their battery of measures is not an exhaustive measure of health status and is not appropriate for all research purposes.

The measures include the three dimensions of health identified in the WHO definition and include behaviours and feeling-states. The Rand researchers singled out one aspect of health, namely physiological health, f o r separate measurement and developed a measure of an integrative concept called general heal th perceptions. These measures form a battery as shown in Table 2.

Extensive reliability testing of the HIS measures was carried out. Reliability coefficients were high for the functional status measures and acceptably high for the health perceptions questionnaire. The reliability of the social health measures was not tested in the preliminary work,

Construct val idat ion of t h e measures proceeded by analysing the magnitude and direction of inter-relationships amongst the battery of measures. In general, scales from the same dimension correlated more highly with each other than with scales from the’other dimensions. Factor analyses of the internal correlations confirmed the existence of the three dimensions and a general health dimension.

The development of scoring systems for the scales proceeded after collection of preliminary HIS data. Scoring of the functional status scales is very complex and tedious due to the scaling method used. No attempt has been made to construct a single health status index summarised by a single score. Rather, items have been combined t o construct scales for each of the health

TABLE 2

Categories and Dimensions of the Rand Measures

Physical Health Physiologic Physical state and functioning

Mental Health

Social Health

General health Perceptions

HALL & MASTERS

Functional State

Interpersonal interaction & social participation

Self assessment of health

150

Chronic and acute limitations in activities of D.L.

More strenuous daily activities

Anxiety Depression Positive Well-being Self Control

Family Work or major role Social Community

Current health Future Health Worry about health Resistance to illness Illness in their lives

COMMUNITY HEALTH STUDIES

dimensions. This means that comparison of group results may be ambiguous; for example,’ if mental health measures show improvement while physical health measures show deterioration, does this represent a general gain or loss in health?

The Rand battery was developed as an outcome measure appropriate for general populations. Its validity and reliability have been investigated extensively. It has been constructed so as to be sensitive to changes in health in ostensibly healthy groups. However, it may be difficult to discriminate among “healthy” people on the basis of physical function. It provides a profile rather than a single index.

The Nottingham Health Profile The Nottingham Health Profile (NHP) was

developed for multiple purposes as a measure of self-perceived health problems. 10 Its intended use includes measurement of outcdmes in health services evaluation. Although the rationale for the development of the NHP is similar to the SIP, there is a different conceptual focus between them. The NHP is not restricted to behavioural impacts of illness, it also includes the feeling-state dimension as well as behavioural measures by SIP. However, like the SIP, the NHPfocusses on health problems, departures from the “normal”, and not on positive aspects of wellness. The aspects covered by the NHP are shown in Table 3.

The NHP is a self-administered questionnaire and respondents are simply required to answer “yes” or “no” to each item according to whether the statement applies to them. Preliminary testing of the profile’s test/re-test reliability with a group of osteoarthritis patients was encouraging although evidence of the NHP’s reliability with other groups needs to be gathered.

Validity testing of the N H P has provided good evidence for its convergent validity with respect to physicians’ assessment of function, and self perceived health status. The N H P has also demonstrated an ability to distinguish between groups of people with varying levels of disability. However less evidence of the discriminant ability of the profile has been accumulated. The sensitivity of the instrument can be inferred from some of the results of validity testing. It does not appear to be sensitive enough to be used with ostensibly healthy adults, even in the over 60 years age group. It appears that it should only be used when measuring the self-perceived health status of disabled people.

The NHP has not been investigated as extensively as S I P or HIS but it represents a promising development for use in a limited area.

The Duke- UNC Health Profile The Duke-UNC Health Profne (DUHP) was

developed primarily as an outcome measure in the evaluation of primary care services, but the authors note its applicability as a day to day clinical assessment tool in the primary care setting.” The DUHP also measures functioning in terms of physical, emotional and social activities. This instrument introduces another dimension, that of symptom status (see Table 4). Its authors argue for the inclusion of symptoms as these are often the first manifestation of changing health. Further treatment is influenced by the presence of symptoms and therefore symptom status is important to outcome measurement in the primary care setting.

Good evidence of the reliability- of the symptom status, physical function and emotional function measures exists, but there is less convincing evidence of the reliability of the 5-item social function dimension.

TABLE 3

Sections in Part 1 of the Nottingham Health Profile

Section No. of Items

Energy Pain Emotional Reactions Sleep Social Isolation Physical Mobility

HALL & MASTERS 151 COMMUNITY HEALTH STUDIES

TABLE 4

Categories and Dimensions of the DUHP

Symptom Status

Physical .Function

Emotional Function

Social Function

Presence / absence Psychological digestive other

Capacity to perform Disability days Ambulation Use of upper extremities

Liking and respect for oneself Ability to get along with others

Self Esteem

Ability to perform Self care usual-role

Considerable effort has been devoted to construct validation of the profile. The instrument’s discriminant validity was also investigated. However, the reliability and validity testing were conducted with a fairly young group of people with transient health problems. Further testing is required to establish the reliability and validity with older or more dysfunctional persons.

Scoring of the DUHP is done on a simple linear scale with the response for the least desirable states receiving a score of zero and the most desirable scoring two, thereby giving a three point scale. No weighting method was used to validate this scoring system so the assumption that the items are equally weighted has not been supported. A further problem with the DUHP is its relative insensitivity. Testing of the profile showed that scores are highly skewed towards the favourable end of the four health dimensions. The DUHP has demonstrated validity and reliability in a fairly

Function at work/home Personal interactions Participation Community/social

young group of people with transient health problems. Its major shortcoming is its lack of explicitly derived weights and apparent insensitivity.

The Index of Well-Being Fanshel and Bush conceptualise health as

having two components, function level and prognosis.2 The I WB attempts to measure physical function. 12 This is defined as a person’s ability to carry on usual daily activities. Three mutually exclusive scales have been defined with separate levels as shown in Table 5. Forty three separate levels or steps are further modified by the presence/absence of 36 symptoms/problems. Some confusion exists as to whether the symptom/problem complexes are to provide information on the cause of loss of function. The conceptual focus of the IWB is narrower than that of the other instruments reviewed; functional

TABLE 5

Scales and Dimensions of the IWB

Mobility Scale

Physical Activity Scale

Social Activity Scale

HALL & MASTERS

Ability to travel outside home

Ability to walk

Ability to perform work or major activity

152 COMMUNITY HEALTH STUDIES

status is measured exclusively and there’ is no coverage of mental health status or the social and psychological aspects of health.

The IWB was conceived initially as a self administered instrument but Bush and his colleagues now strongly support using interviewer administered versions only.

Little direct testing of the reliability of the IWB has been undertaken but results of the many studies on the utility weighting of the index provide strong support for its reliability. The IWB authors examined its convergent and discriminant validity by correlating IWB scores with other measures of health status including self-rated well-being, number of chronic condi t ions a n d o ther symptoms, and use of G P services.

Most development work was devoted to deriving weights for each of the forty-three levels. The category scaling method has been shown to produce reliable and valid weights on a 0-1 scale. These are amenable to sophisticated statistical manipulation.

The application of the IWB may be limited by its availability in a n interviewer administered version only. There is some uncertainty as to the purpose of the symptom/problem check list which augments the function levels. This may lead to difficulties in interpretation. Availability

One of the foremost considerations in whether to employ any of the instruments reviewed is its ready availability. The acquisition of t h e instrument per se can be a problem. Copies of the Sickness Impact Profile are available from the authors in Seattle although there can be a time lag of several months in t h e exchange of correspondence. The battery of measures fielded in the Dayton trial of the Rand Corporation’s Health Insurance Study and subsequent revisions are reproduced in the Health Insurance Study (now the Heal th Insurance Experiment) Series.P However, scoring systems are not included in these publications although, as with all Rand material, they are readily available from the Rand Corporation.

HALL & MASTERS 153

It has been more difficult to sbtain the actual instruments of the DUHP, the N H P and the IWB. Acquisition of the IWB requires purchase of a user’s manual and signing of a copyright /conditions of use agreement. The content and conceptual focus of the D U H P and the N H P are clearly discernible from the papers published on their development.

Conclusion Measures of health status that are valid and

reliable, yet go beyond the traditional indicators of mortality and morbidity, are now available. Most notable among the contributions to this rapidly expanding field are the Rand HIS battery and the Sickness Impact Profile. In any study which requires a measure of health s ta tus , the investigator will be able to choose from a number of instruments. There have been few studies comparing different instruments on the same populat ions. A recent s tudy i n Australia comparing the S I P and the Rand instruments in a primary care setting and in a general population suggests that the SIP does not discriminate as well in a healthy group.”

The instruments reviewed in this article were selected on the basis of their broad approach to operationalising ‘health’ and a substantial attempt to assess validity and reliability. The SIP is widely applicable but will measure low level sickness related dysfunction. Similar in its wide applicability is the Rand set of measures but with better discrimination in well populations. The NHP has been shown t o work in elderly populations with disabilities. The DUHC appears applicable to younger age groups but again with definite health problems. The IWB has been tested in general populations; it has wide applicability but is a more complicated instrument to administer. The first Criterion in selecting an instrument to use has to be whether its focus is relevant to those aspects of health which are to be measured. The use of a systematic selection process as outlined in this article will enable researchers to choose the most appropriate measure for their study.

COMMUNITY HEALTH STUDIES

e

m

P

Y

Y

TA

BL

E 6

Com

pari

sons

of

5 H

ealth

Sta

tus M

easu

res

Inst

rum

ent

No.

of

How

R

elia

bilit

y V

alid

ity

Wha

t Se

nsiti

vity

L

evel

of

Eas

e O

f It

ems

Adm

inist

ered

C

onve

rg. D

iscr

im.

Groups

Mov

emen

t Sc

orin

g

***

***

***

SIP

136

Self

Inte

rvie

wer

M

ail

RA

ND

94

**

* **

* **

* Se

lf

* **

N

HP

38

Se

lf

***

***

DU

HP

63

Se

lf In

terv

iew

er

**

**

**

IWB

11

6 In

terv

iew

er

Gen

eral

*+

In

terv

al

**

popu

latio

n to

chr

onic

ally

di

sabl

ed

Gen

eral

**

* O

rdin

al-

popu

latio

n on

ly

inte

rval

(o

ver

15)

*

Dis

able

d on

ly

Ord

inal

**

Gen

eral

pop

ula-

*

Nom

inal

**

tio

n un

der

SO y

ears

Gen

eral

In

terv

al

**

popu

latio

n

***

Goo

d ev

iden

ce

** F

air

evid

ence

Poor e

vide

nce

References

1.

2.

3.

4.

5.

6.

7.

8.

Howell R. Three healthstatus measures and content validity. Unpublished paper. New Zealand Medical Research Council, 1982. Fanshel S, Bush JW. A health-status index and its application t o health-services outcomes. Operations Research 1970; 1'8:

Goldberg DP. The Detection.of Psychiatric Illness by guestionnoire. London. Oxford University Press. 1972. Anastasi A, Psychological Testing. 4th ed. New York: MacMillan, 1976. Campbell DT, Fiske DW. Convergent and discriminant validation by the Multitrait- multimethod matrix. Psychological Bulletin

Shortell S M and Richardspn WC. Health Program Evaluation. St Louis: CV Mosby, 1978. Ware JW, Brook RH, Davies AR, Lohr KN. Choosing measures of health status for individuals in general populations. A m J Public Health 1981; 71:62@624. Bergner M, Bobbitt RA, Carter WB and Gilson BS. The Sickness Impact Profile:

102 1 - 1065.

1959; 56~810-105.

9.

10.

11.

12.

13.

D e v e l o p m e n t a n d F i n a l R e v i s i o n . Med. Care, 1981; 19(8):787-805.

Brook RH, Ware JE, Davies-Avery A et al. Conceptualisation and Measurement of Health for Adults in the Health Insurance Study: Vol VIII, Overview. Santa Monica CA: Rand Corporation. R-1987/8-HEW, 1979. Hunt SM, McEwen J. The development of a subjective health indicator. Sociology of Health and Illness 1980; 24: 281-286. Parkerson GR, Gehlbach SH, Wagner EH et 01. The Duke-UNC Health Profile: an adult health status instrument for primary care. Medical Care 1981; 19:806-828. Kaplan R, Bush JW, and Berry CC. Health Status: types of validity apd the Index of Well-being. HealthServices Research, 1976;

Hall J, Menyhart J, Hall N and Tweedie R. Measuring Health Status in Health Programme Evaluation. Paper presented at ANZSERCH/ APHA Annual Conference, 1984.

1 1: 478-507.

HALL & MASTERS 155 COMMUNITY HEALTH STUDIES