7.dr. alan innovations in the measurement of post-acute care outcomes

Alan M. Jette, PT, PhD Health & Disability Research Institute

BU School of Public Health

Describe the measurement dilemma in Post-Acute Care (PAC)

Illustrate how Classical Test Theory methods have difficulty solving this dilemma

Introduce Contemporary Measurement Methods and illustrate how they can solve the measurement dilemma in PAC

The Centers for Medicare and Medicaid Services (CMS) has acknowledged that steps need to be taken toward developing a PAC system in the United States that provides payment and assures quality for an overall episode of PAC, rather than for each individual component.

As an essential step toward accomplishing this policy objective, assessment methods are needed to collect and compare relevant outcome and quality of care data across various sites where PAC is provided.

PAC treatment regimens and costs vary considerably between sites of service.

Little is known about which care setting provides the most

effective and cost/effective PAC due to difficulties in measuring functional outcomes across PAC settings using the same outcome metric.

►Administration:

A set of items (scale) is presented to participants

Responses to each item are assigned a number (item scores)

Item scores manipulated (usually totaled) to get a score for the measure

Focus is on “true score”

Proportion of correct responses or summated score –

Item-total correlations

Scale reliability (internal consistency)

Standard error of measurement

Measurement as a problem of search. CTT is linear in its approach

Suppose our subject is 73 on a 1 to a 100 scale…

In a CTT measure, each item in the measure is assessed, and a total score is calculated to determine where a person is located on the scale.

0 100 73

Classic Testing Theory

Found!

PAC outcome instruments are setting specific: FIM; MDS;OASIS;SF-36

Instruments differ in: # of items,

different response options, different time periods, or different item content; do not communicate with each other.

Need many items or many instruments to cover all the relevant functional outcomes across a broad range of PAC patients

The traditional administration of all

items to each person is burdensome & costly

1

3

1

3

5

7

Not Enough

Marks –

Too Noisy Too

Low - Ceiling

Effect

Example: FIM Example: PF-10

PF-10

Vigorous activities

Moderate activities

Moderate activities

Trouble bending, stooping

Need help to bathe

Cannot maintain balance

Move about with help

Stand up with help

ADL

FIM

PF-10

Source: Andres PL, Black-Schaffer R, Ni P, Haley SM. Computer adaptive testing: a strategy for monitoring stroke rehabilitation across settings. Topics in Stroke Rehabilitation ; 11(2); 33-39, 2004

Ideal Ruler

1

2

3

4

5

6

7

Self Care Eating Oral hygiene Toileting hygiene Shower/bathe self Upper body dressing Lower body dressing Putting on/taking off footwear Mobility Roll left & right Sit to lying Lying to sitting on side of the bed Sit to stand Chair/bed-to-chair transfer

Toilet transfer Car transfer Walk 10 feet Walk 50 feet Walk 150 feet Walk 10 ft on uneven surface 1 Step (curb) 4 steps 12 steps Picking up an object Wheel 50 ft. with 2 turns Wheel 150 ft.

Item difficulty with the hardest item (converse/Noisy)at the top and the easiest item (Converse/family)at the bottom.

Vertical spacing between the items on the hierarchy shows the approximate placement of the item on Communication (the trait) and corresponds to the

item difficulty estimate

The metric for this trait is along the bottom of the graphic and ranges from −7 to 7; the figure shows the relative placement of each item’s response levels on the ruler.

Each item is shown along the ruler where a “1” response for “unable” to “2” “a lot of difficulty” to “3” for “a little difficulty” and a “4” for “no difficulty”).

The expected responses for each item are denoted by their category value (eg., a value of “2” for a lot of difficulty), and the threshold between two levels of response is in the middle between the two responses usually represented by a colon (:). This threshold represents the 50 percent chance that a patient with a given ability has of responding in either a higher or lower category.

Figure 9. Mobility (N=2,980) Keyform (Admissions)

-4 -3 -2 -1 0 1 2 3 4

|------+------+------+------+------+------+------+------| NUM ITEM

1 1 : 2 : 3 : 4 : 5 : 6 6 66 III_F10_funcmob12steps

| |

1 1 : 2 : 3 : 4 : 5 : 6 6 67 III_F11_funcmobupdownramp

1 1 : 2 : 3 : 4 : 5 : 6 6 62 III_F6_funcmobpickup

| |

| |

1 1 : 2 : 3 : 4 : 5 : 6 6 64 III_F8_funcmobwalkuneven

| |


| |

| |

1 1 : 2 : 3 : 4 : 5 : 6 6 68 III_F12a_funcmobwalk500

| |

1 1 : 2 : 3 : 4 : 5 : 6 6 63 III_F7_funcmobwalk50turn

| |

1 1 : 2 : 3 : 4 : 5 : 6 6 59 III_F3_funcmoblie2sit

1 1 : 2 : 3 : 4 : 5 : 6 6 72 III_F13a_funcmobwheel500

1 1 : 2 : 3 : 4 : 5 : 6 6 60 III_F4_funcmobsit2stand

1 1 : 2 : 3 : 4 : 5 : 6 6 75 III_F13d_funcmobwheelroom

1 1 : 2 : 3 : 4 : 5 : 6 6 71 III_F12d_funcmobwalkroom

1 1 : 2 : 3 : 4 : 5 : 6 6 70 III_F12c_funcmobwalk50

1 1 : 2 : 3 : 4 : 5 : 6 6 69 III_F12b_funcmobwalk150

1 1 : 2 : 3 : 4 : 5 : 6 6 61 III_F5_funcmobchairtrans

1 1 : 2 : 3 : 4 : 5 : 6 6 57 III_F1_funcmobsitlie

1 1 : 2 : 3 : 4 : 5 : 6 6 58 III_F2_funcmobrolllr

1 1 : 2 : 3 : 4 : 5 : 6 6 74 III_F13c_funcmobwheel50

| |

| |

1 1 : 2 : 3 : 4 : 5 : 6 6 73 III_F13b_funcmobwheel150

|------+------+------+------+------+------+------+------| NUM ITEM

-4 -3 -2 -1 0 1 2 3 4

1

1 2 0

21 233244545567776445635935626168

1 3 1 1 321346879798554857173224955669233176958334 PERSON

T S M

0 10 20 30 40 50 60 99 PERCENTILE

A patient at ‘1’ on the ruler aligns perfectly with the “6” category response for the easiest item, (wheelchair mobility) , and therefore that person has a very high likelihood of responding (no difficulty) in wheelchair mobility


-4 -3 -2 -1 0 1 2 3 4

|------+------+------+------+------+------+------+------| NUM ITEM


| |



| |

| |


| |


| |

| |


| |


| |












| |

| |


|------+------+------+------+------+------+------+------| NUM ITEM

-4 -3 -2 -1 0 1 2 3 4

1

1 2 0

21 233244545567776445635935626168

1 3 1 1 321346879798554857173224955669233176958334 PERSON

T S M

0 10 20 30 40 50 60 99 PERCENTILE

The person distribution of ‘Mobility’ constitutes the three rows above and next to the PERSON label. Each column creating the person distribution represents the number of persons at a given point on the ruler.


-4 -3 -2 -1 0 1 2 3 4

|------+------+------+------+------+------+------+------| NUM ITEM


| |



| |

| |


| |


| |

| |


| |


| |












| |

| |


|------+------+------+------+------+------+------+------| NUM ITEM

-4 -3 -2 -1 0 1 2 3 4

1

1 2 0

21 233244545567776445635935626168

1 3 1 1 321346879798554857173224955669233176958334 PERSON

T S M

0 10 20 30 40 50 60 99 PERCENTILE

The letters underneath the person distribution indicate the placement of the average person ability estimate (M), and one and two times the standard deviation around that estimate (S and T respectively).

Item Response Theory (IRT) Methodology & Computer Adaptive Testing….

Ideal Ruler

1

2

3

4

5

6

7

Patient

scores

here

Match Questions to the Respondent

=

a probability theory of measurement stating that the correct answer to an

item (question) is a function of an underlying ability of

the person and the characteristics of that item.

http://psychologydictionary.org/probability-theory/

http://psychologydictionary.org/function/

http://psychologydictionary.org/ability/

Large pools of functional status items are selected/written to cover the entire continuum of an outcome domain. Items in the pool provide different information about the outcome across the continuum. Items from other instruments can be included Scores are calibrated to an equal interval score. Reliability & precision are estimated across the continuum

Physical Functioning Item Bank

Item

1

Item

2

Item

3

Item

4

Item

5

Item

6

Item

7

Item

8

Item

9

Item

n

100500

Are you able to get in and out of bed?

Are you able to stand without losing your balance for 1 minute?

Are you able to walk from one room to another?

Are you able to walk a block on flat ground?

Are you able to run or jog for two miles?

Are you able to run five miles?

Physical Function


Item

1

Item

2

Item

3

Item

4

Item

5

Item

6

Item

7

Item

8

Item

9

Item

n

100500








Item

1

Item

2

Item

3

Item

4

Item

5

Item

6

Item

7

Item

8

Item

9

Item

n

100500








Item

1

Item

2

Item

3

Item

4

Item

5

Item

6

Item

7

Item

8

Item

9

Item

n

100500








Item

1

Item

2

Item

3

Item

4

Item

5

Item

6

Item

7

Item

8

Item

9

Item

n

100500








Item

1

Item

2

Item

3

Item

4

Item

5

Item

6

Item

7

Item

8

Item

9

Item

n

100500







A tailored, individualized measure which involves

selecting a set of test items from a calibrated item pool for each patient that best

measures the ‘ability’ of that person

Integrates IRT with computers to administer a measure

selects questions on the basis

of a patient’s response to previously administered questions

measurement is “adapted” to individual

skips uninformative items to minimize response burden

allows determination of person’s standing on a domain without a loss in measurement precision.

Measurement as a problem of search. IRT/CAT measurement is binary in its approach

Suppose our subject is 73 on a 1 to 100 scale…

IRT with CAT

0 100 73

Found!

Includes three patient-reported outcome scales, or domains.

Basic Mobility Daily Activities Applied Cognitive

34

•3 quantitative PRO scales; each takes 2-3 minutes to complete & score •Assesses assistance or difficulty in functional performance •Developed with Federal Government funding •Developed & validated in a sample of 1041; selected from IRF, SNF, LTCH, HHA, outpatient services •Available in over 20 languages Jette, Haley, et al: Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings. PHYSICAL THERAPY 87(4): 385-398,2007 Haley,Jette, et al. Replenishing a Computerized Adaptive Test (CAT) of Patient Reported Outcomes. Quality of Life Research, 18(4) 461-471, 2009.

35

Activity Measure for Post-acute

Care (AM-PAC) 269 items

Basic Mobility 131 items

Applied Cognition 50 items

Daily Activities 88 items

Ambulation

Bend, Carry, Lift

Locomotion with Device

Meal Preparation

Grooming/Dressing

Feeding Communication

Print Information

Complex Instructions

Transfer

Provides a quantitative score

Can compare functioning within an individual (admission to discharge) and across individuals

Can compare across an episode of care: transitions from inpatient – to outpatient – to home settings

Carefully selected items from the AM-PAC calibrated item banks. Need to assess different kinds of activities in

inpatient and outpatient settings ▪ Developed separate short forms for each setting

Inpatient and outpatient short form items include items calibrated on the same metric ▪ Scores from inpatient and outpatient short forms assess

patients’ function as they move from inpatient to outpatient settings.

-3

-2

-1

0

1

2

3Daily Activity Short Form Item Difficulty N = 88

Selected 6 Items for Inpatient; 15 Items for Outpatient

Inpatient

Outpatient

Table 5a. CARE-C and CARE-F Analytic Subscale Information: Patient Self-Report

Subscale Item Content Variable Name

CARE-C

Basic Mobility How much difficulty do you currently have…

Moving from sitting at the side of the bed to

lying down on your back?

II_E1a_patmobbed2back

Moving up in bed? II_E1b_patmobupbed

Standing for at least one minute? II_E1c_patmobstand1min

Sitting down in an armless straight chair? II_E1d_patmobsitchair

Standing up from an armless straight chair? II_E1e_patmobstandchair

Getting into and out of a car/taxi? II_E1f_patmobcar

Walking around on one floor, taking into

consideration thresholds, doors, furniture,

and a variety of floor coverings?

II_E1g_patmobwalkonefloor

Going up and down a flight of stairs inside,

using a handrail?

II_E1h_patmobstairs

How much difficulty do you currently have…

Bending over from a standing position to

pick up a piece of clothing from the floor

without holding onto anything?

II_E1i_patmobbend

Walking several blocks? II_E1j_patmobwalkblocks

Walking up and down steep unpaved

inclines?

II_E1k_patmobincline

Carrying something in both arms while

climbing a flight of stairs?

II_E1l_patmobcarrystairs

How much help from another person do you

currently need moving to and from a bed to

a chair?

II_E1m_patmobbed2chair

Figure 1. Basic Mobility (N=2,277) Keyform (Admissions)

-6 -4 -2 0 2 4 6 8

|-------+-------+-------+-------+-------+-------+-------| NUM ITEM

0 0 : 1 : 2 : 3 3 12 II_E1l_patmobcarrystairs

| |

| |

0 0 : 1 : 2 : 3 3 11 II_E1k_patmobincline

0 0 : 1 : 2 : 3 3 10 II_E1j_patmobwalkblocks

| |

| |

0 0 : 1 : 2 : 3 3 8 II_E1h_patmobstairs

0 0 : 1 : 2 : 3 3 9 II_E1i_patmobbend

| |

0 0 : 1 : 2 : 3 3 5 II_E1e_patmobstandchair

0 0 : 1 : 2 : 3 3 6 II_E1f_patmobcar

| |

0 0 : 1 : 2 : 3 3 2 II_E1b_patmobupbed

0 0 : 1 : 2 : 3 3 7 II_E1g_patmobwalkonefloor

0 0 : 1 : 2 : 3 3 1 II_E1a_patmobbed2back

0 0 : 1 : 2 : 3 3 4 II_E1d_patmobsitchair

0 0 : 1 : 2 : 3 3 3 II_E1c_patmobstand1min

| |

| |

00 : 1 : 2 : 3 3 13 II_E1m_patmobbed2chair

|-------+-------+-------+-------+-------+-------+-------| NUM ITEM

-6 -4 -2 0 2 4 6 8

1 1 11111111

1 1 3334057480821202537767 51213 11 4

3 21 41416608670497497789687003304465720860234711 25 PERSON

T S M S T

0 10 20 40 50 70 80 90 99 PERCENTILE

A typical CAT takes 2-3 minutes to administer Avoids unnecessary items Adequate reliability and sensitivity Ability to cover wide content scope Score reports can be developed to assist in

interpretation

SCI-FI/AT raw scores are converted to standardized z-scores normed on the full item bank which are then transformed to ‘T’-scores such that the mean is 50 and the standard deviation is 10.

AM-PAC Short Forms and CATs are all on the same common ‘T’ scale to facilitate functional assessment across PAC settings

Requires IT platforms unavailable in some rehabilitation settings or use of short forms

Legacy instruments are required by regulation Discomfort in asking different items over time and across

patients Create vehicles for clinician training & support Create an system allowing access to item banks and CATs

IRT and CAT technologies could

transform outcome assessment in PAC Contemporary outcomes measures like

AM-PAC have great potential for overcoming the measurement dilemma in PAC

Challenges to adopting measurement

innovations like IRT/CAT need to be overcome.

Jette A, Haley S: Contemporary measurement techniques for rehabilitation outcome assessment. Journal of Rehabilitation Medicine. 37(6):339-345, 2005.

Haley, S.M., Coster, W.J., Andres, P.L., Ludlow, L.H., Bond, T., Sinclair, S.J., & Jette, A.M. (2004). Activity Outcome Measurement for Post-acute Care. Medical Care, 42(1 Suppl), I-49-I-69.

Jette, A.M., Haley, S., Tao, W., Ni, P., Meyers, D., & Zurek, M. (2007). Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings. Physical Therapy, 87(4), 385-98.

Jette, A.M., Norweg, A., & Haley, S. (2008). Achieving meaningful measurements of ICF concepts. Disability and Rehabilitation, 30(12-13), 963-9.

Jette, A.M., Haley, S.M., Ni, P., & Moed, R. (2008). Adaptive short forms for outpatient rehabilitation outcome assessment. American Journal of Physical Medicine and Rehabilitation, 87(10), 842-52.

Jette, D.U., Stilphen, M., Ranganathan, V.K., Passek, S.D., Frost, F.S., & Jette, A.M. (2014). AM-PAC “6-Clicks” functional assessment scores predict acute care hospital discharge destination. Physical Therapy, 94(9), 1252-61.

•Jette, A.M., Haley, S., Tao, W., Ni, P., Meyers, D., & Zurek, M. (2007). Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings. Physical Therapy, 87(4), 385-98.

•Jette, A.M., Haley, S., Tao, W., Ni, P., Meyers, D., & Zurek, M. (2007). Prospective evaluation of the AM-PAC-CAT in outpatient rehabilitation settings. Physical Therapy, 87(4), 385-98.

7.dr. alan innovations in the measurement of post-acute care outcomes

Healthcare