Development of a smartphone application for the objective detection of attentional

deficits in delirium

Authors

Dr. Zoë Tieges1,2* Antaine Stíobhairt1,3*, Katie Scott3, Klaudia Suchorab3, Alexander Weir4, Dr

Stuart Parks4, Dr Susan Shenkin1,2, Professor Alasdair MacLullich1,2.

Affiliations

1Edinburgh Delirium Research Group, University of Edinburgh

2Centre for Cognitive Ageing and Cognitive Epidemiology, University of Edinburgh

3Department of Psychology, University of Edinburgh

4Medical Devices Unit, NHS Greater Glasgow and Clyde

*Z. Tieges and A. Stíobhairt contributed equally to this paper.

Corresponding author

Dr. Zoë Tieges

Edinburgh Delirium Research Group

University of Edinburgh, Room S1642

Royal Infirmary of Edinburgh, 51 Little France Crescent

Edinburgh, EH16 4SA, UK

Tel. +44 (0) 131 242 6482, Fax. +44 (0)131 242 6370, E-Mail Zoe.Tieges@ed.ac.uk

1

ABSTRACT

Background

Delirium is an acute, severe deterioration in mental functioning. Inattention is the core

feature, yet there are few objective methods for assessing attentional deficits in delirium. We

previously developed a novel, graded test for objectively detecting inattention in delirium,

implemented on a computerised device (Edinburgh Delirium Test Box (EDTB)). Although

the EDTB is effective, tests on universally available devices have potential for greater

impact. Here we assessed feasibility and validity of the DelApp, a smartphone application

based on the EDTB.

Methods

This was a preliminary case-control study in hospital inpatients (aged 60-96) with delirium

(N=50), dementia (N=52) or no cognitive impairment (N=54) who performed the DelApp

assessment, which comprises an arousal assessment followed by counting of lights presented

serially. Delirium was assessed using the Confusion Assessment Method and Delirium Rating

Scale-Revised-98 (DRS-R98), and cognition with conventional tests of attention (e.g. digit

span) and the Short Orientation-Memory-Concentration test (OMCT).

Results

DelApp scores (maximum score=10) were lower in delirium (scores(median(IQR)): 6(4-7))

compared to dementia (10(9-10)) and control groups (10(10-10), p-values<0.001). Receiver

Operating Characteristic (ROC) analyses revealed excellent accuracy of the DelApp for

discriminating delirium from dementia (AUC=0.93), and delirium from controls (AUC=0.99,

2

p-values<0.001). DelApp and DRS-R98 severity scores were moderately well correlated

(Kendall's tau= -.60, p<0.001). OMCT scores did not differ between delirium and dementia.

Conclusions

The DelApp test showed good performance, supporting the utility of objectively measuring

attention in delirium assessment. This study provides evidence of the feasibility of using a

smartphone test for attentional assessment in hospital inpatients with possible delirium, with

potential applications in research and clinical practice.

Key words

Delirium; Attention; Objective; Measurement; Neuropsychological; Smartphone; Dementia;

Cognition

Running head

Detecting attentional deficits in delirium

3

INTRODUCTION

Delirium is an acute, severe neuropsychiatric syndrome characterized by fluctuating

disturbances in attention, arousal and cognition. It is highly prevalent in older hospitalized

patients and is associated with adverse outcomes including functional decline, new

institutionalization, persistent cognitive impairments including dementia, and higher

mortality (Siddiqi et al., 2006; Witlox et al., 2010).

The core diagnostic feature of delirium is 'inattention', defined as a 'reduced ability to

direct, focus, sustain and shift attention' in the DSM-5 criteria for delirium (American

Psychiatric Association, 2013). The evidence suggests that several aspects of attention are

likely to be affected in delirium, including the basic orienting response, focusing, sustaining

and dividing attention (Tieges et al., 2014). The extent to which each of these aspects of

attention is affected in delirium is poorly understood, though deficits in sustained attention

(i.e. the ability to maintain attention to stimuli over time) have been implicated (Brown et al.,

2011; O'Keeffe and Gosney, 1997; Tieges et al., 2014).

A range of methods is used in research and in clinical practice to ascertain inattention.

These include subjective assessments based on interview and clinical observation, and

objective assessments using brief neuropsychological tests (Hall et al., 2012). Objective tests

likely offer some advantages over subjective methods, including standardization of

instructions, and greater reliability and reproducibility. In addition, objective tests are less

likely to be operator dependent and reliant upon clinical experience than subjective

assessments. Several neuropsychological tests are currently used to assess inattention in

delirium. These include digit span, spatial span, months of the year and days of the week

4

backwards, vigilance 'A' and serial 7s (O'Regan et al., 2014; Tieges et al., 2014). Generally

these tests perform well in detecting delirium. However, the available data show that most

existing tests discriminate poorly between delirium and other mental disorders including

dementia (Morandi et al., 2012). Indeed, many studies show that patients with dementia show

significant deficits in tests often used to assess inattention in delirium, such as spatial span

(Meagher et al., 2010), serial 7s (Bronnick et al., 2007) and months of the year backwards

(Katzman et al., 1983). Further, existing tests of attention have not been validated for severity

grading in delirium, which is useful in monitoring delirium progress over time and providing

a more fine-grained measure of delirium than dichotomous scoring methods.

To help address the lack of robust, objective assessments for inattention in delirium,

we previously developed a novel attentional test implemented on a custom-built

computerized device entitled the Edinburgh Delirium Test Box (EDTB) (Brown et al., 2011;

Tieges et al., 2013). The EDTB tasks require participants to count and verbally report how

many times either one or two slowly presented lights on the device illuminate. The lights are

illuminated for 1 sec, and the inter-trial intervals are 1-4 sec. The design of this test was

motivated by the need for a task that was cognitively simple whilst placing demands on

attentional functioning. Prior research had found that the ability to sustain attention in a

simple counting task may be relatively spared in Alzheimer’s disease (Lines et al., 1991;

Morandi et al., 2012; Perry et al., 2000). The EDTB differs from these prior tasks in that the

stimuli are visual instead of auditory, and distracting stimuli are presented in some of the

trials in order to increase task difficulty. In our work with the EDTB we found that patients

with delirium showed marked deficits on these tasks, whereas patients with dementia and

cognitively unimpaired controls performed at or near ceiling level. The receiver operating

characteristic areas under the curves were between 0.80 and 1.00, indicating good to excellent

5

accuracy of the EDTB in discriminating delirium from dementia and control groups (Brown

et al., 2011). We concluded therefore that these new objective attentional tests offered

potential utility in detecting delirium and differentiating it from dementia. More broadly,

validated, objective neuropsychological assessments potentially offer a more robust approach

to the measurement of inattention in delirium than presently available methods.

We recently developed a software application for detecting deficits of visual sustained

attention in delirium ('DelApp'), which is based on our EDTB sustained attention tasks. The

rationale is that although the EDTB showed good performance, tests on devices that are

universally available, such as smartphones, are more readily applicable in research and

clinical settings. Smartphone-based applications are increasingly being used as a method for

assessing cognitive function in older individuals (Brouillette et al., 2013).

In this preliminary study, we first conducted a feasibility study to assess acceptability

of the DelApp for the assessment of attention in older hospitalised patients. We then

proceeded with a pilot case-control study to compare performance on the DelApp in a

selected sample of older hospitalised patients with dementia, delirium or no cognitive

impairment.

6

METHODS

Study 1

Design

This study was designed to assess the feasibility and acceptability of the DelApp in

hospitalised older patients, and to compare the EDTB and DelApp. Formal diagnostic

assessment of each patient was not performed as this was not required to meet the aims of

Study 1. A within-subjects design was used. Patients were studied on a single occasion and

underwent EDTB and DelApp tests. Semi-structured interviews with patients were conducted

to assess their opinions of these tests. Study 1 and Study 2 were approved by the Scotland A

Research Ethics Committee.

Participants

Patients aged 60 and older who were able to communicate in English were eligible.

All patients had to demonstrate capacity to provide written, informed consent; this was

because the ethics committee ruled that for Study 1 it was unnecessary to involve patients

requiring proxy consent. Exclusion criteria were: severe sensory impairments or severe acute

illness that would impede testing and interview, and where clinical staff considered that

participation would adversely affect the patient’s care.

The researchers (AS, KSc and KSu) recruited 20 patients from Medicine of the

Elderly and orthopedic wards at the Royal Infirmary of Edinburgh, Scotland. Patients

7

meeting eligibility criteria were identified through consultation with the clinical care team.

Forty-three patients were identified, and of these nine patients declined to participate, three

patients were not available for testing because they were discharged or moved by the time the

researcher approached them, and eleven were deemed unsuitable by the researcher due to

lack of capacity (cognitive impairment, reduced arousal) or severity of illness. This

recruitment process yielded the final target sample size of 20 patients.

Measurement and Procedures

Participants first undertook a brief visual acuity test to ensure that they could perceive

the test stimuli. The visual acuity test comprised six short trials in which participants were

asked to (1) identify a change in the colour of a white circle (5cm diameter) presented on the

smartphone screen (white-to-grey and grey-to-white); (2) identify a change in the shape of

the stimulus (circle-to-star and star-to-circle); and (3) name two letters presented on the

screen one at a time (B and E) (Figure 1A). Testing proceeded only when the participants

could reliably perceive the stimuli.

Patients then completed visual sustained attention counting tasks on the EDTB and

DelApp (www.edinburghdelirium.ed.ac.uk) in counterbalanced order (see figure S1

published as supplementary material online attached to the electronic version of this paper at

http://journals.cambridge.org/ipg). The EDTB Mark 2 is a purpose-built cuboidal, grey

plastic box (13 x 21.5 x 7.5 cm) with two protruding circular illuminable buttons (5 cm

diameter), presented to participants in landscape orientation. Each button contains concealed

(when not illuminated) light-emitting diodes (LEDs) and is surrounded by four LEDs

8

concealed under a translucent white cover. The box further contains a concealed central 7 x 7

matrix of LEDs to display distracting stimuli such as checkered patterns (Tieges et al., 2013).

The DelApp task was presented on the 9.2 x 5.9 cm display of a Samsung Galaxy S2

running the Android 3.2 operating system. The brightness was set automatically based on the

ambient light. The target stimulus was a single large white circular illumination (5 cm

diameter) presented in the centre of the screen against a black background. On some trials,

this was surrounded by pseudo-random subsets of eight small downward-pointing triangles

(0.3 x 0.3 x 0.4 cm), each presented for 500 ms, to provide distraction (Figure 1B). The

smartphone was held in portrait orientation at a distance of approximately 50 cm from

participants.

All EDTB tasks involved presentation of one light only, instead of the two lights used

in the original EDTB studies, so as to more closely resemble the DelApp and thus provide a

more direct comparison. Participants were asked to count and then verbally report at the

conclusion of the sequence (as indicated by the researcher) the number of times they saw the

illuminations (i.e. illuminations of the button on the EDTB or circles presented on the

smartphone screen).

Both the EDTB and DelApp counting tasks comprised seven trials of increasing

difficulty, presented in a fixed order. This served to minimise floor and ceiling effects. The

illuminations lasted 1000 ms, and distracting stimuli lasted 500 ms each. The number of

illuminations increased from 2 to 8 across trials. The inter-stimulus interval of Level 1 (trials

1-3) ranged between 1000 ms and 2000 ms. Distracting stimuli were added for Level 2 (trials

4-5) and approximately twice as many distracting stimuli were shown in Level 3 (trials 6-7).

Trial one was a practice trial, which was repeated a number of times if necessary. Trials were

9

scored as correct or incorrect. If no answer was given, this was scored as incorrect. Thus, we

did not distinguish between incorrect responses and omissions. {Association, 2014

#173}Likewise, patients who could attempt only a subset of the trials still received a score

(i.e. total number of correct completed trials) and were included in the statistical analyses.

The end of each trial in the DelApp task was signalled to the researcher by a short

vibration generated by the phone.

The DelApp counting task was preceded by a brief assessment of level of arousal

(LoA). This was included because some patients with abnormal LoA cannot engage with

cognitive testing, and these patients are considered as showing severe inattention (see the

guidance notes in the 2013 DSM-5 diagnostic criteria for delirium (American Psychiatric

Association, 2013)). To allow for some grading of the DelApp test scores in patients unable

to perform the attention task, the arousal assessment comprised the following three items: (1)

Can the patient keep his/her eyes open for 10 sec? (2) Can the patient say his/her name? (3)

Can the patient track an object (e.g. badge, phone) for 5 sec? The arousal assessment was

combined with the seven-item counting task to provide an overall score out of 10. The full

DelApp assessment generally took less than five minutes to complete.

Following the EDTB and DelApp tests, participants underwent a semi-structured

interview in which they were asked a series of scripted open-ended questions relating to their

experience of the DelApp in comparison to the EDTB. Interviews were recorded and

transcribed. The whole testing and interview assessment typically lasted around 15 minutes.

10

Statistical Analysis

Comparisons between test scores on the DelApp and EDTB tasks were analysed using

a Wilcoxon signed-rank test. The threshold for statistical significance was p ≤ .05. A

qualitative thematic analysis was conducted on verbatim transcriptions of recorded interviews

(Braun and Clarke, 2006). All quantitative analyses were carried out in R version 3.0.1 (R

Core Team, 2014).

Study 2

Design

This study employed a between-subjects design, comparing performance on the

DelApp test in three groups: patients with delirium, dementia, and no severe cognitive

impairment. This was a case-control study and so we deliberately aimed to have groups that

were clear-cut clinically.

Participants

Inclusion and exclusion criteria were as for Study 1. Proxy consent was sought for

patients lacking capacity to provide consent for themselves. Patients were recruited from

Medicine of the Elderly and orthopedic wards at the Royal Infirmary of Edinburgh, and acute

and rehabilitation wards in Liberton Hospital, Edinburgh.

A total of 269 patients were initially identified by the clinical staff and researchers as

potentially appropriate for the study. Of these, 27 patients declined, two patients were not

11

available for testing because they had been discharged or moved, and two patients were

participating in a similar study and were therefore excluded. Thirty-six patients were not

suitable, due to severe cognitive impairments, severely reduced level of arousal such that no

engagement was possible, severity of illness or distress. Twenty-five patients who required

proxy consent or asked researchers to come again at a different time could not be approached

again due to researcher unavailability. Proxy consent was sought but could not be obtained

for one patient. A sample of 181 patients underwent assessment in the study. However, 25

patients were excluded after assessment because they could not be allocated to any of the pre-

defined study groups. The final study sample size was 156 (delirium: N = 50; dementia: N =

52; control: N = 54).

Measurement and Procedures

Cognition and delirium status were assessed on one occasion. All cognitive and

delirium assessments were conducted by graduate psychologists (AS, KSc and KSu) who had

been fully trained by a geriatrician (AM) and a postdoctoral psychologist (ZT). This training

process included: regular supervisor meetings; ward round observations; role play and mock

assessments; a supervisor-led teaching session on extracting information from case notes to

inform categorisation; and a certified course on Good Clinical Practice. Students followed

detailed Standard Operating Procedures for the delirium and cognitive assessments and they

were closely monitored throughout the recruitment and data acquisition stage by their

supervisors (AM and ZT). All tests were administered at the patient’s bedside, with the

curtain surrounding their bed closed.

12

The presence of delirium was assessed using the CAM (Inouye et al., 1990). Patients

who met CAM criteria for delirium were included in the delirium group irrespective of their

current or prior level of cognition. Severity was measured using the Delirium Rating Scale-

Revised-98 (DRS-R98 (Trzepacz et al., 2001). Scoring of the DRS-R98 was based on the

preceding 24 hours. Higher scores indicate greater severity and amount to a total score of 39

(severity subscale). Discussions with the clinical staff and scrutiny of the case notes provided

additional diagnostic and severity information.

Patients in the dementia group either had a prior formal diagnosis, or clearly met

DSM-IV criteria for dementia (using information from case notes and informants) as

determined by a consultant geriatrician. Patients with dementia did not have delirium at the

time of assessment. Patients were included in the control group only if they demonstrated

normal cognition on a general cognitive measure described below, and had no documented

history of chronic cognitive or functional impairment.

The two letter trials of the visual acuity assessment were discontinued as they were

considered redundant and related to a separate task involving letter sequences. The remaining

four trials and DelApp procedures were the same as for Study 1 These were accompanied by

additional measures of attention, arousal and cognition, as follows.

The short Orientation Memory Concentration Test (OMCT (Katzman et al., 1983))

was used to assess the overall level of cognition. This test is a validated measure of cognitive

impairment (Katzman et al., 1983). To facilitate analyses we scored the OMCT such that

higher scores indicate better performance. The suggested scores for categorisation are: 24-28

= normal cognition; 19-23 = questionable impairment, ≤18 = dementia (Morris et al.,

1989). A set of brief attention tests (Marcantonio, 2008) established as a method of assessing

13

inattention in delirium and here termed the Brief Attention Tests (BAT) was used to provide

a standardised measure of inattention. The BAT comprised: digit span forwards (3-5 digits),

digit span backward (3-4 digits), and days of the week and months of the year in reverse

order. Each correct item equated to one point and scores of 5 (out of 7) or below were

considered indicative of inattention. The BAT informed scoring of the inattention item on the

CAM and served as a comparison for the DelApp.

The Observational Scale of Level of Arousal (OSLA;

www.edinburghdelirium.ed.ac.uk) (Tieges et al., 2013) was included to provide a quantitative

measure of LoA and to aid assessment of delirium. It was developed in-house and designed

specifically to characterize abnormalities in LoA in patients with delirium. The OSLA

comprises four graded items: eye opening, eye contact, posture, and movement. Higher

scores indicate greater abnormality in LoA (maximum score is 15).

The order of attention assessments (DelApp, BAT) and cognitive assessment (OMCT)

was systematically counterbalanced across patients using a Latin squares design. The DRS-

R98, OSLA and CAM were completed following the assessments. All assessments including

DelApp and reference diagnosis for delirium were performed by single raters.

Following the assessment of delirium and cognition (including the DelApp), some

patients showed patterns of results on the cognitive tests that suggested the initial group

assignment (based on case notes and discussions with ward staff) was incorrect. For example,

patients who were initially deemed as potential controls by ward staff sometimes had

cognitive impairment as determined by the more detailed cognitive testing performed as part

of the present study. For such patients the most appropriate grouping was decided blind to

14

DelApp results by a consultant geriatrician (AM) based on all the other available information

(i.e. cognitive test scores, information taken from medical notes and conversations with ward

staff). Some patients could not be allocated to any group, because their test scores deviated

considerably from group medians and/or they did not present a symptom profile that was

clearly characteristic of any one group. These patients were excluded, again without

knowledge of the DelApp scores, since for the purposes of this study we aimed to select

patients who could be classified as being part of one of the predefined clinical groups.

Statistical Analysis

Non-parametric statistical tests were used as the majority of the data was non-

normally distributed and there was hetereogeneity of variance across groups. Kruskal-Wallis

and Mann-Whitney U tests were used for between-group analyses. Continuity corrections

were applied to Mann-Whitney U tests. Holm corrections were applied to multiple

comparisons. For these comparisons the 95% Confidence Interval (CI) limits are expressed as

differences in mean rank between each pair of groups. Kendall’s Tau was used for

correlations as there were many ties in the data (Field et al., 2012). Pearson’s chi-squared

tests were used for proportional data. Linear regressions were carried out using a conservative

threshold for significance of p ≤ .01, due to non-normality. Receiver operating characteristic

(ROC) curves were compared using DeLong's method (DeLong et al., 1988). Chi-square tests

were used to explore differences in performance between subsequent levels of task difficulty.

Odds ratios are also reported.

Medium to large effect sizes were found in a similar study (Brown et al., 2011).

Therefore, taking a relatively conservative estimate of effect size as 0.35, where p = 0.05 and

15

N = 141, all statistical tests used in the present analysis will have >95% power. This

calculation defined the minimum acceptable sample size.

16

RESULTS

Study 1

Twenty patients, aged 66-93, were recruited. No significant difference was found

between scores on the DelApp (median = 10, range 8-10) and EDTB (median = 10, range =

8-10; p = .41, r = -.19). Qualitative thematic analysis indicated that responses were centred

around four themes: Physical presentation of the DelApp; ease of DelApp task; further

development of the application; and device preference (EDTB or DelApp). All patients stated

that they had no difficulty perceiving the stimuli, or completing the tests on the DelApp.

Three participants mentioned that the peripheral distractors of the DelApp were not as

distracting as those on the EDTB. When asked which device they preferred, nine out of 20

patients had no preference, seven preferred the DelApp and four preferred the EDTB.

Study 2

Patient characteristics

Age differed significantly among groups (p < .001; Table 1). Delirium and dementia

groups were overall older than controls (delirium vs control: p < .001, 95% CI = [13.0,

52.27]; dementia vs. control: p < .001, 95% CI = [23.93, 62.81]). There was no difference in

age between dementia and delirium groups.

General cognition, as measured with the OMCT, differed between groups (Table 1).

Scores in the delirium and dementia groups were significantly lower compared to controls

17

(delirium vs. control: p < .001, 95% CI = [64.78, 91.04]; dementia vs. control: p < .001, 95%

CI = [58.59, 84.58]). There was no difference between dementia and delirium groups.

Between-group differences in attention

DelApp: Of the 156 patients included in the study, ten participants (eight from the

delirium group and two from the dementia group) were unable to provide answers on some or

all trials on the counting task. Where this occurred trials were scored as incorrect, as this was

considered the result of severe inattention, and the overall scores for these patients were

included in the statistical analyses. DelApp scores differed among groups (p < .001; Table 1

and Figure 2). Patients with delirium scored significantly lower than patients with dementia

(p < .001, 95% CI = [50.58, 74.38]) and controls (p < .001, 95% CI = [69.25, 92.84]). The

dementia group scored significantly lower than controls (p < .001, 95% CI = [6.90, 30.25]).

Age was not a significant predictor of DelApp score (β = -0.05, t (154) = -2.33, p

= .021, R2 = .034, 95% CI = [8.88, 15.86]).

BAT: BAT scores also differed between groups (p < .001; Table 1). The delirium and

dementia groups scored lower on the BAT than controls (delirium vs. control: p < .001, 95%

CI = [51.43, 83.86]; dementia vs. control: p < .001, 95% CI = [20.85, 52.96]). Scores were

also lower in delirium compared to dementia (p < .001, 95% CI = [14.37, 47.0]).

18

DelApp and abnormal LoA

OSLA scores reflecting LoA differed between groups (p < .001; Table 1). As

expected, the delirium group scored higher than patients with dementia (p < .001, 95% CI =

[60.2, 78.4]) and controls (p < .001, 95% CI = [66.67, 84.7]), indicating greater abnormality

in LoA in patients with delirium, while there was no difference between dementia and control

groups. There was a moderate negative correlation between OSLA and DelApp scores (τ =

-.61, p < .001), indicating that a more abnormal LoA was associated with worse inattention.

A total of seven patients, all from the delirium group, scored less than 3 on the DelApp

arousal sub-scale delirium group.

ROC analysis

ROC analysis was performed on the DelApp with delirium diagnosis as a reference.

The area under the curve (AUC) was 0.96 (p < .001, 95% CI = [0.93, 0.995]), with 98%

sensitivity and 93% specificity for detecting delirium in the study sample as a whole using a

cut-off of 8 (out of a maximum possible score of 10). The seven CAM-negative participants

who scored at or below this cut-off were all from the dementia group and had severe

cognitive impairments (median OMCT score = 5, range = 0-9). Analysis based on the

delirium and control groups alone revealed an AUC of 0.99 (p < .001, CI = [0.97, 1]), with

98% sensitivity and 100% specificity for detecting delirium. Analysis based on the delirium

and dementia groups alone returned an AUC of 0.93 (p < .001, CI = [0.88, 0.99]), with 98%

sensitivity and 87% specificity using the same cut-off of 8.

19

By comparison, the AUC of the BAT for detecting delirium in the entire study sample

using a cut-off of 5 (out of 7) was 0.82 (p < .001, 95% CI = [0.75, 0.89]), with 76%

sensitivity and 72% specificity. Direct comparison of the discriminative ability of these tests

found that the AUC was significantly higher for the DelApp compared to the BAT (z = 3.8,

p < .001, r = .37).

The DelApp as a measure of severity of attentional deficits

There was a moderate negative correlation between DelApp and DRS-R98 severity

scores (τ = -.60, p < .001). A 1-point decrease in DelApp score reflecting more inattention

was associated with a 4.8-point increase in DRS-R98 severity score (β = -0.208, t (154) = -

15.23, p < .001, R2 = .62, CI = [9.66, 10.3]; Figure 3).

The proportion of correct responses on the DelApp test given by patients in the

delirium group decreased as the level of difficulty increased. Based on odds ratios, the odds

of giving the correct response was 2.2 times higher for level 1 compared to level 2 (p

= .003, 95% CI = [1.26, 3.86]). Similarly, the odds of giving the correct response was 1.33

times higher for level 2 compared to level 3, though this was not statistically significant. No

effects of task level were found for the dementia group or controls. Frequencies of responses

are presented in Table 2.

20

DISCUSSION

This preliminary study showed initial feasibility and validity of a new smartphone-

based test of attention for use in delirium assessment. The DelApp was effective in

objectively measuring inattention associated with delirium in a selected sample of older

hospitalised patients. The DelApp was 98% sensitive and 93% specific to delirium in the

sample as a whole, which is similar to the diagnostic performance of the original EDTB tasks

(Brown et al., 2011). Furthermore, patients with delirium performed poorly on this test

compared to cognitively normal patients and patients with dementia, even though delirium

and dementia groups did not differ with respect to overall cognitive impairment as assessed

by a standard test. This is important because symptom profiles in delirium and dementia

commonly overlap, yet these disorders have radically different etiologies and treatments.

Patients with delirium performed more poorly on both the DelApp and BAT, which

includes digit span, than patients with dementia. However, there was substantial overlap in

BAT scores between delirium and dementia groups. Direct comparison of the discriminative

ability of DelApp and BAT tests indicated that the DelApp was more effective in detecting

delirium and discriminating delirium from dementia. This is not surprising because digit span

is not only employed as a test of attention in delirium, but also as a measure of IQ, working

memory, and executive functioning (Buchanan et al., 2010; Greneche et al., 2011; Iverson,

2001). Moreover, studies have shown that patients with dementia perform poorly on the digit

span backward (Meagher et al., 2010) and that the digit span forward does not distinguish

between delirium and dementia (O'Keeffe and Gosney, 1997). In contrast, the DelApp

attention assessment appears to provide a purer measure of attention, placing low demands on

other aspects of cognition.

21

The broad range of DelApp scores in the delirium group supports the notion that this

test potentially provides a graded measure of the severity of inattention in delirium. Further,

scores on the DelApp and the DRS-R98 were moderately well correlated. Thus, the DelApp

may help with severity grading of delirium. However, linear regression revealed large

residuals in the delirium group, which may be explained by the range of symptoms and

different time frames covered by each test. Specifically, the DelApp assesses current level of

arousal and inattention, whereas the DRS-R98 accounts for a broad range of delirium

symptoms occurring in the preceding 24 hours.

The DelApp test appears to have a number of strengths including its simplicity, ease-

of-use, objectivity, and portability. Controlled presentation of stimuli and automated scoring

make the DelApp less prone to experimenter bias or error, and it also requires little training

and does not strongly rely on clinical judgment. As such, the DelApp may be of particular

value when used by non-experts (Kean and Ryan, 2008). Our preliminary results suggest that

the DelApp could provide a valid measure of delirium severity and as such may be useful in

tracking change in delirium presence and severity over time.

Although abnormal LoA as measured with the OSLA was found in delirium, in line

with previous findings (Tieges et al., 2013), only seven patients (all with delirium) had

abnormal scores on the arousal assessment of the DelApp. In the present study, this subscale

therefore made a relatively small contribution to the overall diagnostic performance of the

DelApp. This finding may indicate that the DelApp arousal assessment was insensitive to

subtle abnormalities in LoA. However, spectrum bias may have played a role here, because

ward staff generally did not recommend patients who were asleep or very drowsy, or

22

extremely agitated, and so these patients were mostly not approached for recruitment.

Importantly, in clinical practice 10% of all patients may have abnormalities in LoA as

detected by routine assessments (Prytherch et al., 2010). Therefore, the utility of the DelApp

arousal subscale needs to be addressed in future studies with consecutive, unselected patients.

This study has several limitations that must be acknowledged. The DelApp

assessment and reference diagnosis of delirium were both carried out by single raters.

Although the raters did not consider the DelApp assessment when completing the CAM and

DRS-R98 (which always followed the DelApp assessment), it cannot be stated with certainty

if DelApp scores may have influenced delirium diagnosis to some extent or not. This may

have resulted in incorporation bias. It should be noted however that a diagnosis of delirium

and/or dementia was ascertained through communication with the clinical team responsible

for the care of a patient and by examination of the case notes. Further, the clinical profile of

most patients, including information from case notes and neuropsychological tests (blind to

the DelApp) was discussed with an experienced geriatrician to validate the diagnosis

clinically. Another study limitation is the use of a selected patient sample, which may affect

the generalizability of the present findings. The DelApp requires further validation by

confirming diagnostic test accuracy and potential for clinical applicability in a representative

population of consecutive, unselected patients with blinded raters who perform the reference

standard and index tests independently. Assessment of global cognition was done using the

OMCT (Katzman et al., 1983). Whilst this measure is validated and has the advantages of

brevity and ease-of-use, it does not provide a fine-grained measure of cognitive impairment.

Future studies should consider using a more comprehensive assessment of cognitive

impairment and also include a retrospective informant questionnaire such as the IQCODE

(Jorm and Jacomb, 1989) in order to obtain more accurate information about the possible

23

presence of pre-morbid cognitive impairment prior to admission. Finally, grouping of

delirium and co-morbid delirium-dementia precluded an investigation of the DelApp's ability

to diagnose delirium superimposed on dementia. Our reason for grouping these patients was

that the majority of older patients with delirium had some degree of underlying cognitive

impairment or (often undiagnosed) dementia and therefore it was hard to recruit a "pure"

delirium group with the means available to us at the time. Nonetheless, it would be

interesting in future studies to assess differences in DelApp performance between delirium

groups with and without dementia.

Feedback from patients indicated that the distractor lights on the hardware box were

experienced as more distracting than the distractor triangles in the DelApp task. Future

versions of the DelApp should therefore use distracting stimuli with a modified appearance

(larger size, brighter colour, and/or more frequent presentation) to make them stand out more.

A related issue is that, though patients with delirium performed worse on the third level of the

attention task compared to previous levels, this performance difference did not reach

significance. Thus, the attentional load of task level 3 could be increased in future versions of

the DelApp. In sum, further development of the DelApp is required, including optimisation of

test parameters, user interface and data acquisition and display. Also, in order for the DelApp

to be readily applied as a screening tool in routine clinical practice, the duration of the

DelApp assessment needs to be further shortened. This could be done by reducing the

number of trials, the length of counting sequences and/or the inter-stimulus delays, and also

by adopting a scoring system whereby the test is terminated following a specified number of

repeated errors. Further, a more detailed characterization of type and severity of dementia is

required to determine if our method can distinguish moderate to severe forms of dementia

from milder forms of delirium. Finally, performance on the DelApp should be compared

24

across different subtypes of dementia, including dementia with Lewy bodies which is

associated with prominent attentional deficits (Metzler-Baddeley, 2007).

Conclusions

The current study provides the first evidence for the feasibility and validity of a

smartphone-based assessment of attention in older hospitalised patients with delirium,

including those who are frail or acutely unwell. The findings also support and extend

previous observations that patients with delirium have specific impairments of sustained

attention. The DelApp shows promise as a sensitive, specific and valid tool to assist

identification and severity grading of delirium in research and clinical practice. In this regard,

the DelApp could potentially be integrated into current delirium assessment tools by

providing a robust and validated assessment of inattention. Further development of the

DelApp is now required to make it suitable as a screening tool for delirium in routine clinical

practice. Finally, studies are needed to assess validity in representative, unselected patients

using independent raters.

25

Conflicts of interest

AMacL holds patents on computerized devices and tests for measuring attention in delirium.

This study was funded by a Medical Research Council Centenary Early Career Award to Z.

Tieges. Funding from the Biotechnology and Biological Sciences Research Council, the

Engineering and Physical Sciences Research Council, the Economic and Social Research

Council, and the Medical Research Council is gratefully acknowledged. The authors also

thank the patients and staff from the Medicine of the Elderly and acute orthopedic wards of

the Royal Infirmary of Edinburgh and the acute and rehabilitation wards of Liberton Hospital

in Edinburgh.

Description of authors’ roles

Alasdair MacLullich (AM) and Zoë Tieges (ZT) designed the study and provided

supervision. Antaine Stíobhairt (AS), Klaudia Suchorab (KSu) and Katie Scott (KSc) carried

out patient recruitment, data collection and data analysis (with help of ZT), and also

contributed to study design. The manuscript was drafted by AS and ZT; AM, KSc and KSu

assisted with writing the manuscript. The DelApp software was developed by Alexander

Weir and Stuart Parks in collaboration with ZT and AM. Susan Shenkin helped with patient

recruitment, and provided input into the project plan and manuscript.

26

REFERENCES

Field, A. P., Miles, J. N. V. and Field, Z. C. (2012). Discovering Statistics Using R.

London: SAGE Publications.

American Psychiatric Association (2013). Diagnostic and statistical manual of mental

disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Braun, V. and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative

Research in Psychology, 3, 77-101.

Bronnick, K., Emre, M., Lane, R., Tekin, S. and Aarsland, D. (2007). Profile of cognitive

impairment in dementia associated with Parkinson's disease compared with Alzheimer's

disease. Journal of Neurology, Neurosurgery and Psychiatry, 78, 1064-1068.

Brouillette, R. M., et al. (2013). Feasibility, reliability, and validity of a smartphone based

application for the assessment of cognitive function in the elderly. PloS One, 8, e65925.

Brown, L. J., Fordyce, C., Zaghdani, H., Starr, J. M. and MacLullich, A. M. (2011).

Detecting deficits of sustained visual attention in delirium. Journal of Neurology,

Neurosurgery and Psychiatry, 82, 1334-1340.

Buchanan, T., Heffernan, T. M., Parrott, A. C., Ling, J., Rodgers, J. and Scholey, A. B.

(2010). A short self-report measure of problems with executive function suitable for

administration via the Internet. Behavior Research Methods, 42, 709-714.

DeLong, E. R., DeLong, D. M. and Clarke-Pearson, D. L. (1988). Comparing the areas

under two or more correlated receiver operating characteristic curves: a nonparametric

approach. Biometrics, 44, 837-845.

Greneche, J., Krieger, J., Bertrand, F., Erhardt, C., Maumy, M. and Tassi, P. (2011).

Short-term memory performances during sustained wakefulness in patients with obstructive

sleep apnea-hypopnea syndrome. Brain and Cognition, 75, 39-50.

27

Hall, R. J., Meagher, D. J. and MacLullich, A. M. (2012). Delirium detection and

monitoring outside the ICU. Best Practice & Research: Clinical Anaesthesiology, 26, 367-

383.

Inouye, S. K., van Dyck, C. H., Alessi, C. A., Balkin, S., Siegal, A. P. and Horwitz, R. I.

(1990). Clarifying confusion: the confusion assessment method. A new method for detection

of delirium. Annals of Internal Medicine, 113, 941-948.

Iverson, G. L. (2001). Interpreting change on the WAIS-III/WMS-III in clinical samples.

Archives of Clinical Neuropsychology, 16, 183-191.

Jorm, A. F. and Jacomb, P. A. (1989). The Informant Questionnaire on Cognitive Decline

in the Elderly (IQCODE): socio-demographic correlates, reliability, validity and some norms.

Psychological Medicine, 19, 1015-1022.

Katzman, R., Brown, T., Fuld, P., Peck, A., Schechter, R. and Schimmel, H. (1983).

Validation of a short Orientation-Memory-Concentration Test of cognitive impairment.

American Journal of Psychiatry, 140, 734-739.

Kean, J. and Ryan, K. (2008). Delirium detection in clinical practice and research: critique

of current tools and suggestions for future development. Journal of Psychosomatic Research,

65, 255-259.

Lines, C. R., Dawson, C., Preston, G. C., Reich, S., Foster, C. and Traub, M. (1991).

Memory and attention in patients with senile dementia of the Alzheimer type and in normal

elderly subjects. Journal of Clinical and Experimental Neuropsychology, 13, 691-702.

Marcantonio, E. R. (2008). Clinical management and prevention of delirium. Psychiatry, 7,

42-48.

Meagher, D. J., Leonard, M., Donnelly, S., Conroy, M., Saunders, J. and Trzepacz, P. T.

(2010). A comparison of neuropsychiatric and cognitive profiles in delirium, dementia,

28

comorbid delirium-dementia and cognitively intact controls. Journal of Neurology,

Neurosurgery and Psychiatry, 81, 876-881.

Metzler-Baddeley, C. (2007). A review of cognitive impairments in dementia with Lewy

bodies relative to Alzheimer's disease and Parkinson's disease with dementia. Cortex, 43,

583-600.

Morandi, A., et al. (2012). Tools to detect delirium superimposed on dementia: a systematic

review. Journal of the American Geriatrics Society, 60, 2005-2013.

Morris, J. C., et al. (1989). The Consortium to Establish a Registry for Alzheimer's Disease

(CERAD). Part I. Clinical and neuropsychological assessment of Alzheimer's disease.

Neurology, 39, 1159-1165.

O'Keeffe, S. T. and Gosney, M. A. (1997). Assessing attentiveness in older hospital

patients: global assessment versus tests of attention. Journal of the American Geriatrics

Society, 45, 470-473.

O'Regan, N. A., et al. (2014). Attention! A good bedside test for delirium? Journal of

Neurology, Neurosurgery and Psychiatry.

Perry, R. J., Watson, P. and Hodges, J. R. (2000). The nature and staging of attention

dysfunction in early (minimal and mild) Alzheimer's disease: relationship to episodic and

semantic memory impairment. Neuropsychologia, 38, 252-271.

Prytherch, D. R., Smith, G. B., Schmidt, P. E. and Featherstone, P. I. (2010). ViEWS--

Towards a national early warning score for detecting adult inpatient deterioration.

Resuscitation, 81, 932-937.

Siddiqi, N., House, A. O. and Holmes, J. D. (2006). Occurrence and outcome of delirium in

medical in-patients: a systematic literature review. Age and Ageing, 35, 350-364.

R Core Team (2014). R: A language and environment for statistical computing. R

Foundation for Statistical Computing. Vienna, Austria.

29

Tieges, Z., Brown, L. J. and Maclullich, A. M. (2014). Objective assessment of attention in

delirium: a narrative review. International Journal of Geriatric Psychiatry.

Tieges, Z., McGrath, A., Hall, R. J. and Maclullich, A. M. (2013). Abnormal level of

arousal as a predictor of delirium and inattention: an exploratory study. American Journal of

Geriatric Psychiatry, 21, 1244-1253.

Trzepacz, P. T., Mittal, D., Torres, R., Kanary, K., Norton, J. and Jimerson, N. (2001).

Validation of the Delirium Rating Scale-revised-98: comparison with the delirium rating

scale and the cognitive test for delirium. Journal of Neuropsychiatry and Clinical

Neurosciences, 13, 229-242.

Witlox, J., Eurelings, L. S., de Jonghe, J. F., Kalisvaart, K. J., Eikelenboom, P. and van

Gool, W. A. (2010). Delirium in elderly patients and the risk of postdischarge mortality,

institutionalization, and dementia: a meta-analysis. JAMA, 304, 443-451.

30

Table legends

Table 1. Demographics and pairwise comparisons between groups. Demographics and test

scores for each group are presented as median (inter-quartile range) unless otherwise

specified. Statistical comparisons between groups were performed using Kruskal-Wallis tests.

Pairwise comparisons were carried out using Mann-Whitney U tests. Holm corrections were

applied to control for type I error. The 95% Confidence Interval (CI) limits for these group

comparisons represent the differences in mean rank between each pair of groups.

Note. Calculations represent percentages of each group in which diagnoses are present.

Exhaustive comorbidity data was not recorded for all participants, therefore calculations are

subject to the availability of data. COPD = Chronic Obstructive Pulmonary Disease; CON

= controls; DEM = dementia; DEL = delirium; CI = Confidence intervals; OSLA =

Observational Scale of Level of Arousal; OMCT = Orientation-Memory-Concentration

Test.

Table 2. Distribution of responses for each group on the DelApp across three levels of

difficulty.

31

Figure legends

Figure 1. (A) The DelApp visual acuity test. The stimulus in trials 1-4 changed 5 sec after

trial onset and total trial duration was 10 sec. Letters (trials 5 and 6) were presented on screen

for 8 sec. The outer edges of these stimuli represent the outer edges of the smartphone’s

screen. (B) The DelApp attention task comprised seven trials of increasing difficulty

presented in a fixed order. Participants were instructed to count centrally presented circles

with a duration of 1 sec each, sometimes in the presence of distracting triangles (500ms each;

Levels 2 and 3). Number of circles to be counted ranged from 2 to 8. The inter-stimulus

interval between circles increased from 1-2 sec in Levels 1 and 2 to 3-4 sec in Level 3.

Patients were asked to verbally report the total number of counts per trial.

Figure 2. Boxplots illustrating the DelApp results of each group. The median is represented

by the thick horizontal bar. The interquartile range is represented by the height of the inner

boxes. The position of vertical bars represents the value of the most distant scores that are not

considered to be outliers. Outliers are represented by open circles.

Figure 3. Scatterplot demonstrating the relationship detween DelApp and DRS-R98 severity

scores. The diagonal black line is the overall regression line. Each group is represented

separately using the shapes listed in the legend above. Data points have been jittered. There

was a moderate negative correlation between DelApp and DRS-R98 severity scores (τ =

-.60, p < .001).

32

Supplementary material

Supplementary Figure 1. The Edinburgh Delirium Test Box (EDTB) Mark 2 instrument and

the smartphone which was used to administer the DelApp task.

33

Delirium

(N=50)

Dementia

(N=52)

Control

(N=54)

Statistical test results Pairwise comparisons

Age, years 85 (79.25-90) 87 (80-91) 75 (70-86) H (2) = 26.62, p < .001 DEL vs. CON: U = 750.5, p < .001, 95%

CI = [13, 52.27]

DEL vs. DEM: U = 1086.5, p = .19,

95% CI = [-9.09, 30.55]

DEM vs. CON: U = 658, p < .001, 95%

CI = [23.93, 62.81]

Comorbidities, %

Hypertension 48 37 44 X2 (2) = 1.44, p = .52

COPD 14 8 9 X2 (2) = 1.19, p = .57

Ischaemic heart

disease

14 10 11 X2 (2) = 0.5, p = .78

34

Diabetes mellitus 18 19 30 X2 (2) = 2.48, p = .32

Chronic kidney

disease

10 15 56 X2 (2) = 2.79, p = .25

Test scores

DelApp

(max. score = 10)

6 (4-7) 10 (9-10) 10 (10-10) H (2) = 102.67, p < .001 DEL vs. CON: U = 31.5, p < .001, 95%

CI = [69.25, 92.84]

DEL vs. DEM: U = 174.5, p < .001, 95%

CI = [50.58, 74.38]

DEM vs. CON: U = 985.5, p < .001, 95%

CI = [6.90, 30.25]

DRS-R-98

(max. score = 39)

25 (18.25-

29.5)

6 (4-8.25) 0.5 (0-1) H (2) = 132.41, p < .001 DEL vs. CON: U = 0, p < .001, 95% CI

= [21, 26]

35

DEL vs. DEM: U = 123.5 , p < .001,

95% CI = [15, 20]

DEM vs. CON: U = 80, p < .001 , 95%

CI = [4, 6]

OMCT

(max. score = 28)

8 (3-14.75) 12 (5.75-

15.25)

26 (24-28) H (2) = 97.47, p < .001 DEL vs. CON: U = 95.5, p < .001, 95%

CI = [64.78, 91.04]

DEL vs. DEM: U = 1100.5, p = .25, 95%

CI = [-6.91, 19.57]

DEM vs. CON: U = 21.5, p < .001, 95%

CI = [58.59, 84.58]

OSLA

(max. score = 15)

3 (2-5.75) 0 (0-0) 0 (0-0) H (2) = 118.65, p < .001 DEL vs. CON: U = 64, p < .001, 95 % CI

= [66.67, 84.70]

36

DEL vs. DEM: U = 121, p < .001, 95%

CI = [60.20, 78.40]

DEM vs. CON: U = 1265, p = .0995% CI

= [-15.32, 2.54]

BAT

(max. score = 7)

4 (2.25-5) 6 (4.75-6.26) 7 (6-7) H (2) = 62.19, p < .001 DEL vs. CON: U = 262.5, p < .001; 95%

CI = [51.43, 83.86]

DEL vs. DEM: U = 704.5, p < .001, 95%

CI = [14.37, 47.10]

DEM vs. CON: U = 656.5, p < .001, 95%

CI = [20.85, 52.96]

Table 1.

37

Group Leve

l

n (%) Correct n (%) Incorrect

Delirium 1 78 (52) 72 (48)

2 33 (33) 67 (67)

3 27 (27) 73 (73)

Dementia 1 143 (91.7) 13 (8.3)

2 89 (85.6) 15 (14.4)

3 89 (85.6) 15 (14.4)

Control 1 161 (99.4) 1 (0.62)

2 103 (95.3) 5 (4.6)

3 105 (97.2) 3 (2.8)

Table 2.

38

Figure 1

39

Figure 2

40

Figure 3.

41

Supplementary figure

42