· web viewdevelopment of a smartphone application for the objective detection of attentional...
Post on 21-Mar-2018
216 Views
Preview:
TRANSCRIPT
Development of a smartphone application for the objective detection of attentional
deficits in delirium
Authors
Dr. Zoë Tieges1,2* Antaine Stíobhairt1,3*, Katie Scott3, Klaudia Suchorab3, Alexander Weir4, Dr
Stuart Parks4, Dr Susan Shenkin1,2, Professor Alasdair MacLullich1,2.
Affiliations
1Edinburgh Delirium Research Group, University of Edinburgh
2Centre for Cognitive Ageing and Cognitive Epidemiology, University of Edinburgh
3Department of Psychology, University of Edinburgh
4Medical Devices Unit, NHS Greater Glasgow and Clyde
*Z. Tieges and A. Stíobhairt contributed equally to this paper.
Corresponding author
Dr. Zoë Tieges
Edinburgh Delirium Research Group
University of Edinburgh, Room S1642
Royal Infirmary of Edinburgh, 51 Little France Crescent
Edinburgh, EH16 4SA, UK
Tel. +44 (0) 131 242 6482, Fax. +44 (0)131 242 6370, E-Mail Zoe.Tieges@ed.ac.uk
1
ABSTRACT
Background
Delirium is an acute, severe deterioration in mental functioning. Inattention is the core
feature, yet there are few objective methods for assessing attentional deficits in delirium. We
previously developed a novel, graded test for objectively detecting inattention in delirium,
implemented on a computerised device (Edinburgh Delirium Test Box (EDTB)). Although
the EDTB is effective, tests on universally available devices have potential for greater
impact. Here we assessed feasibility and validity of the DelApp, a smartphone application
based on the EDTB.
Methods
This was a preliminary case-control study in hospital inpatients (aged 60-96) with delirium
(N=50), dementia (N=52) or no cognitive impairment (N=54) who performed the DelApp
assessment, which comprises an arousal assessment followed by counting of lights presented
serially. Delirium was assessed using the Confusion Assessment Method and Delirium Rating
Scale-Revised-98 (DRS-R98), and cognition with conventional tests of attention (e.g. digit
span) and the Short Orientation-Memory-Concentration test (OMCT).
Results
DelApp scores (maximum score=10) were lower in delirium (scores(median(IQR)): 6(4-7))
compared to dementia (10(9-10)) and control groups (10(10-10), p-values<0.001). Receiver
Operating Characteristic (ROC) analyses revealed excellent accuracy of the DelApp for
discriminating delirium from dementia (AUC=0.93), and delirium from controls (AUC=0.99,
2
p-values<0.001). DelApp and DRS-R98 severity scores were moderately well correlated
(Kendall's tau= -.60, p<0.001). OMCT scores did not differ between delirium and dementia.
Conclusions
The DelApp test showed good performance, supporting the utility of objectively measuring
attention in delirium assessment. This study provides evidence of the feasibility of using a
smartphone test for attentional assessment in hospital inpatients with possible delirium, with
potential applications in research and clinical practice.
Key words
Delirium; Attention; Objective; Measurement; Neuropsychological; Smartphone; Dementia;
Cognition
Running head
Detecting attentional deficits in delirium
3
INTRODUCTION
Delirium is an acute, severe neuropsychiatric syndrome characterized by fluctuating
disturbances in attention, arousal and cognition. It is highly prevalent in older hospitalized
patients and is associated with adverse outcomes including functional decline, new
institutionalization, persistent cognitive impairments including dementia, and higher
mortality (Siddiqi et al., 2006; Witlox et al., 2010).
The core diagnostic feature of delirium is 'inattention', defined as a 'reduced ability to
direct, focus, sustain and shift attention' in the DSM-5 criteria for delirium (American
Psychiatric Association, 2013). The evidence suggests that several aspects of attention are
likely to be affected in delirium, including the basic orienting response, focusing, sustaining
and dividing attention (Tieges et al., 2014). The extent to which each of these aspects of
attention is affected in delirium is poorly understood, though deficits in sustained attention
(i.e. the ability to maintain attention to stimuli over time) have been implicated (Brown et al.,
2011; O'Keeffe and Gosney, 1997; Tieges et al., 2014).
A range of methods is used in research and in clinical practice to ascertain inattention.
These include subjective assessments based on interview and clinical observation, and
objective assessments using brief neuropsychological tests (Hall et al., 2012). Objective tests
likely offer some advantages over subjective methods, including standardization of
instructions, and greater reliability and reproducibility. In addition, objective tests are less
likely to be operator dependent and reliant upon clinical experience than subjective
assessments. Several neuropsychological tests are currently used to assess inattention in
delirium. These include digit span, spatial span, months of the year and days of the week
4
backwards, vigilance 'A' and serial 7s (O'Regan et al., 2014; Tieges et al., 2014). Generally
these tests perform well in detecting delirium. However, the available data show that most
existing tests discriminate poorly between delirium and other mental disorders including
dementia (Morandi et al., 2012). Indeed, many studies show that patients with dementia show
significant deficits in tests often used to assess inattention in delirium, such as spatial span
(Meagher et al., 2010), serial 7s (Bronnick et al., 2007) and months of the year backwards
(Katzman et al., 1983). Further, existing tests of attention have not been validated for severity
grading in delirium, which is useful in monitoring delirium progress over time and providing
a more fine-grained measure of delirium than dichotomous scoring methods.
To help address the lack of robust, objective assessments for inattention in delirium,
we previously developed a novel attentional test implemented on a custom-built
computerized device entitled the Edinburgh Delirium Test Box (EDTB) (Brown et al., 2011;
Tieges et al., 2013). The EDTB tasks require participants to count and verbally report how
many times either one or two slowly presented lights on the device illuminate. The lights are
illuminated for 1 sec, and the inter-trial intervals are 1-4 sec. The design of this test was
motivated by the need for a task that was cognitively simple whilst placing demands on
attentional functioning. Prior research had found that the ability to sustain attention in a
simple counting task may be relatively spared in Alzheimer’s disease (Lines et al., 1991;
Morandi et al., 2012; Perry et al., 2000). The EDTB differs from these prior tasks in that the
stimuli are visual instead of auditory, and distracting stimuli are presented in some of the
trials in order to increase task difficulty. In our work with the EDTB we found that patients
with delirium showed marked deficits on these tasks, whereas patients with dementia and
cognitively unimpaired controls performed at or near ceiling level. The receiver operating
characteristic areas under the curves were between 0.80 and 1.00, indicating good to excellent
5
accuracy of the EDTB in discriminating delirium from dementia and control groups (Brown
et al., 2011). We concluded therefore that these new objective attentional tests offered
potential utility in detecting delirium and differentiating it from dementia. More broadly,
validated, objective neuropsychological assessments potentially offer a more robust approach
to the measurement of inattention in delirium than presently available methods.
We recently developed a software application for detecting deficits of visual sustained
attention in delirium ('DelApp'), which is based on our EDTB sustained attention tasks. The
rationale is that although the EDTB showed good performance, tests on devices that are
universally available, such as smartphones, are more readily applicable in research and
clinical settings. Smartphone-based applications are increasingly being used as a method for
assessing cognitive function in older individuals (Brouillette et al., 2013).
In this preliminary study, we first conducted a feasibility study to assess acceptability
of the DelApp for the assessment of attention in older hospitalised patients. We then
proceeded with a pilot case-control study to compare performance on the DelApp in a
selected sample of older hospitalised patients with dementia, delirium or no cognitive
impairment.
6
METHODS
Study 1
Design
This study was designed to assess the feasibility and acceptability of the DelApp in
hospitalised older patients, and to compare the EDTB and DelApp. Formal diagnostic
assessment of each patient was not performed as this was not required to meet the aims of
Study 1. A within-subjects design was used. Patients were studied on a single occasion and
underwent EDTB and DelApp tests. Semi-structured interviews with patients were conducted
to assess their opinions of these tests. Study 1 and Study 2 were approved by the Scotland A
Research Ethics Committee.
Participants
Patients aged 60 and older who were able to communicate in English were eligible.
All patients had to demonstrate capacity to provide written, informed consent; this was
because the ethics committee ruled that for Study 1 it was unnecessary to involve patients
requiring proxy consent. Exclusion criteria were: severe sensory impairments or severe acute
illness that would impede testing and interview, and where clinical staff considered that
participation would adversely affect the patient’s care.
The researchers (AS, KSc and KSu) recruited 20 patients from Medicine of the
Elderly and orthopedic wards at the Royal Infirmary of Edinburgh, Scotland. Patients
7
meeting eligibility criteria were identified through consultation with the clinical care team.
Forty-three patients were identified, and of these nine patients declined to participate, three
patients were not available for testing because they were discharged or moved by the time the
researcher approached them, and eleven were deemed unsuitable by the researcher due to
lack of capacity (cognitive impairment, reduced arousal) or severity of illness. This
recruitment process yielded the final target sample size of 20 patients.
Measurement and Procedures
Participants first undertook a brief visual acuity test to ensure that they could perceive
the test stimuli. The visual acuity test comprised six short trials in which participants were
asked to (1) identify a change in the colour of a white circle (5cm diameter) presented on the
smartphone screen (white-to-grey and grey-to-white); (2) identify a change in the shape of
the stimulus (circle-to-star and star-to-circle); and (3) name two letters presented on the
screen one at a time (B and E) (Figure 1A). Testing proceeded only when the participants
could reliably perceive the stimuli.
Patients then completed visual sustained attention counting tasks on the EDTB and
DelApp (www.edinburghdelirium.ed.ac.uk) in counterbalanced order (see figure S1
published as supplementary material online attached to the electronic version of this paper at
http://journals.cambridge.org/ipg). The EDTB Mark 2 is a purpose-built cuboidal, grey
plastic box (13 x 21.5 x 7.5 cm) with two protruding circular illuminable buttons (5 cm
diameter), presented to participants in landscape orientation. Each button contains concealed
(when not illuminated) light-emitting diodes (LEDs) and is surrounded by four LEDs
8
concealed under a translucent white cover. The box further contains a concealed central 7 x 7
matrix of LEDs to display distracting stimuli such as checkered patterns (Tieges et al., 2013).
The DelApp task was presented on the 9.2 x 5.9 cm display of a Samsung Galaxy S2
running the Android 3.2 operating system. The brightness was set automatically based on the
ambient light. The target stimulus was a single large white circular illumination (5 cm
diameter) presented in the centre of the screen against a black background. On some trials,
this was surrounded by pseudo-random subsets of eight small downward-pointing triangles
(0.3 x 0.3 x 0.4 cm), each presented for 500 ms, to provide distraction (Figure 1B). The
smartphone was held in portrait orientation at a distance of approximately 50 cm from
participants.
All EDTB tasks involved presentation of one light only, instead of the two lights used
in the original EDTB studies, so as to more closely resemble the DelApp and thus provide a
more direct comparison. Participants were asked to count and then verbally report at the
conclusion of the sequence (as indicated by the researcher) the number of times they saw the
illuminations (i.e. illuminations of the button on the EDTB or circles presented on the
smartphone screen).
Both the EDTB and DelApp counting tasks comprised seven trials of increasing
difficulty, presented in a fixed order. This served to minimise floor and ceiling effects. The
illuminations lasted 1000 ms, and distracting stimuli lasted 500 ms each. The number of
illuminations increased from 2 to 8 across trials. The inter-stimulus interval of Level 1 (trials
1-3) ranged between 1000 ms and 2000 ms. Distracting stimuli were added for Level 2 (trials
4-5) and approximately twice as many distracting stimuli were shown in Level 3 (trials 6-7).
Trial one was a practice trial, which was repeated a number of times if necessary. Trials were
9
scored as correct or incorrect. If no answer was given, this was scored as incorrect. Thus, we
did not distinguish between incorrect responses and omissions. {Association, 2014
#173}Likewise, patients who could attempt only a subset of the trials still received a score
(i.e. total number of correct completed trials) and were included in the statistical analyses.
The end of each trial in the DelApp task was signalled to the researcher by a short
vibration generated by the phone.
The DelApp counting task was preceded by a brief assessment of level of arousal
(LoA). This was included because some patients with abnormal LoA cannot engage with
cognitive testing, and these patients are considered as showing severe inattention (see the
guidance notes in the 2013 DSM-5 diagnostic criteria for delirium (American Psychiatric
Association, 2013)). To allow for some grading of the DelApp test scores in patients unable
to perform the attention task, the arousal assessment comprised the following three items: (1)
Can the patient keep his/her eyes open for 10 sec? (2) Can the patient say his/her name? (3)
Can the patient track an object (e.g. badge, phone) for 5 sec? The arousal assessment was
combined with the seven-item counting task to provide an overall score out of 10. The full
DelApp assessment generally took less than five minutes to complete.
Following the EDTB and DelApp tests, participants underwent a semi-structured
interview in which they were asked a series of scripted open-ended questions relating to their
experience of the DelApp in comparison to the EDTB. Interviews were recorded and
transcribed. The whole testing and interview assessment typically lasted around 15 minutes.
10
Statistical Analysis
Comparisons between test scores on the DelApp and EDTB tasks were analysed using
a Wilcoxon signed-rank test. The threshold for statistical significance was p ≤ .05. A
qualitative thematic analysis was conducted on verbatim transcriptions of recorded interviews
(Braun and Clarke, 2006). All quantitative analyses were carried out in R version 3.0.1 (R
Core Team, 2014).
Study 2
Design
This study employed a between-subjects design, comparing performance on the
DelApp test in three groups: patients with delirium, dementia, and no severe cognitive
impairment. This was a case-control study and so we deliberately aimed to have groups that
were clear-cut clinically.
Participants
Inclusion and exclusion criteria were as for Study 1. Proxy consent was sought for
patients lacking capacity to provide consent for themselves. Patients were recruited from
Medicine of the Elderly and orthopedic wards at the Royal Infirmary of Edinburgh, and acute
and rehabilitation wards in Liberton Hospital, Edinburgh.
A total of 269 patients were initially identified by the clinical staff and researchers as
potentially appropriate for the study. Of these, 27 patients declined, two patients were not
11
available for testing because they had been discharged or moved, and two patients were
participating in a similar study and were therefore excluded. Thirty-six patients were not
suitable, due to severe cognitive impairments, severely reduced level of arousal such that no
engagement was possible, severity of illness or distress. Twenty-five patients who required
proxy consent or asked researchers to come again at a different time could not be approached
again due to researcher unavailability. Proxy consent was sought but could not be obtained
for one patient. A sample of 181 patients underwent assessment in the study. However, 25
patients were excluded after assessment because they could not be allocated to any of the pre-
defined study groups. The final study sample size was 156 (delirium: N = 50; dementia: N =
52; control: N = 54).
Measurement and Procedures
Cognition and delirium status were assessed on one occasion. All cognitive and
delirium assessments were conducted by graduate psychologists (AS, KSc and KSu) who had
been fully trained by a geriatrician (AM) and a postdoctoral psychologist (ZT). This training
process included: regular supervisor meetings; ward round observations; role play and mock
assessments; a supervisor-led teaching session on extracting information from case notes to
inform categorisation; and a certified course on Good Clinical Practice. Students followed
detailed Standard Operating Procedures for the delirium and cognitive assessments and they
were closely monitored throughout the recruitment and data acquisition stage by their
supervisors (AM and ZT). All tests were administered at the patient’s bedside, with the
curtain surrounding their bed closed.
12
The presence of delirium was assessed using the CAM (Inouye et al., 1990). Patients
who met CAM criteria for delirium were included in the delirium group irrespective of their
current or prior level of cognition. Severity was measured using the Delirium Rating Scale-
Revised-98 (DRS-R98 (Trzepacz et al., 2001). Scoring of the DRS-R98 was based on the
preceding 24 hours. Higher scores indicate greater severity and amount to a total score of 39
(severity subscale). Discussions with the clinical staff and scrutiny of the case notes provided
additional diagnostic and severity information.
Patients in the dementia group either had a prior formal diagnosis, or clearly met
DSM-IV criteria for dementia (using information from case notes and informants) as
determined by a consultant geriatrician. Patients with dementia did not have delirium at the
time of assessment. Patients were included in the control group only if they demonstrated
normal cognition on a general cognitive measure described below, and had no documented
history of chronic cognitive or functional impairment.
The two letter trials of the visual acuity assessment were discontinued as they were
considered redundant and related to a separate task involving letter sequences. The remaining
four trials and DelApp procedures were the same as for Study 1 These were accompanied by
additional measures of attention, arousal and cognition, as follows.
The short Orientation Memory Concentration Test (OMCT (Katzman et al., 1983))
was used to assess the overall level of cognition. This test is a validated measure of cognitive
impairment (Katzman et al., 1983). To facilitate analyses we scored the OMCT such that
higher scores indicate better performance. The suggested scores for categorisation are: 24-28
= normal cognition; 19-23 = questionable impairment, ≤18 = dementia (Morris et al.,
1989). A set of brief attention tests (Marcantonio, 2008) established as a method of assessing
13
inattention in delirium and here termed the Brief Attention Tests (BAT) was used to provide
a standardised measure of inattention. The BAT comprised: digit span forwards (3-5 digits),
digit span backward (3-4 digits), and days of the week and months of the year in reverse
order. Each correct item equated to one point and scores of 5 (out of 7) or below were
considered indicative of inattention. The BAT informed scoring of the inattention item on the
CAM and served as a comparison for the DelApp.
The Observational Scale of Level of Arousal (OSLA;
www.edinburghdelirium.ed.ac.uk) (Tieges et al., 2013) was included to provide a quantitative
measure of LoA and to aid assessment of delirium. It was developed in-house and designed
specifically to characterize abnormalities in LoA in patients with delirium. The OSLA
comprises four graded items: eye opening, eye contact, posture, and movement. Higher
scores indicate greater abnormality in LoA (maximum score is 15).
The order of attention assessments (DelApp, BAT) and cognitive assessment (OMCT)
was systematically counterbalanced across patients using a Latin squares design. The DRS-
R98, OSLA and CAM were completed following the assessments. All assessments including
DelApp and reference diagnosis for delirium were performed by single raters.
Following the assessment of delirium and cognition (including the DelApp), some
patients showed patterns of results on the cognitive tests that suggested the initial group
assignment (based on case notes and discussions with ward staff) was incorrect. For example,
patients who were initially deemed as potential controls by ward staff sometimes had
cognitive impairment as determined by the more detailed cognitive testing performed as part
of the present study. For such patients the most appropriate grouping was decided blind to
14
DelApp results by a consultant geriatrician (AM) based on all the other available information
(i.e. cognitive test scores, information taken from medical notes and conversations with ward
staff). Some patients could not be allocated to any group, because their test scores deviated
considerably from group medians and/or they did not present a symptom profile that was
clearly characteristic of any one group. These patients were excluded, again without
knowledge of the DelApp scores, since for the purposes of this study we aimed to select
patients who could be classified as being part of one of the predefined clinical groups.
Statistical Analysis
Non-parametric statistical tests were used as the majority of the data was non-
normally distributed and there was hetereogeneity of variance across groups. Kruskal-Wallis
and Mann-Whitney U tests were used for between-group analyses. Continuity corrections
were applied to Mann-Whitney U tests. Holm corrections were applied to multiple
comparisons. For these comparisons the 95% Confidence Interval (CI) limits are expressed as
differences in mean rank between each pair of groups. Kendall’s Tau was used for
correlations as there were many ties in the data (Field et al., 2012). Pearson’s chi-squared
tests were used for proportional data. Linear regressions were carried out using a conservative
threshold for significance of p ≤ .01, due to non-normality. Receiver operating characteristic
(ROC) curves were compared using DeLong's method (DeLong et al., 1988). Chi-square tests
were used to explore differences in performance between subsequent levels of task difficulty.
Odds ratios are also reported.
Medium to large effect sizes were found in a similar study (Brown et al., 2011).
Therefore, taking a relatively conservative estimate of effect size as 0.35, where p = 0.05 and
15
N = 141, all statistical tests used in the present analysis will have >95% power. This
calculation defined the minimum acceptable sample size.
16
RESULTS
Study 1
Twenty patients, aged 66-93, were recruited. No significant difference was found
between scores on the DelApp (median = 10, range 8-10) and EDTB (median = 10, range =
8-10; p = .41, r = -.19). Qualitative thematic analysis indicated that responses were centred
around four themes: Physical presentation of the DelApp; ease of DelApp task; further
development of the application; and device preference (EDTB or DelApp). All patients stated
that they had no difficulty perceiving the stimuli, or completing the tests on the DelApp.
Three participants mentioned that the peripheral distractors of the DelApp were not as
distracting as those on the EDTB. When asked which device they preferred, nine out of 20
patients had no preference, seven preferred the DelApp and four preferred the EDTB.
Study 2
Patient characteristics
Age differed significantly among groups (p < .001; Table 1). Delirium and dementia
groups were overall older than controls (delirium vs control: p < .001, 95% CI = [13.0,
52.27]; dementia vs. control: p < .001, 95% CI = [23.93, 62.81]). There was no difference in
age between dementia and delirium groups.
General cognition, as measured with the OMCT, differed between groups (Table 1).
Scores in the delirium and dementia groups were significantly lower compared to controls
17
(delirium vs. control: p < .001, 95% CI = [64.78, 91.04]; dementia vs. control: p < .001, 95%
CI = [58.59, 84.58]). There was no difference between dementia and delirium groups.
Between-group differences in attention
DelApp: Of the 156 patients included in the study, ten participants (eight from the
delirium group and two from the dementia group) were unable to provide answers on some or
all trials on the counting task. Where this occurred trials were scored as incorrect, as this was
considered the result of severe inattention, and the overall scores for these patients were
included in the statistical analyses. DelApp scores differed among groups (p < .001; Table 1
and Figure 2). Patients with delirium scored significantly lower than patients with dementia
(p < .001, 95% CI = [50.58, 74.38]) and controls (p < .001, 95% CI = [69.25, 92.84]). The
dementia group scored significantly lower than controls (p < .001, 95% CI = [6.90, 30.25]).
Age was not a significant predictor of DelApp score (β = -0.05, t (154) = -2.33, p
= .021, R2 = .034, 95% CI = [8.88, 15.86]).
BAT: BAT scores also differed between groups (p < .001; Table 1). The delirium and
dementia groups scored lower on the BAT than controls (delirium vs. control: p < .001, 95%
CI = [51.43, 83.86]; dementia vs. control: p < .001, 95% CI = [20.85, 52.96]). Scores were
also lower in delirium compared to dementia (p < .001, 95% CI = [14.37, 47.0]).
18
DelApp and abnormal LoA
OSLA scores reflecting LoA differed between groups (p < .001; Table 1). As
expected, the delirium group scored higher than patients with dementia (p < .001, 95% CI =
[60.2, 78.4]) and controls (p < .001, 95% CI = [66.67, 84.7]), indicating greater abnormality
in LoA in patients with delirium, while there was no difference between dementia and control
groups. There was a moderate negative correlation between OSLA and DelApp scores (τ =
-.61, p < .001), indicating that a more abnormal LoA was associated with worse inattention.
A total of seven patients, all from the delirium group, scored less than 3 on the DelApp
arousal sub-scale delirium group.
ROC analysis
ROC analysis was performed on the DelApp with delirium diagnosis as a reference.
The area under the curve (AUC) was 0.96 (p < .001, 95% CI = [0.93, 0.995]), with 98%
sensitivity and 93% specificity for detecting delirium in the study sample as a whole using a
cut-off of 8 (out of a maximum possible score of 10). The seven CAM-negative participants
who scored at or below this cut-off were all from the dementia group and had severe
cognitive impairments (median OMCT score = 5, range = 0-9). Analysis based on the
delirium and control groups alone revealed an AUC of 0.99 (p < .001, CI = [0.97, 1]), with
98% sensitivity and 100% specificity for detecting delirium. Analysis based on the delirium
and dementia groups alone returned an AUC of 0.93 (p < .001, CI = [0.88, 0.99]), with 98%
sensitivity and 87% specificity using the same cut-off of 8.
19
By comparison, the AUC of the BAT for detecting delirium in the entire study sample
using a cut-off of 5 (out of 7) was 0.82 (p < .001, 95% CI = [0.75, 0.89]), with 76%
sensitivity and 72% specificity. Direct comparison of the discriminative ability of these tests
found that the AUC was significantly higher for the DelApp compared to the BAT (z = 3.8,
p < .001, r = .37).
The DelApp as a measure of severity of attentional deficits
There was a moderate negative correlation between DelApp and DRS-R98 severity
scores (τ = -.60, p < .001). A 1-point decrease in DelApp score reflecting more inattention
was associated with a 4.8-point increase in DRS-R98 severity score (β = -0.208, t (154) = -
15.23, p < .001, R2 = .62, CI = [9.66, 10.3]; Figure 3).
The proportion of correct responses on the DelApp test given by patients in the
delirium group decreased as the level of difficulty increased. Based on odds ratios, the odds
of giving the correct response was 2.2 times higher for level 1 compared to level 2 (p
= .003, 95% CI = [1.26, 3.86]). Similarly, the odds of giving the correct response was 1.33
times higher for level 2 compared to level 3, though this was not statistically significant. No
effects of task level were found for the dementia group or controls. Frequencies of responses
are presented in Table 2.
20
DISCUSSION
This preliminary study showed initial feasibility and validity of a new smartphone-
based test of attention for use in delirium assessment. The DelApp was effective in
objectively measuring inattention associated with delirium in a selected sample of older
hospitalised patients. The DelApp was 98% sensitive and 93% specific to delirium in the
sample as a whole, which is similar to the diagnostic performance of the original EDTB tasks
(Brown et al., 2011). Furthermore, patients with delirium performed poorly on this test
compared to cognitively normal patients and patients with dementia, even though delirium
and dementia groups did not differ with respect to overall cognitive impairment as assessed
by a standard test. This is important because symptom profiles in delirium and dementia
commonly overlap, yet these disorders have radically different etiologies and treatments.
Patients with delirium performed more poorly on both the DelApp and BAT, which
includes digit span, than patients with dementia. However, there was substantial overlap in
BAT scores between delirium and dementia groups. Direct comparison of the discriminative
ability of DelApp and BAT tests indicated that the DelApp was more effective in detecting
delirium and discriminating delirium from dementia. This is not surprising because digit span
is not only employed as a test of attention in delirium, but also as a measure of IQ, working
memory, and executive functioning (Buchanan et al., 2010; Greneche et al., 2011; Iverson,
2001). Moreover, studies have shown that patients with dementia perform poorly on the digit
span backward (Meagher et al., 2010) and that the digit span forward does not distinguish
between delirium and dementia (O'Keeffe and Gosney, 1997). In contrast, the DelApp
attention assessment appears to provide a purer measure of attention, placing low demands on
other aspects of cognition.
21
The broad range of DelApp scores in the delirium group supports the notion that this
test potentially provides a graded measure of the severity of inattention in delirium. Further,
scores on the DelApp and the DRS-R98 were moderately well correlated. Thus, the DelApp
may help with severity grading of delirium. However, linear regression revealed large
residuals in the delirium group, which may be explained by the range of symptoms and
different time frames covered by each test. Specifically, the DelApp assesses current level of
arousal and inattention, whereas the DRS-R98 accounts for a broad range of delirium
symptoms occurring in the preceding 24 hours.
The DelApp test appears to have a number of strengths including its simplicity, ease-
of-use, objectivity, and portability. Controlled presentation of stimuli and automated scoring
make the DelApp less prone to experimenter bias or error, and it also requires little training
and does not strongly rely on clinical judgment. As such, the DelApp may be of particular
value when used by non-experts (Kean and Ryan, 2008). Our preliminary results suggest that
the DelApp could provide a valid measure of delirium severity and as such may be useful in
tracking change in delirium presence and severity over time.
Although abnormal LoA as measured with the OSLA was found in delirium, in line
with previous findings (Tieges et al., 2013), only seven patients (all with delirium) had
abnormal scores on the arousal assessment of the DelApp. In the present study, this subscale
therefore made a relatively small contribution to the overall diagnostic performance of the
DelApp. This finding may indicate that the DelApp arousal assessment was insensitive to
subtle abnormalities in LoA. However, spectrum bias may have played a role here, because
ward staff generally did not recommend patients who were asleep or very drowsy, or
22
extremely agitated, and so these patients were mostly not approached for recruitment.
Importantly, in clinical practice 10% of all patients may have abnormalities in LoA as
detected by routine assessments (Prytherch et al., 2010). Therefore, the utility of the DelApp
arousal subscale needs to be addressed in future studies with consecutive, unselected patients.
This study has several limitations that must be acknowledged. The DelApp
assessment and reference diagnosis of delirium were both carried out by single raters.
Although the raters did not consider the DelApp assessment when completing the CAM and
DRS-R98 (which always followed the DelApp assessment), it cannot be stated with certainty
if DelApp scores may have influenced delirium diagnosis to some extent or not. This may
have resulted in incorporation bias. It should be noted however that a diagnosis of delirium
and/or dementia was ascertained through communication with the clinical team responsible
for the care of a patient and by examination of the case notes. Further, the clinical profile of
most patients, including information from case notes and neuropsychological tests (blind to
the DelApp) was discussed with an experienced geriatrician to validate the diagnosis
clinically. Another study limitation is the use of a selected patient sample, which may affect
the generalizability of the present findings. The DelApp requires further validation by
confirming diagnostic test accuracy and potential for clinical applicability in a representative
population of consecutive, unselected patients with blinded raters who perform the reference
standard and index tests independently. Assessment of global cognition was done using the
OMCT (Katzman et al., 1983). Whilst this measure is validated and has the advantages of
brevity and ease-of-use, it does not provide a fine-grained measure of cognitive impairment.
Future studies should consider using a more comprehensive assessment of cognitive
impairment and also include a retrospective informant questionnaire such as the IQCODE
(Jorm and Jacomb, 1989) in order to obtain more accurate information about the possible
23
presence of pre-morbid cognitive impairment prior to admission. Finally, grouping of
delirium and co-morbid delirium-dementia precluded an investigation of the DelApp's ability
to diagnose delirium superimposed on dementia. Our reason for grouping these patients was
that the majority of older patients with delirium had some degree of underlying cognitive
impairment or (often undiagnosed) dementia and therefore it was hard to recruit a "pure"
delirium group with the means available to us at the time. Nonetheless, it would be
interesting in future studies to assess differences in DelApp performance between delirium
groups with and without dementia.
Feedback from patients indicated that the distractor lights on the hardware box were
experienced as more distracting than the distractor triangles in the DelApp task. Future
versions of the DelApp should therefore use distracting stimuli with a modified appearance
(larger size, brighter colour, and/or more frequent presentation) to make them stand out more.
A related issue is that, though patients with delirium performed worse on the third level of the
attention task compared to previous levels, this performance difference did not reach
significance. Thus, the attentional load of task level 3 could be increased in future versions of
the DelApp. In sum, further development of the DelApp is required, including optimisation of
test parameters, user interface and data acquisition and display. Also, in order for the DelApp
to be readily applied as a screening tool in routine clinical practice, the duration of the
DelApp assessment needs to be further shortened. This could be done by reducing the
number of trials, the length of counting sequences and/or the inter-stimulus delays, and also
by adopting a scoring system whereby the test is terminated following a specified number of
repeated errors. Further, a more detailed characterization of type and severity of dementia is
required to determine if our method can distinguish moderate to severe forms of dementia
from milder forms of delirium. Finally, performance on the DelApp should be compared
24
across different subtypes of dementia, including dementia with Lewy bodies which is
associated with prominent attentional deficits (Metzler-Baddeley, 2007).
Conclusions
The current study provides the first evidence for the feasibility and validity of a
smartphone-based assessment of attention in older hospitalised patients with delirium,
including those who are frail or acutely unwell. The findings also support and extend
previous observations that patients with delirium have specific impairments of sustained
attention. The DelApp shows promise as a sensitive, specific and valid tool to assist
identification and severity grading of delirium in research and clinical practice. In this regard,
the DelApp could potentially be integrated into current delirium assessment tools by
providing a robust and validated assessment of inattention. Further development of the
DelApp is now required to make it suitable as a screening tool for delirium in routine clinical
practice. Finally, studies are needed to assess validity in representative, unselected patients
using independent raters.
25
Conflicts of interest
AMacL holds patents on computerized devices and tests for measuring attention in delirium.
This study was funded by a Medical Research Council Centenary Early Career Award to Z.
Tieges. Funding from the Biotechnology and Biological Sciences Research Council, the
Engineering and Physical Sciences Research Council, the Economic and Social Research
Council, and the Medical Research Council is gratefully acknowledged. The authors also
thank the patients and staff from the Medicine of the Elderly and acute orthopedic wards of
the Royal Infirmary of Edinburgh and the acute and rehabilitation wards of Liberton Hospital
in Edinburgh.
Description of authors’ roles
Alasdair MacLullich (AM) and Zoë Tieges (ZT) designed the study and provided
supervision. Antaine Stíobhairt (AS), Klaudia Suchorab (KSu) and Katie Scott (KSc) carried
out patient recruitment, data collection and data analysis (with help of ZT), and also
contributed to study design. The manuscript was drafted by AS and ZT; AM, KSc and KSu
assisted with writing the manuscript. The DelApp software was developed by Alexander
Weir and Stuart Parks in collaboration with ZT and AM. Susan Shenkin helped with patient
recruitment, and provided input into the project plan and manuscript.
26
REFERENCES
Field, A. P., Miles, J. N. V. and Field, Z. C. (2012). Discovering Statistics Using R.
London: SAGE Publications.
American Psychiatric Association (2013). Diagnostic and statistical manual of mental
disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.
Braun, V. and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative
Research in Psychology, 3, 77-101.
Bronnick, K., Emre, M., Lane, R., Tekin, S. and Aarsland, D. (2007). Profile of cognitive
impairment in dementia associated with Parkinson's disease compared with Alzheimer's
disease. Journal of Neurology, Neurosurgery and Psychiatry, 78, 1064-1068.
Brouillette, R. M., et al. (2013). Feasibility, reliability, and validity of a smartphone based
application for the assessment of cognitive function in the elderly. PloS One, 8, e65925.
Brown, L. J., Fordyce, C., Zaghdani, H., Starr, J. M. and MacLullich, A. M. (2011).
Detecting deficits of sustained visual attention in delirium. Journal of Neurology,
Neurosurgery and Psychiatry, 82, 1334-1340.
Buchanan, T., Heffernan, T. M., Parrott, A. C., Ling, J., Rodgers, J. and Scholey, A. B.
(2010). A short self-report measure of problems with executive function suitable for
administration via the Internet. Behavior Research Methods, 42, 709-714.
DeLong, E. R., DeLong, D. M. and Clarke-Pearson, D. L. (1988). Comparing the areas
under two or more correlated receiver operating characteristic curves: a nonparametric
approach. Biometrics, 44, 837-845.
Greneche, J., Krieger, J., Bertrand, F., Erhardt, C., Maumy, M. and Tassi, P. (2011).
Short-term memory performances during sustained wakefulness in patients with obstructive
sleep apnea-hypopnea syndrome. Brain and Cognition, 75, 39-50.
27
Hall, R. J., Meagher, D. J. and MacLullich, A. M. (2012). Delirium detection and
monitoring outside the ICU. Best Practice & Research: Clinical Anaesthesiology, 26, 367-
383.
Inouye, S. K., van Dyck, C. H., Alessi, C. A., Balkin, S., Siegal, A. P. and Horwitz, R. I.
(1990). Clarifying confusion: the confusion assessment method. A new method for detection
of delirium. Annals of Internal Medicine, 113, 941-948.
Iverson, G. L. (2001). Interpreting change on the WAIS-III/WMS-III in clinical samples.
Archives of Clinical Neuropsychology, 16, 183-191.
Jorm, A. F. and Jacomb, P. A. (1989). The Informant Questionnaire on Cognitive Decline
in the Elderly (IQCODE): socio-demographic correlates, reliability, validity and some norms.
Psychological Medicine, 19, 1015-1022.
Katzman, R., Brown, T., Fuld, P., Peck, A., Schechter, R. and Schimmel, H. (1983).
Validation of a short Orientation-Memory-Concentration Test of cognitive impairment.
American Journal of Psychiatry, 140, 734-739.
Kean, J. and Ryan, K. (2008). Delirium detection in clinical practice and research: critique
of current tools and suggestions for future development. Journal of Psychosomatic Research,
65, 255-259.
Lines, C. R., Dawson, C., Preston, G. C., Reich, S., Foster, C. and Traub, M. (1991).
Memory and attention in patients with senile dementia of the Alzheimer type and in normal
elderly subjects. Journal of Clinical and Experimental Neuropsychology, 13, 691-702.
Marcantonio, E. R. (2008). Clinical management and prevention of delirium. Psychiatry, 7,
42-48.
Meagher, D. J., Leonard, M., Donnelly, S., Conroy, M., Saunders, J. and Trzepacz, P. T.
(2010). A comparison of neuropsychiatric and cognitive profiles in delirium, dementia,
28
comorbid delirium-dementia and cognitively intact controls. Journal of Neurology,
Neurosurgery and Psychiatry, 81, 876-881.
Metzler-Baddeley, C. (2007). A review of cognitive impairments in dementia with Lewy
bodies relative to Alzheimer's disease and Parkinson's disease with dementia. Cortex, 43,
583-600.
Morandi, A., et al. (2012). Tools to detect delirium superimposed on dementia: a systematic
review. Journal of the American Geriatrics Society, 60, 2005-2013.
Morris, J. C., et al. (1989). The Consortium to Establish a Registry for Alzheimer's Disease
(CERAD). Part I. Clinical and neuropsychological assessment of Alzheimer's disease.
Neurology, 39, 1159-1165.
O'Keeffe, S. T. and Gosney, M. A. (1997). Assessing attentiveness in older hospital
patients: global assessment versus tests of attention. Journal of the American Geriatrics
Society, 45, 470-473.
O'Regan, N. A., et al. (2014). Attention! A good bedside test for delirium? Journal of
Neurology, Neurosurgery and Psychiatry.
Perry, R. J., Watson, P. and Hodges, J. R. (2000). The nature and staging of attention
dysfunction in early (minimal and mild) Alzheimer's disease: relationship to episodic and
semantic memory impairment. Neuropsychologia, 38, 252-271.
Prytherch, D. R., Smith, G. B., Schmidt, P. E. and Featherstone, P. I. (2010). ViEWS--
Towards a national early warning score for detecting adult inpatient deterioration.
Resuscitation, 81, 932-937.
Siddiqi, N., House, A. O. and Holmes, J. D. (2006). Occurrence and outcome of delirium in
medical in-patients: a systematic literature review. Age and Ageing, 35, 350-364.
R Core Team (2014). R: A language and environment for statistical computing. R
Foundation for Statistical Computing. Vienna, Austria.
29
Tieges, Z., Brown, L. J. and Maclullich, A. M. (2014). Objective assessment of attention in
delirium: a narrative review. International Journal of Geriatric Psychiatry.
Tieges, Z., McGrath, A., Hall, R. J. and Maclullich, A. M. (2013). Abnormal level of
arousal as a predictor of delirium and inattention: an exploratory study. American Journal of
Geriatric Psychiatry, 21, 1244-1253.
Trzepacz, P. T., Mittal, D., Torres, R., Kanary, K., Norton, J. and Jimerson, N. (2001).
Validation of the Delirium Rating Scale-revised-98: comparison with the delirium rating
scale and the cognitive test for delirium. Journal of Neuropsychiatry and Clinical
Neurosciences, 13, 229-242.
Witlox, J., Eurelings, L. S., de Jonghe, J. F., Kalisvaart, K. J., Eikelenboom, P. and van
Gool, W. A. (2010). Delirium in elderly patients and the risk of postdischarge mortality,
institutionalization, and dementia: a meta-analysis. JAMA, 304, 443-451.
30
Table legends
Table 1. Demographics and pairwise comparisons between groups. Demographics and test
scores for each group are presented as median (inter-quartile range) unless otherwise
specified. Statistical comparisons between groups were performed using Kruskal-Wallis tests.
Pairwise comparisons were carried out using Mann-Whitney U tests. Holm corrections were
applied to control for type I error. The 95% Confidence Interval (CI) limits for these group
comparisons represent the differences in mean rank between each pair of groups.
Note. Calculations represent percentages of each group in which diagnoses are present.
Exhaustive comorbidity data was not recorded for all participants, therefore calculations are
subject to the availability of data. COPD = Chronic Obstructive Pulmonary Disease; CON
= controls; DEM = dementia; DEL = delirium; CI = Confidence intervals; OSLA =
Observational Scale of Level of Arousal; OMCT = Orientation-Memory-Concentration
Test.
Table 2. Distribution of responses for each group on the DelApp across three levels of
difficulty.
31
Figure legends
Figure 1. (A) The DelApp visual acuity test. The stimulus in trials 1-4 changed 5 sec after
trial onset and total trial duration was 10 sec. Letters (trials 5 and 6) were presented on screen
for 8 sec. The outer edges of these stimuli represent the outer edges of the smartphone’s
screen. (B) The DelApp attention task comprised seven trials of increasing difficulty
presented in a fixed order. Participants were instructed to count centrally presented circles
with a duration of 1 sec each, sometimes in the presence of distracting triangles (500ms each;
Levels 2 and 3). Number of circles to be counted ranged from 2 to 8. The inter-stimulus
interval between circles increased from 1-2 sec in Levels 1 and 2 to 3-4 sec in Level 3.
Patients were asked to verbally report the total number of counts per trial.
Figure 2. Boxplots illustrating the DelApp results of each group. The median is represented
by the thick horizontal bar. The interquartile range is represented by the height of the inner
boxes. The position of vertical bars represents the value of the most distant scores that are not
considered to be outliers. Outliers are represented by open circles.
Figure 3. Scatterplot demonstrating the relationship detween DelApp and DRS-R98 severity
scores. The diagonal black line is the overall regression line. Each group is represented
separately using the shapes listed in the legend above. Data points have been jittered. There
was a moderate negative correlation between DelApp and DRS-R98 severity scores (τ =
-.60, p < .001).
32
Supplementary material
Supplementary Figure 1. The Edinburgh Delirium Test Box (EDTB) Mark 2 instrument and
the smartphone which was used to administer the DelApp task.
33
Delirium
(N=50)
Dementia
(N=52)
Control
(N=54)
Statistical test results Pairwise comparisons
Age, years 85 (79.25-90) 87 (80-91) 75 (70-86) H (2) = 26.62, p < .001 DEL vs. CON: U = 750.5, p < .001, 95%
CI = [13, 52.27]
DEL vs. DEM: U = 1086.5, p = .19,
95% CI = [-9.09, 30.55]
DEM vs. CON: U = 658, p < .001, 95%
CI = [23.93, 62.81]
Comorbidities, %
Hypertension 48 37 44 X2 (2) = 1.44, p = .52
COPD 14 8 9 X2 (2) = 1.19, p = .57
Ischaemic heart
disease
14 10 11 X2 (2) = 0.5, p = .78
34
Diabetes mellitus 18 19 30 X2 (2) = 2.48, p = .32
Chronic kidney
disease
10 15 56 X2 (2) = 2.79, p = .25
Test scores
DelApp
(max. score = 10)
6 (4-7) 10 (9-10) 10 (10-10) H (2) = 102.67, p < .001 DEL vs. CON: U = 31.5, p < .001, 95%
CI = [69.25, 92.84]
DEL vs. DEM: U = 174.5, p < .001, 95%
CI = [50.58, 74.38]
DEM vs. CON: U = 985.5, p < .001, 95%
CI = [6.90, 30.25]
DRS-R-98
(max. score = 39)
25 (18.25-
29.5)
6 (4-8.25) 0.5 (0-1) H (2) = 132.41, p < .001 DEL vs. CON: U = 0, p < .001, 95% CI
= [21, 26]
35
DEL vs. DEM: U = 123.5 , p < .001,
95% CI = [15, 20]
DEM vs. CON: U = 80, p < .001 , 95%
CI = [4, 6]
OMCT
(max. score = 28)
8 (3-14.75) 12 (5.75-
15.25)
26 (24-28) H (2) = 97.47, p < .001 DEL vs. CON: U = 95.5, p < .001, 95%
CI = [64.78, 91.04]
DEL vs. DEM: U = 1100.5, p = .25, 95%
CI = [-6.91, 19.57]
DEM vs. CON: U = 21.5, p < .001, 95%
CI = [58.59, 84.58]
OSLA
(max. score = 15)
3 (2-5.75) 0 (0-0) 0 (0-0) H (2) = 118.65, p < .001 DEL vs. CON: U = 64, p < .001, 95 % CI
= [66.67, 84.70]
36
DEL vs. DEM: U = 121, p < .001, 95%
CI = [60.20, 78.40]
DEM vs. CON: U = 1265, p = .0995% CI
= [-15.32, 2.54]
BAT
(max. score = 7)
4 (2.25-5) 6 (4.75-6.26) 7 (6-7) H (2) = 62.19, p < .001 DEL vs. CON: U = 262.5, p < .001; 95%
CI = [51.43, 83.86]
DEL vs. DEM: U = 704.5, p < .001, 95%
CI = [14.37, 47.10]
DEM vs. CON: U = 656.5, p < .001, 95%
CI = [20.85, 52.96]
Table 1.
37
Group Leve
l
n (%) Correct n (%) Incorrect
Delirium 1 78 (52) 72 (48)
2 33 (33) 67 (67)
3 27 (27) 73 (73)
Dementia 1 143 (91.7) 13 (8.3)
2 89 (85.6) 15 (14.4)
3 89 (85.6) 15 (14.4)
Control 1 161 (99.4) 1 (0.62)
2 103 (95.3) 5 (4.6)
3 105 (97.2) 3 (2.8)
Table 2.
38
Figure 1
39
Figure 2
40
Figure 3.
41
Supplementary figure
42
top related