visually defining and querying consistent multi-granular clinical temporal abstractions

Va

CD

a

ARRA

KTTMVTH

0d

Artificial Intelligence in Medicine 54 (2012) 75– 101

Contents lists available at SciVerse ScienceDirect

Artificial Intelligence in Medicine

jou rn al h om epage: www.elsev ier .com/ locate /a i im

isually defining and querying consistent multi-granular clinical temporalbstractions

arlo Combi, Barbara Oliboni ∗

epartment of Computer Science, University of Verona, Ca’ Vignal 2, Strada le Grazie 15, I-37134 Verona, VR, Italy

r t i c l e i n f o

rticle history:eceived 13 February 2008eceived in revised form 12 October 2011ccepted 16 October 2011

eywords:emporal abstractionsemporal granularitiesetaphors

isual query languagesemporal clinical dataemodialysis data

a b s t r a c t

Objective: The main goal of this work is to propose a framework for the visual specification and query ofconsistent multi-granular clinical temporal abstractions. We focus on the issue of querying patient clinicalinformation by visually defining and composing temporal abstractions, i.e., high level patterns derivedfrom several time-stamped raw data. In particular, we focus on the visual specification of consistenttemporal abstractions with different granularities and on the visual composition of different temporalabstractions for querying clinical databases.Background: Temporal abstractions on clinical data provide a concise and high-level description of tem-poral raw data, and a suitable way to support decision making. Granularities define partitions on the timeline and allow one to represent time and, thus, temporal clinical information at different levels of detail,according to the requirements coming from the represented clinical domain. The visual representationof temporal information has been considered since several years in clinical domains. Proposed visual-ization techniques must be easy and quick to understand, and could benefit from visual metaphors thatdo not lead to ambiguous interpretations. Recently, physical metaphors such as strips, springs, weights,and wires have been proposed and evaluated on clinical users for the specification of temporal clinicalabstractions. Visual approaches to boolean queries have been considered in the last years and confirmedthat the visual support to the specification of complex boolean queries is both an important and difficultresearch topic.Methodology: We propose and describe a visual language for the definition of temporal abstractions basedon a set of intuitive metaphors (striped wall, plastered wall, brick wall), allowing the clinician to usedifferent granularities. A new algorithm, underlying the visual language, allows the physician to specifyonly consistent abstractions, i.e., abstractions not containing contradictory conditions on the componentabstractions. Moreover, we propose a visual query language where different temporal abstractions canbe composed to build complex queries: temporal abstractions are visually connected through the usuallogical connectives AND, OR, and NOT.Results: The proposed visual language allows one to simply define temporal abstractions by using intu-itive metaphors, and to specify temporal intervals related to abstractions by using different temporalgranularities. The physician can interact with the designed and implemented tool by point-and-clickselections, and can visually compose queries involving several temporal abstractions. The evaluation ofthe proposed granularity-related metaphors consisted in two parts: (i) solving 30 interpretation exer-cises by choosing the correct interpretation of a given screenshot representing a possible scenario, and(ii) solving a complex exercise, by visually specifying through the interface a scenario described onlyin natural language. The exercises were done by 13 subjects. The percentage of correct answers to theinterpretation exercises were slightly different with respect to the considered metaphors (54.4 – stripedwall, 73.3 – plastered wall, 61 – brick wall, and 61 – no wall), but post hoc statistical analysis on means
confirmed that differences were not statistically significant. The result of the user’s satisfaction ques-tionnaire related to the evaluation of the proposed granularity-related metaphors ratified that there areno preferences for one of them. The evaluation of the proposed logical notation consisted in two parts:
tion exercises provided by a screenshot representing a possible scenario and
(i) solving five interpreta by three different possible interpretations, of which only one was correct, and (ii) solving five exercises,by visually defining through the interface a scenario described only in natural language. Exercises had
∗ Corresponding author.E-mail addresses: [email protected] (C. Combi), [email protected] (B. Oliboni).

933-3657/$ – see front matter © 2011 Elsevier B.V. All rights reserved.oi:10.1016/j.artmed.2011.10.004

dx.doi.org/10.1016/j.artmed.2011.10.004

http://www.sciencedirect.com/science/journal/09333657

http://www.elsevier.com/locate/aiim

mailto:[email protected]

mailto:[email protected]

dx.doi.org/10.1016/j.artmed.2011.10.004

76 C. Combi, B. Oliboni / Artificial Intelligence in Medicine 54 (2012) 75– 101

an increasing difficulty. The evaluation involved a total of 31 subjects. Results related to this evaluationphase confirmed us about the soundness of the proposed solution even in comparison with a well knownproposal based on a tabular query form (the only significant difference is that our proposal requiresmore time for the training phase: 21 min versus 14 min). In this work we have considered the issueof visually composing and querying temporal clinical patient data. In this context we have proposed avisual framework for the specification of consistent temporal abstractions with different granularitiesand for the visual composition of different temporal abstractions to build (possibly) complex queries onclinical databases. A new algorithm has been proposed to check the consistency of the specified granularabstraction. From the evaluation of the proposed metaphors and interfaces and from the comparison ofthe visual query language with a well known visual method for boolean queries, the soundness of theoverall system has been confirmed; moreover, pros and cons and possible improvements emerged fromthe comparison of different visual metaphors and solutions.

1

ecrsaittapwem

aplsOfpaoNfatioi“irb

iunielmsfsifa

. Introduction

Visualizing and exploring patient clinical information is a rel-vant need in the medical domain [1]. Focusing on temporallinical information [2], we may identify two main and intertwinedesearch directions: (i) the visualization of patient information,uch as histories, therapeutic plans, clinical parameters [3–6],nd (ii) the visual exploration and query of temporal clinicalnformation [7–10]. Roughly speaking, the first research direc-ion is more focused on the specification of metaphors andools allowing the visual display of several time-oriented datand the visual identification and highlighting of specific tem-oral clinical data, while the second one supports the userith graphical and visual solutions allowing the interactive

xploration and query of huge amounts of temporal clinical infor-ation.Temporal clinical information may be represented through data

t different levels of abstraction: from temporal raw data to tem-oral abstractions [11]: temporal abstractions may consist of high

evel information derived from several time-stamped raw data,uch as “systolic blood pressure increase in follow-up visits fromctober 23, 2009 to November 30, 2009” or “heart rate decrease

rom December 1 to December 15, 2009”; further (composite) tem-oral abstractions may be obtained from other previously definedbstractions, as, for example “hypoglicemia overlapping a decreasef systolic blood pressure in the period from November 25 toovember 27, 2009”. Previous attempts at providing user interfaces

or the definition and querying of composite abstractions [12,13]ssumed a skilled user, able to manage all the technical details ofhe definition process, and used visualization only in a very lim-ted way. Temporal abstractions could also be based on propertiesf abstractions and raw data involving different time granularities,.e., different time units. As an example, the previous abstractionhypoglicemia overlapping a decrease of systolic blood pressuren the period from November 25 to November 27, 2009” could beefined by specifying that “hypoglicemia and decrease of systoliclood pressure end in November, 2009”.

Going back to human–computer interaction aspects, visualnterfaces gained an increasing interest in these last years. Indeed,sers’ requirements and tools evolved in a fast way: currently, bothovice and expert users give priority to systems that require min-

mal time for training, and assist in avoiding syntactical mistakes,ven when they require systems that provide highly expressiveanguages and high speed of execution. It is worth noting that

ouse-based interactions and visual interfaces are nowadays thetandard way of interacting with computers, both for novice andor experts users: both complex tools for experts users and more
imple tools for novice users are based on easy-to-use graphicalnterfaces. In summary, visual systems have to be considered bothor raw and abstract temporal clinical information and for naivend skilled users.
© 2011 Elsevier B.V. All rights reserved.

According to this scenario, the main goal of this work is topropose a framework for the visual specification and query of con-sistent multi-granular clinical temporal abstractions. The specificissues we will face in this paper with original solutions are:

1. using and extending a well known visual formalism [7] withquantitative durations, granularities and other features, to sup-port the visual composition of consistent temporal abstractions,

2. defining a visual notation for the logical definition of queriesinvolving different temporal abstractions, and

3. providing a first design, implementation, and evaluation of thevisual framework and apply it to the real world clinical domainof hemodialysis.

In particular, we propose a visual language for defining tem-poral clinical abstractions, based on a suitable extension of thePaint Strips metaphor proposed in [7]. Paint Strips metaphor wasaimed mainly at representing qualitative relations between inter-vals, through the use of a physical metaphor using paint strips,rollers, weights, and so on. The main original extension we proposehere is the capability of defining abstractions involving differ-ent granularities (hereinafter, temporal granular abstractions) andof representing metric constraints between endpoints. Granulari-ties are represented through either different metaphor objects orgraphical icons. The introduction of different granularities whendefining a temporal abstraction may introduce some kind of incon-sistency in the specified temporal scenario: as an example, theabstraction “hypoglicemia overlapping a decrease of systolic bloodpressure in the period from November 25 to November 27, 2009”cannot be refined by specifying that “hypoglicemia must endone month before the decrease of systolic blood pressure starts”because overlapping requires that hypoglicemia end after the startof the decrease of systolic blood pressure. To this regard, we pro-pose here a new algorithm that prevents the user from the visualspecification of inconsistent temporal abstractions. To the best ofour knowledge, both the visual specification of abstractions involv-ing several granularities and the consistency checking of specifiedabstractions are original contributions of this paper.

A further original contribution of this paper is related to the useof abstractions to query a clinical database. Indeed, after the defi-nition of several temporal abstractions, the physician may need toquery a clinical abstraction database through the (logical) composi-tion of the previously defined abstractions, possibly sharing somecomponent abstractions: for example, a user could be interestedin visually identifying those patients who had both “hypoglicemiaoverlapping a decrease of systolic blood pressure in follow-upvisits” and “systolic blood pressure decrease in follow-up visits
followed by a stationary state of heart rate”, with the constraintthat the two composite abstractions must share the componentabstraction “systolic blood pressure decrease in follow-up visits”(in other words, patients having both the considered abstractions,

lligen

brtiree

iisbc[atfteoh

sivfitoidfidc

2

tamb

2

gcs

cfftr

Dsaa

gttf

C. Combi, B. Oliboni / Artificial Inte

ut referring to disjoint sets of basic abstractions, have to be dis-egarded). The proposed framework supports the physician in thisask through the use of a new visual language based on a metaphornspired by a set-oriented notation, where suitable objects cor-espond to the classical logical connectives AND, OR, and NOT,ven considering their proper nesting to compose complex booleanxpressions.

Finally, the proposed visual framework has been designed,mplemented, and evaluated in their parts: different metaphors andcons for granularities have been considered and evaluated withubjects in the healthcare domain; the visual language proposed foroolean abstraction-based queries has been evaluated as well andompared with a well known visual formalism for boolean queries14]. Furthermore, the visual framework has been adopted for thenalysis of temporal data coming from hemodialysis sessions. Inhis clinical domain several timestamped raw data are acquiredrom patients and from hemodialyzers: suitable temporal abstrac-ions are then derived to represent at higher level the temporalvolution of sessions. In the context of the decision-oriented taskf evaluating the quality of the provided care, the visual frameworkas been applied to identify issues in hemodialytic treatments.

The structure of the paper is as follows: Section 2 providesome basics on temporal granularity, temporal abstraction of clin-cal data, and reports about the main proposals in the context ofisualization of clinical data and in the context of visual systemsor boolean queries. Section 3 describes the metaphors and the usernterface we designed both for the definition of temporal abstrac-ions and for the specification of queries. A detailed descriptionf the algorithm checking the consistency of abstractions involv-ng multiple granularities is provided too. Section 4 discusses theesign and implementation aspects of a prototype for the visualramework, provides the main results of a first evaluation of thentroduced metaphors, and shows a real world application of theeveloped prototype for analyzing hemodialysis data. Section 6oncludes the paper with some final remarks.

. Background and related work

In this section, we introduce some background and contribu-ions related to the concept of temporal granularity, to temporalbstractions on clinical data, to the visualization of clinical infor-ation (either temporal or not), and to the visual specification of

oolean queries.

.1. Temporal granularity

A time granularity introduces a partition of a time domain inroups of indivisible elements, called granules. Time granularitiesan be modeled according to the following definition [15], whichpecializes the more general definition of granularity given in [16].

Let T be the time domain and I be the domain of a granularity G,alled index set. Informally, a granularity is a special kind of mappingrom the index set to subsets of the time domain. In the followingormal definition [15], we assume that both the index set I andhe time domain T are the linear discrete domain N ordered by a ≤elationship (denoted as (N, ≤)).

efinition 1. A time granularity is a mapping G from integers toubsets of a totally ordered time domain (N, ≤) such that: (i) if i < jnd G(i), G(j) /= , then, for all n ∈ G(i) and m ∈ G(j), n < m; (ii) if i < k < jnd G(i), G(j) /= , then G(k) /= .

The first part of the above definition states that granules in a
ranularity do not overlap and that their order is the same as theirime domain order. The second part of the above definition stateshat the subset of the index set that maps to nonempty granulesorms an initial segment.
ce in Medicine 54 (2012) 75– 101 77

Typical granularities are the calendric ones, i.e., seconds, minutes,hours, days months, and years. In general, granularities form a latticewith respect to the relation finer than [17]: informally, a granularityis finer than another granularity if any granule of the first one iscontained in some granule of the second granularity. For example,granularity days is finer than granularity months, while granularityweeks is not comparable with granularity months, as weeks is notfiner than months and months is not finer than weeks.

Granularities may also be adopted when we need to specifydurations, i.e., distances between two time points: to this regard, in[18] a (calendric) granularity is defined as a unit of measurement forspans of time. For example, the granularity of days (day) stands fora duration of 24 h. More generally, a granularity is a special kind of,possibly varying, duration that can be used as a unit of time. Suchgranularities may thus be used as a unit of measure for expressingdurations and also for specifying time points; in such a case, gran-ularities are used for expressing the distance of a time point froma reference time point, chosen as origin of the time axis.

In the following, according to both the described approaches,we will consider only sets of granularities that have a total orderwith respect to the finer than relation. More particularly, we willconsider in this paper the set of calendric granularities {seconds,minutes, hours, days, months, years}, when we need to specify timepoints. Their “unanchored” counterparts will be used for represent-ing durations with different time units.

Granularities have been deeply considered in the managementof clinical data; indeed, on one hand, clinical data often requireto be stored and represented at different temporal granularities,while, on the other hand, several and different granularities couldbe needed when querying huge amounts of patients’ data. To thisregard, several proposals have been made related to the definitionof data models and query languages dealing with different tem-poral granularities and clinical data: they allow users to specifycomplex temporal features possibly involving different temporalgranularities both in storing and querying clinical data [2,19–21].

2.2. Temporal abstractions on clinical data

Temporal abstraction provides a concise and high-level descrip-tion of a collection of time-stamped raw data [11]. It plays a centralrole in supplying care providers with data at a suitable level forsupporting decision making [11–13,22].

In [22] a general framework for abstraction called Knowledge-Based Temporal-Abstraction (KBTA) is proposed. On the basis ofa theoretical model for time and for propositions that hold overtime, KBTA decomposes the temporal abstraction task into fivecomputational subtasks, suitably solved by corresponding formalmechanisms, such as temporal inference, temporal interpolation,and temporal pattern matching [2]. The output of these mecha-nisms includes the basic abstractions of type state, gradient, rate (e.g.,LOW, DECREASING, and FAST are some abstractions applicable toseveral clinical timestamped parameters), and composite abstrac-tions, named also patterns, defined in terms of basic/compositeabstractions connected through specific interval-based temporalrelations: for example, basic abstractions HYPERGLYCEMIA andGLYCOSURIA ABSENT may be defined on the basis on several times-tamped measurements of glucose level over a given threshold andof the absence of glucose in the urine, respectively; the patternHYPERGLYCEMIA overlaps GLYCOSURIA ABSENT is the compos-ite abstraction suitably defined in monitoring diabetic patients [11],that holds when, for a given patient, there is an interval wherehyperglycemia holds overlapping an interval where glycosuria does
not hold. The KBTA framework underlines the need of representingthe knowledge required for abstraction of time-oriented clinicaldata, and facilitates its acquisition, maintenance, reuse, and shar-ing. The KBTA method has been implemented by the RÉSUMÉ

7 elligen

sbgdsst

2

ptscmh(gtooampckc

mdamottI

dfipapsgadisdiwecrihsafa

roea

8 C. Combi, B. Oliboni / Artificial Int

ystem and evaluated in several clinical domains, such as guideline-ased care of oncology and AIDS patients, monitoring of children’srowth, and management of patients who have insulin-dependentiabetes [11]. Moreover, the KBTA framework has been the basis ofeveral further research efforts dealing with temporal informationystems able to manage and use temporal abstractions of medicalime-oriented data [2].

.3. Information visualization of (temporal) clinical data

The amount of clinical data available and useful for the sup-ort of clinical decision-based tasks is increasing every day: this ishe result of technology advancements in computer performance,torage capacity, networking and of the corresponding increasingapability of producing medical data [1]. As the number of accu-ulating data items increases over time, it becomes evident that

uman cognitive and perceptual capabilities limit the quantity ofoften historical) information a user can observe and manage at aiven time [2]. In this situation, clinical users might not be ableo properly assess and analyze the implications of large amountsf time-oriented data and might be overwhelmed by them, with-ut suitable computer-based solutions for presenting clinical datand for interacting with them. Answers to these issues are theain theme of Information Visualization (IV). According to the pro-

osed definitions of IV, we could say that IV can be defined as theomputer-assisted process of transforming data, information, andnowledge into visual form making use of humans’ natural visualapabilities [2].

Focusing on the visualization of temporal information, in theost cited and influential IV taxonomy, proposed in [23], temporal

ata is one of the seven data types for the items to be displayednd plays an important role in the development of medical infor-ation systems, given the emphasis on the importance of time and

n the need for applying domain-specific temporal knowledge: inhis direction, several specific approaches for the visualization ofemporal clinical data and knowledge have been proposed in theV research area [1,2,6,24–30].

The visualization of patient histories, based on time-stamped rawata, has been considered since the early 1990s [25]: one of therst systems proposed was Time Line Browser [25] which visualizedoint-based events (such as the measure of a clinical parameter)nd intervals with duration (such as continuative situation of theatient) on a timeline. In this direction, in [24] the authors con-ider the issue of visualizing temporal clinical data given at differentranularities or with uncertainty when visually depicting the over-ll patient history. In this approach, intervals are represented withifferently colored, encapsulated bars on the time axis, to explic-

tly represent both the kind of clinical information (visits, therapies,ymptoms) and the possible temporal extension of the consideredata [2,24]. As temporal interval-based relationships between clin-

cal data given at different granularities cannot be always statedith certainty and visually observed in a simple way, in [5] a differ-

nt visualization tool is proposed, where temporal interval-basedlinical data are represented through nodes composed by sectorsepresenting the start, the end, and the duration of the considerednterval, respectively. Both the proposed visualization approachesave been integrated into the object-oriented web-based KHOSPADystem (Knocking at the Hospital for PAtient Data) [2,31]. In [32] theuthors adopt the idea and extended it into the PlanningLines visualormalism which is specifically targeted to the needs of planningnd controlling tasks.

A more general approach for visualization of patient histo-
ies is proposed in Lifelines [6], where facts are displayed as linesn a graphic time axis, according to their temporal location andxtension; color and thickness are used to represent categoriesnd significance of the represented information. Lifelines visually
ce in Medicine 54 (2012) 75– 101

summarizes the relevant events and intervals, organized into dif-ferent screen areas, each one related to different parts of theconsidered medical record (such as visit-related parameters, symp-toms, ongoing therapies, . . .): the user can interact with the visualrepresentation of the patient history by selecting items of interestand getting details on demand (e.g., a lab report) or by perform-ing a zoom (either in or out) of the examined range of time, witha dynamic rearrangement of the displayed data. Lifelines [6] is oneof the best known visualization environments and has been con-sidered and improved in further research efforts. As an example, in[3] the authors propose an interactive visualization approach pro-viding multiple simultaneous views of patient data. Their approachallows the interactive integration of different visualization meth-ods, and represents a novel way to combine, relate, and analyzedifferent kinds of medical data and information that are usually sep-arated. The views are based on well-known visualization methodssuch as LifeLines [6,33].

In the management of massive amounts of clinical data, filteringand extracting relevant information is more difficult and requiresthe mapping of high-dimensional data to lower-dimensional visualrepresentation. In [34], the authors present an automated approachfor generating potentially candidate representations that best showphenomena contained in the high-dimensional data, and support-ing the user in the exploration process.

In the context of explorations of time-varying data, an inter-esting issue is related to the visualization of data across temporalscales. In [35], the authors propose a method for exploring dataat different temporal resolution to discover data based on time-varying trends, and classify them with respect to multiscaletemporal activities. In [27], the authors provide three different tech-niques for visualizing and interpreting temporal information. Eachproposed technique is based on a different way to interpret tem-poral data, and is complementary with respect to the other ones.In [36], the authors deal with multi-focus interaction, i.e., the capa-bility to simultaneously view different parts of a data set whileretaining context and distance awareness. The proposed approachis based on stacks of strips, where each strip represents data andeach subsequent stack represents a higher zoom level.

Furthermore, visualization of medical information has todeal with the need of representing and analyzing data byconsidering a number of highly structured, time-oriented quali-tative/quantitative parameters. This aspect has been extensivelyfaced: for example, in [37], the authors consider psychotherapeu-tic data and propose an interactive visualization method to analyzethis data by introducing the possibility of observing new interde-pendencies between various kinds of parameters. This is obtainedby a system using different metaphors, where patients are, forexample, iron spheres attracted by magnets, standing for ques-tions; the movement of spheres depending on the attractive powerof different magnets represents how different patients behave withrespect to the proposed questions.

Another interesting problem often appearing in the contextof medical temporal data is to augment existing standard repre-sentations used by clinicians, by the visualization of additionalinformation. For example, in [38] the authors consider the dis-play methods typical of anesthesia monitors, specifically devotedto the visualization of quantitative patient parameters, and proposeseveral extensions that convey additional information concerningcertainty and vagueness of the displayed derived trend data: forexample, color coding is used to give an indication of different levelsof certainty by means of various shades of the same color.

Being impossible to describe here all the different proposals
dealing with the visual representation of temporal clinical data,as different application domains, data, goals, clinical users, andtechniques have been considered, it is important here to considersome recent contributions trying to provide some common criteria

lligen

aa[orsawqfreaaustt(tmclcteaf

2d

tstdpatdctrttevooIawh

qatsrttigpdi


llowing some kind of classification/comparison for the propos-ls dealing with temporal clinical data [2,39,40]; in particular, in39] some categorization criteria are proposed for visual meth-ds devoted to the analysis of time-oriented data: time, data, andepresentation, are the three dimensions, further decomposed ineveral sub-aspects, used for categorization. In [40] different char-cteristics of the visualized time dimensions are discussed togetherith the focus either on quantitative data (e.g., time series) or on

ualitative data; moreover, several example of visualization toolsor temporal data are given. A second aspect discussed in [40] iselated to the need of visual tools for the analysis of time ori-nted data: temporal data abstraction, with principal componentnalysis and clustering, is a method allowing the analysis of hugemount of time-oriented data. The third aspect is related to theser, who has to be active part of the visualization process, throughuitable interaction tools and techniques. Finally, in [2] two majorypes of visualization are distinguished: (i) visualization of theime-oriented data or of the concepts derivable from these data;ii) visualization of knowledge regarding time-oriented informa-ion. Four dimensions of visualization and interactive exploration of

edical time-oriented data or knowledge are identified: (i) subjectardinality (individual versus multiple patients), (ii) abstractionevel (raw data versus abstract concepts versus knowledge), (iii)oncept cardinality (one concept versus multiple concepts), (iv)emporal granularity (single versus multiple time units). Somexamples of different visualization systems are then presented,ccording to the given taxonomy and to the requirements discussedor the medical domain [2].

.3.1. Visual exploration and query systems for clinical temporalata

According to recent contributions, it is becoming evident thathe visual exploration and mining of temporal data is relevant ineveral domains and in particular in medicine: to this regard, visualools for the exploration of huge amount of temporal raw anderived data have been proposed for several medical domains. IPBC,roposed in [8], is a tool allowing the interactive, visual explorationnd mining of raw time series data: it is based on a 3D represen-ation of data and has been applied to the analysis of hemodialysisata. In [13,41] the authors propose different tools supporting thelinical user to explore patient data according to different perspec-ives, such as the visual knowledge-based exploration of temporalaw data and abstractions of single patients, and, more recently,he visual exploration and query of temporal raw data and abstrac-ions of multiple patients [9,42]. Similarly, Lifelines2 is the naturalvolution of the Lifeline approach to consider issues related to theisualization of data from multiple patients, to the visual discoveryf temporal patterns, and to the derivation of temporal summariesf multiple patient data at different temporal granularities [10,43].n [44], the authors discuss several solutions for interactive visu-lization and exploration of clinical data in the context of ICUs,here physicians and nurses need to monitor a huge amount ofigh-dimensional, time-oriented and task-oriented data.

The difference between interactive visual exploration and visualuerying of data is sometime fuzzy to distinguish: indeed, the inter-ctive selection and focus on relevant temporal data correspondo queries on the overall temporal data set. Here, we would con-ider visual query systems, i.e., those systems able to support theetrieval of clinical data through the visual composition of selec-ion conditions. It is worth noting that, even with this meaning,he existing proposals range from visual query systems provid-ng some “filtering” tools to systems supporting the user with
raphical/visual tools in building a more general query on the tem-oral clinical database. A system supporting the specification ofynamic visual queries on time series data is proposed in [45]: it
s based on the selection of time series data through timeboxes,

ce in Medicine 54 (2012) 75– 101 79

i.e., rectangular query regions drawn directly on a two-dimensionaldisplay of time series data. Moving to more abstract and gen-eral temporal clinical data, PatternFinder [46] allows the user tobuild temporal patterns through visual and graphical tools: thespecified temporal patterns will be used to query temporal clin-ical histories. PatternFinder manages explicitly the integration ofvisualization methods for query definition and for presentationof query results. In [9,42], the authors propose the VISITORS sys-tem, providing tools for the knowledge-based visualization andexploration of multiple patients’ data and abstractions; VISITORSincludes a graphical module allowing the specification of therelevant patients, time intervals and values. An ontology-basedtemporal-aggregation specification language underlies the graph-ical expression-specification module: it enables the specificationof three types of expression: Select Patients, Select Time Intervals,and Get Patients Data.

A very important issue to consider when representing temporalinformation is related to the visualization of temporal relation-ships between data varying over time. In [7], the authors providesome solutions to the problem of visualizing temporal intervals andtheir qualitative relations when querying databases containing sev-eral (abstraction-based) histories. They propose visual vocabulariesbased on real-world metaphoric objects, such as strips, springs,elastic bands, weights, and wires; suitable methods automaticallymap visual queries into SQL queries. The proposed solutions wereevaluated with two proper user studies: the first one focused ondetermining the metaphors more frequently perceived and under-stood in a correct way and was based on a questionnaire; thesecond study considered the two solutions which scored betterin the first phase and studied them with a further experiment,based also on user interfaces implementing the two metaphors.The visual vocabulary which provided the best results has beenadopted in a medical system for visual querying clinical tempo-ral histories [7]. In [47], the authors define a temporal categoricalsimilarity measure based on the concept of aligning records bysentinel events. The proposed measure is a combination of time dif-ferences between events, and number of missing and extra events,and is used in an interactive search and visualization tool for tem-poral categorical records. The information visualization systemproposed in [47] has been generalized into an information-seekingprocess model for multiple electronic health record (EHR) sys-tems in [48]. In [49], the authors present a novel algorithm (toimprove existing feature of the tool proposed in [47]) for search-ing for temporal patterns of events in patient histories. In [50],the authors deal with methods for supporting users in discover-ing temporal pattern and their relationships. In particular, theypropose a pixel-based visualization of time-oriented data by over-laying several time granularities in one visualization. In [51], a newanalysis method for visualizing multivariate time-varying data setsis proposed. This approach allows the user to identify and visualizetemporal trends, and their spatiotemporal relationships. Displayingthe relationships provides the opportunity of identifying correla-tions and causal effects among multiple variables and searching forsalient properties of the data set. In [52], the authors try to over-come limits of traditional temporal displays to better support theuser understanding and reasoning about critical relationships. In[53], the authors propose an interactive visual query environmentusing a comic strip metaphor to allow the user to easily defineand locate complex temporal pattern. In particular, the proposedapproach helps users to obtain answer regarding temporal relation-ships about events or time series data, and to incrementally perturba question having immediately the answer changes. The proposal
uses comic strip metaphor for representing temporal queries orpattern, where character or actors in the comic strip are used torepresent events, and the visual language of comic strips is used torepresent the temporal relationships among the events.

8 elligen

pvbtcaotbr

2

abcgutrPPbTaeo

tafiIidatttaK[cpqrOebaimr“wAjfff

itn


Finally, further proposals [54,55] consider queries on com-lex information such as multimedia data, i.e., images, sounds,ideos and long text data. In particular, they query clinical datay means of the query language Mquery [56], which is an evolu-ion of the PICQUERY+language [57]. Mquery allows the user toompose the query by means of a sequence of “cut-and-paste” oper-tions. Some graphical and visual tools allows the representationf patient-oriented multimedia data [58], as well as the support tohe composition of complex queries, where query results need toe displayed according to the multimedia and temporal features ofetrieved data [59].

.4. Visual systems for boolean queries

The composition of boolean queries have been acknowledgeds a difficult task, where several kinds of mistakes could be doney even expert users, such as the improper nesting of AND and ORonnectives, or the use of AND for OR and vice versa. Thus, bothraphical and visual interfaces have been proposed to support theser in building boolean queries. Some kind of graphical supporto the composition of boolean queries is often provided by the cur-ent form-based user interfaces: as an example, we mention hereubMed Assistant [60], a system supporting boolean queries on theubMed database. Queries are represented through trees, whereoolean operators are internal nodes and keywords are leaf nodes.he hierarchical structure of a tree represents intuitively oper-tor precedence (i.e., parentheses of the corresponding booleanxpression). The structuring of a query is done as a drag-and-dropperation on a tree.

Moving to a visual support of boolean queries, where connec-ives AND, OR, and NOT are not explicitly expressed, some proposalsre based on filter/flow-based metaphors and use sequential flowsor conjunction, parallel flows for disjunction, while the negations sometimes represented through “negative” (i.e., inverse) colors.nfoCrystal and VQuery [61,62] are two examples of systems adopt-ng flow-based metaphors. Other proposals extend/adapt Venniagrams: in a Venn diagram each query term is associated with

ring or circle, with intersections of circles indicating conjunc-ions of terms. In such kind of systems the user has to identifyhe regions of the various circles (or regions) corresponding tohe considered query [63,64]. Another proposal, named KMVQL,llows users to specify boolean queries by selecting cells in aarnaugh map, eliminating the necessity of using logic operators

65]: the Karnaugh map provides a table where different (logical)ombinations of terms are visually displayed. In [66], the authorsropose a predicate tree, to visually represent complex booleanueries: nodes represent simple conditions, edges connect nodesequired to hold together (logical AND), different paths stand forR-connected conditions. The predicate tree metaphor has beenvaluated for the task of composing complex queries for web-ased search engines. Finally, in [14], a tabular query form thatvoids the need to explicitly use logical connectives is proposed;t clearly distinguishes between conjunction and disjunction, and

akes grouping more explicit. The proposed, graphical languageepresents queries through a card metaphor. Each card is namedmatch form”: simple terms (possibly negated by prefacing a termith the NOT operator) are placed into slots, one term per slot.ll of the terms on a single form implicitly are evaluated as a con-

unction. Disjunction is specified by including an additional matchorm adjacent to the first one. In other words, this tabular queryorm supports boolean queries expressed in disjunctive normalorm (possibly using a negation also for a form).

In general, user studies on visual/graphical interfaces support-ng boolean queries show that the proposed interfaces eliminatehe need of explicitly specifying logical connectives, rely on recog-ition, and maximize concreteness. As an example, comparing the

ce in Medicine 54 (2012) 75– 101

tabular language with textual boolean expressions, the authorsobserve that untrained users perform better when they expresstheir queries in the tabular language, and about equally well wheninterpreting queries written in either language [14].

Even though the provided experimental user studies show inter-esting results of different proposals in comparison with the textualexpression of boolean queries, it is emerging that such user stud-ies are difficult to be reproduced, as the number of uncontrolledaspects is extremely high. As an example, the number of the con-sidered queries in the user studies is usually low and in general oflow complexity [67]. Moreover, user studies could provide contra-dicting results. For example, in [68] four mechanisms for composingboolean queries have been evaluated: boolean expressions (BE: tex-tual), an if-then-else procedural language (ITE: textual), Filter/Flow(FF: graphical), and Venn diagrams (VD: graphical). The task wasquery generation, and the accuracy together with the time takento compose a query was measured. The obtained results aboutease-of-use ordered first BE, then VD, F/F, and ITE: it contradictsa previous study, where a flow-based visual language was found tooutperform SQL [67].

3. A visual query language for temporal abstractions

In designing a system for the visual definition of temporal clin-ical abstractions, we followed the KBTA framework [11]. Withinthis framework, we focused on the definition of composite abstrac-tions, called patterns in KBTA terminology. In the following, weassume that component (basic or composite) abstractions (here-inafter, concisely referred to as components) have been alreadysuitably defined. Throughout the paper, we will use as examplessome components representing simple symptoms, such as fever,chest pain, cough. As for granularities corresponding to suitablepartitions on the time domain [16], without lack of generality,we will consider the calendric granularities of years, months, days,hours, and minutes, defined on the time domain of seconds.

With respect to the three dimensions for the classification oftemporal visual formalisms proposed by [39], our system can bedescribed as follows: as for the time dimension, temporal primitivesrefer to time intervals and represent relations among time inter-vals, while the structure of time is linear. As for the data dimension,the frame of reference is abstract (i.e., the data we refer to are notcollected in a spatial context) and different variables are handledsimultaneously (multivariate data): the approach is aimed at dataabstraction. Finally, for the representation dimension, the visual rep-resentations are static and the presentation space is 2D. To conceivea visual query language, we chose suitable metaphors and graph-ical notations, and designed a suitable graphical interface. Beforeintroducing the proposed metaphors and notations (see Sections3.1 and 3.2), in the following, we summarize the overall features ofthe visual interface.

The proposed visual query language mainly consider two dif-ferent tasks: (i) that of visually defining a composite temporalabstraction, by specifying its component abstractions and theirtemporal relations, with both qualitative and quantitative aspectsand possibly given at different granularities, and (ii) that of query-ing a temporal clinical database by visually composing booleanexpressions involving several temporal abstractions. According tothese two tasks, the graphical interface is organized into twomain parts, as depicted in Fig. 1: the upper part is the temporalarea and allows the user to visually define composite temporalabstractions. In the temporal area we may identify several sub-
parts: at the top of the area, different panel labels correspondto the composed abstractions (A.1 in Fig. 1); the right part ofthe temporal area contain the names of the component abstrac-tions (A.2); the main part of the temporal area contains the visual

C. Combi, B. Oliboni / Artificial Intelligence in Medicine 54 (2012) 75– 101 81

osed v

rrabtci(aipd(suqfci

n

Fig. 1. The prop

epresentation of the considered component abstractions and theirelative positions (A.4) managed through a suitable toolbar (A.5)nd possibly enriched by quantitative constraints for the distanceetween abstraction intervals and for the considered time frame;he bottom part of the temporal area contains the name of theonsidered composite abstraction and its interval of validity, spec-fied on the base of interval bounds of component abstractionsA.3). The lower part of the user interface is the logical area andllows the visual specification of possibly complex boolean queriesnvolving the composite abstractions specified and displayed in theanels of the temporal area. The main part of the logical area isevoted to the visual composition of the required boolean queryB.3), that is also represented through the corresponding expres-ion at the top of the panel (B.2); a suitable toolbar allows theser to select the required (visual) logical connective to build theuery (B.4). The right part of the logical area allows the user toocus (if it is needed) on composite abstractions sharing some
omponent abstraction (B.1), as we will discuss in the follow-ng.
To summarize, we considered some general criteria in the defi-ition of the overall visual interface:

isual interface.

1. use of different colors, selected by users, to highlight differentabstractions;

2. visual distinction of temporal and logical aspects in two differ-ent graphical windows;

3. clear connection among different graphical objects referring tothe same concept (i.e., abstraction);

4. point-and-click selection of the main components and of thevarious graphical operators (the abstractions and the queriesmay be interactively defined without resorting to the key-board);

5. visual support to the specification of complex temporal abstrac-tions;

6. visual support to the composition of boolean queries involvingdifferent temporal abstractions;

7. use of simple metaphors related to the physical world;8. easy setting of relative temporal positions among components

abstractions;
9. definition of a time frame as a temporal context, when evalu-
ating the given abstraction;10. system-supported definition of the temporal interval related to

the abstraction, on the basis of the intervals of components;

8 elligen

1

3

taraFt

3g

ovsas

w

•

•

•

aiowahh

sceb


1. logical connection of different abstractions through visual con-nectives standing for the classical ones (AND, OR, NOT).

.1. Defining temporal abstractions: the temporal area

In this section, we discuss the basic concepts underlying theemporal area: we first introduce the metaphors for definingbstractions, and detail how we display time intervals, temporalelations, and granularities. Moreover, we introduce the tools avail-ble for the visual composition of granular temporal abstractions.inally, we discuss the issues related to checking the consistency ofhe specified abstraction.

.1.1. Temporal visualization: intervals, temporal relations, andranularities

For defining abstractions, we adopted and modified paint strips,ne of the visual vocabularies proposed in [7], and extended it toisually represent granularities. A paint strip on the screen repre-ents the temporal interval of an abstraction; temporal relationsmong abstractions can be specified by suitably locating the corre-ponding paint strips.

The temporal location of paint strips can be specified in differentays:

paint strips can be represented plainly without any attachedobject (Fig. 2a). In this case, they specify intervals bounds thathave a precisely set position with respect to other intervals. Thecommon sense reasoning motivating this choice is that the boundof a paint strip cannot move by itself.Alternatively, any bound of a strip can be attached to a paint roller,connected to a weight by means of a wire (as shown in Fig. 2b).In this case, the bound of the interval can take different positionson the time axis: the roller can extend the given bound of thepaint strip up to the nail, which stops the roller. It means thatthe considered interval bound may have different temporal rela-tions with respect to other interval bounds. To this regard, themetaphor originally proposed in [7] used a brick (instead of thenail) to stop the roller. We replaced the brick with a nail to makethe metaphor more precisely understood: indeed, we discoveredthat the brick was a little bit confusing for users with regards tothe point where the interval bound stops.Finally, a weight can be connected to more than one roller simul-taneously, to represent intervals bounds which can move whilekeeping their relative position. This is obtained through the rep-resentation of a bar that allows the user to represent the (fixed)relative positions of two or more intervals. This way, the paintroller is related to all the interval bounds combined by the bar, andis graphically connected only to the more external interval: whenthe bar combines start bounds, the roller will be connected to theleft most interval, while when the bar combines end bounds, theroller will be connected to the right most interval, as illustrated,for the left case, by Fig. 2c.

The possibility of specifying different granularities within anbstraction is an important issue. By using different granularitiesn defining composite abstractions, we introduce the capabilityf comparing the temporal location of component abstractionsith respect to different time units. For example, the component

bstractions headache and fever could be related by requiring thateadache is after fever on the basic timeline of seconds, and thateadache and fever start at the same day.

To introduce the concept of granularity in paint strips, a trivial
olution could be to provide paint strips with different textures orolors or shapes, depending on the represented granularity. How-ver, this approach could be misleading, because granularity woulde related to each paint strip, instead of the comparison among
ce in Medicine 54 (2012) 75– 101

component abstractions. For this reason, we did not choose visualfeatures of paint strips, and, instead, we considered different possi-bilities to introduce information about granularity by using eitheronly labels or different graphical backgrounds or icons, and per-formed a first user-based evaluation of the different solutions.

Our first proposal uses only labels: it is possible to specify thename of the considered granularity in the label of the tab contain-ing the component abstractions. For example in Fig. 1 the label ofthe tab is“DEFAULT” and represents the default granularity, i.e., thegranularity seconds.

Our second proposal tries to strengthen the representation ofthe considered granularity through labels, by using also differ-ent graphical backgrounds of the temporal composition area. Thischoice has to be consistent with the Paint Strips metaphor: the ideais painting the strips on a wall built with different materials. Morespecifically, the considered methaphors for backgrounds are: thebrick wall, the striped wall, and the plastered wall.

In the first case, the background is represented as a brick wall,where brick size is related to the represented granularity. The back-ground for the years granularity will thus be a brick wall with hugebricks, while the background for minutes will be a brick wall withtiny bricks. Fig. 3 shows examples where the visual representationof granularities is performed through tab labels and brick-basedbackgrounds.

By introducing this solution, we suppose that bricks do not carryany information about temporal references for starting and endinginstants of granules, and moreover, being bricks of different layersnot aligned, the only meaning of the background is that of repre-senting the fact that constraints are given in a specific context, i.e.,that of the given granularity.

In the second case, we suppose to give the idea of granularity byusing only horizontal lines, and thus we introduce the striped wall.In this solution, a striped wall with wide stripes represents the yearsgranularity, while a striped wall with narrow stripes represents theminutes granularity. Fig. 4 shows examples of backgrounds relatedto different granularities represented by using striped walls.

In the last case, we propose as background a plastered wall,where graines of the plaster are related to the represented gran-ularity. The background for the years granularity will thus be acoarse-grained wall, while the background for minutes will be afine-grained wall. Fig. 5 shows examples of backgrounds relatedto different granularities represented by using different grainedplasters.

The last solution we considered tries to strengthen the tab labelsby the use of explicit graphical icons for representing differentgranularities. In this case, the granularity is represented by usingan icon resembling the usual calendar-like graphical representa-tions, reported in the right part of the temporal composition area,as shown in Fig. 6: the calendar-like icon has the title correspondingto the chosen granularities, while the cells inside represent the finergranules composing a generic granule of the considered granular-ity. As an example, Fig. 6 depicts the icon corresponding to months;the contained cells represent the days of a month (the last daysof the month are gray, to depicts the fact that some months arecomposed by less than 31 days).

Finally, as an abstraction specification involves a visual specifi-cation at the finest granularity and (possibly) several further visualspecifications at different granularities, the Paint Strips metaphorneeds to be suitably extended, to avoid that the user has to userollers with weights in a complex way, just to explicitly managethat she does not want any further condition on some intervalbounds at the considered granularity. Thus, we introduce a new
concept in the Paint Strips metaphor: the broken paint strip: a bro-ken paint strip can be used only in the specification of queries atgranularities coarser than the finest one and represent an inter-val with one (broken) bound having a position not temporally


sentin

suofwffstissalsato

3

attataFdtn

Fig. 2. Graphical notation for repre

pecified with respect to the other interval bounds. It allows theser to specify a granularity-related constraint involving only onef the two bounds of the considered interval. Figs. 3–5 show dif-erent intervals for temporal abstractions at different granularities,here some intervals are represented through broken paint strips;

or example, Figs. 3–5 in their part (a) specify a constraint betweenever, headache, and sore throat at the granularity years, repre-ented through different backgrounds: headache, fever, and sorehroat must start in the same year, the fever must finish at leastn the following year, while it does not matter when headache andore throat end. Part (b) of the considered figures depicts a con-traint between fever and headache at the granularity days: fevernd headache must finish the same day, headache must start ateast one day before its end, and it does not matter when fevertarts. Moreover, in the part (c) of the same figures, it is depicted

pattern at the granularity minutes, where it does not matter theemporal relationship between the end of fever and the beginningf sore throat, being both bounds broken.

.1.2. Composing temporal abstractionsLet us now consider how the introduced metaphors are used

nd integrated with other graphical tools in the visual interface,o allow the user to visually specify a composite temporal abstrac-ion. The temporal area is used to specify temporal relationshipsmong components, and to specify the valid time of abstractionshat will be derived. The steps followed by the user are: 1) the cre-tion of a new abstraction by selecting the item New in the menu
ile, and by indicating the name of the new abstraction; 2) therawing of strips corresponding to component abstractions withinhe temporal area, together with the selection of the abstractioname within a list of names proposed by the system. The proposed
g temporal intervals and relations.

list of abstraction names is previously extracted from the databaseconnected to the system. The selected name will be displayed inthe right part of the temporal area (part A.2 in Fig. 1), at the sameheight of the associated temporal interval. Each abstraction nameends with a parenthesized number, that represents the abstractionidentifier that is system-generated. A selected component abstrac-tion can be moved, scaled or removed by using the usual drag anddrop through mouse-driven pointers; moreover it can be furtherspecified by means of the graphical tools roller, bar, and distance(part A.5 of Fig. 1).

The graphical tools roller and bar, with the related weight, canbe used to specify the behavior of the interval bounds, according tothe meaning of metaphors introduced in the previous section andexemplified in parts (b) and (c) of Fig. 2, and in Figs. 3–5, in theirpart (c).

The graphical tool distance allows the user to define a minimumand maximum distance between two interval bounds. Fig. 7 showsthe dialogue window for defining the distance. In this window, theuser can select the interval bounds to be considered (Start-to-Start,Start-to-Finish, Finish-to-Start, Finish-to-Finish), and introduce theminimum distance and the maximum distance admitted betweenthe considered bounds. The distances defined in the dialogue win-dow of Fig. 7 may be specified at different unanchored granularities,discussed in Section 2.1. Fig. 8 shows the allowed distances set withthe dialogue window of Fig. 7. In this case, the distance constraintimposes that the fever interval must start at least one day, and notmore than three days, after the start of the headache interval.

By using simultaneously the roller and the distance, the user candefine situations in which it is unknown which of the two abstrac-tions starts before the other one, but once one of the consideredabstraction starts, the second one must start before a given time


Fig. 3. Example of brick-based backgrounds related to granularities of (a) years, (b) days, and (c) minutes.

Fig. 4. Example of backgrounds based on striped walls, related to granularities of (a) years, (b) days, and (c) minutes.


s, rela

irataw

stS

Fig. 5. Example of backgrounds, based on grained plaster

nterval. Fig. 9 shows an example of the simultaneous use of theoller and the distance tools: the depicted specification means thatny of the two intervals, i.e., fever and headache, can start beforehe other one, and there must be a temporal distance (between onend three days) between the beginning of the time interval whichill start first and the beginning of the second one.

The user can further refine the introduced abstractions by
uitably specifying temporal patterns at coarser granularities: inhis case the user selects the desired granularity from the menupecify Granularity and a new tab is created in the temporal
Fig. 6. Example of the temporal composition area wit

ted to granularities of (a) years, (b) days, and (c) minutes.

composition area, where the metaphors introduced for granulari-ties (i.e., backgrounds or icons) and for abstractions (i.e., strips, bars,weights and rollers, broken strips) are suitably supported. In partic-ular, the user could need to refine the considered abstraction onlywith respect to some bounds of the components. To define only theleft (right) bound of a broken paint strip, two more graphical toolscan be used: the paintStart and the paintFinish. As already under-
lined, they are useful when the user wants to relate two boundsby specifying a condition at a given granularity, without specify-ing any relationship between the other two bounds. For example,
h the icon representing the months granularity.


Fig. 7. Window for the graphical tool distance.

ation o

iion

ttapsa

Fig. 8. Visual represent

n Fig. 10, the user specifies, at the granularity days, that the fevernterval, the chest pain interval, and the cough interval must startn the same day, while relationships between ends of intervals areot refined at the days granularity.

The lower part of the temporal area, part A.3 in Fig. 1, is devotedo the specification of the valid time that will be associated tohe derived composite abstractions. The valid time of a compositebstraction is visually specified according to starts and ends of com-
osing abstractions: an empty (i.e., transparent) bar is displayedpanning all the overall temporal extension of the composingbstractions and each start/end of a composing abstraction has a
Fig. 9. Example of roller com

f a distance constraint.

corresponding bound in the bar representing the valid time. To setthe valid time, the user simply fills of the chosen abstraction colorthe required part of the bar, using as color delimiters the boundscorresponding to the composing abstraction starts/ends. Moreover,some further (metric) delay can be set with respect to the con-sidered start/end. It is worth noting that valid times of temporalabstractions are derived at the finest granularity: coarser granu-larities may be used only to specify how to derive abstractions.
Fig. 11 illustrates an example of valid time specification. In thiscase, if patients having an episode of fever during an episode ofheadache are retrieved, then the system stores in the database the
bined with distance.


refin

aTthfiatbwo

cmttfiOatitfia

Fig. 10. Example of abstraction

bstraction flu, associated to the patient with fever and headache.he valid time interval associated to the abstraction flu is defined inhe lower part of the temporal area and is delimited by the start ofeadache and the end of fever: it is visually performed by suitablylling the bar corresponding to the composite abstraction. A (neg-tive) delay is then added to the start of headache; thus, flu startshree days before the episode of headache starts (it is representedy the small left red arrow near the start of the interval togetherith the label 3d), and finishes at the same instant of the episode

f fever.Finally, as shown in the higher part of Figs, 1, 10, and 11, the user

an define, by using the Define menu, both a fixed time frame and aoving one related to the abstraction. The fixed time frame allows

he user to impose that the components must happen in a specifiedime interval. In Fig. 11, for example, the interval defined for thexed time frame is [2005-01-01, 2005-12-31], i.e., the year 2005.n the other hand, the moving time frame allows the user to specify

temporal constraint related to the period between the momenthe first component starts and the moment the last component fin-
shes. In Fig. 11, the moving time frame is set to three months. Thus,he specified composite abstraction flu (having abstraction identi-er (0)) is derived only if abstraction components headache (withbstraction identifier (2)) and fever (with abstraction identifier (1))
Fig. 11. Valid time specification fo

ement at the granularity days.

happened in 2005 and in a time window of three months. Afterthe definition of a new abstraction and its derivation with respectto the component abstractions stored into the database, the newabstraction can be used afterward in the specification of furthercomposite abstractions.

3.1.3. Checking the temporal consistency of the composedabstraction

The consistency of an abstraction specification at the finest gran-ularity is directly managed by the systems that does not allow tovisually compose inconsistent scenarios: for example, it is not pos-sible to insert several times the same component abstraction intothe visual pattern. Things change when the user is allowed to visu-ally specify a composite abstraction at the finest granularity andthen he can refine the specified pattern by visually building somefurther patterns at coarser granularities. As the tabs where the userspecify the temporal patterns holding at different granularities aredifferent, it is important that the system does not allow the user tobuild inconsistent overall scenarios, i.e., scenarios containing con-
tradicting patterns. For example, it would be meaningless to specifyon the basic timeline of seconds that nausea must start before feverstarts, and then, at the granularity days, require that fever must startbefore nausea starts.
r the composed abstraction.

8 elligen

itpwrqatFb

trastted

o(sTMetiisem

ai(mttt

tc(dttrt

e(tAeteaii

(iww


Consistency checking is performed in an incremental way, dur-ng the visual definition of an abstraction by the user; with respecto the general case, which can be reduced to the NP-completeroblem of verifying the consistency of a temporal constraint net-ork with different granularities [69], we consider only qualitative

elationships between interval bounds, disregarding distance (i.e.,uantitative) constraints between time points. Thus, we propose

polynomial algorithm for checking consistency of qualitativeemporal relations, based on an extension of the well-knownloyd–Warshall algorithm for evaluating the minimal distanceetween each couple of nodes in a weighted directed graph [70].

The algorithm we propose works on the qualitative rela-ionships existing between each pair of interval bounds. Eachelationship is represented by means of a comparison operatormong the classical ones for time points (<, >, =, ≤, ≥). For example,pecifying that nausea must start before fever means that the startime of the interval when nausea occurs must be before the startime of the interval when fever occurs. Representing the consid-red start bounds as N− and F− respectively, we can represent theescribed relationship with the comparison N− < F−.

The granularities we consider in this work are the calendricnes of years (Y), months (Mo), days (D), hours (H), and minutesMi), defined on the timeline of seconds (S), and are defined asuitable partitions of the timeline, as formally discussed in [16].hus, the set of considered granularities will be G = {S, Mi, H, D,o, Y}. Moreover, we introduce a total order between the consid-

red granularities, i.e., S < Mi < H < D < Mo < Y, according to the finerhan relationship between granularities (intuitively, a granularitys finer than another one when each granule of the first granularitys contained in a granule of the second one) [16]. The granularityeconds is the implicit granularity (basic timeline) and is the onemployed by the user when the user specifies the intervals in theain window.The operators (<i, > i, = i, ≤ i, ≥ i) to compare time instants defined

t different granularities are an extension of the classical compar-son operators and are represented by introducing the subscriptiwith i ∈ G) for the considered granularity. For example, A < DB

eans that the interval bound A must be at least one day (D) beforehe interval bound B: more precisely, the instant A must be beforehe instant B, when they are represented on a timeline having theime unit of days.

To identify inconsistent situations in defining granular abstrac-ions, we preliminarily introduce some equivalences betweenonjunctions of comparisons for a given pair of interval boundsi.e., time instants) A and B. The properties represented in Fig. 12escribe these equivalences and, at the same time, identify allhe consistent situations: only these situations will be allowed byhe visual interface. Moreover, these equivalences will be used toeduce any consistent conjunction to a predefined form of conjunc-ion.

Let us now discuss some of these properties. Considering Prop-rty 1, checking whether two time instants, i.e., A and B, are equalgreater than or equal to, less than or equal to) with respecto two different granularities, i.e., A = iB ∧ A = jB (A ≥ iB ∧ A ≥ jB,

≤ iB ∧ A ≤ jB), is equivalent to check that the two time instants arequal (greater than or equal to, less than or equal to) with respecto the finer granularity, i.e., A = min{i,j}B (A ≥ min{i,j}B, A ≤ min{i,j}B). Forxample, we could require that A occurs the same year of B (A = YB)nd A occurs the same minute of B (A = miB): to check this situation,t is sufficient to check that A occurs the same minute of B becauset implies that A and B occur the same year too.

As for Property 2, checking whether time instant A is before
after) time instant B with respect to two different granularities,.e., A < iB ∧ A < jB, is equivalent to check that A is before (after) B
ith respect to the coarser granularity, i.e., A�max{i,j}B. For example,e could require that A occurs at least one day after B (A > DB) and

ce in Medicine 54 (2012) 75– 101

A occurs at least one month after B (A > MoB): to check this situa-tion it is sufficient to check that A occurs at least one month after Bbecause this implies that A occurs at least one day after B too.

As a further example, we could require that A occurs at least1 min after B (A > MiB) and A occurs in the same or a previous monthto B (A ≤ MoB): to check this situation, it is sufficient to check that Aoccurs at least 1 min after B and A occurs on the same month of B,as stated by Property 6.

Thus, every time we have a conjunction as in the left part of oneof the introduced properties, we equivalently use the right partof the considered property. Any other conjunction or comparisonwill be inconsistent: for example, the conjunction of A > iB and A < jBwith i, j ∈ G and i < j is inconsistent as it represents two incompatiblesituations.

After defining the behavior of comparison operators in conjunc-tions of comparisons relating a given pair of time instants (such asA and B), we will consider their transitive composition, i.e., giventhe existing relationships between the time instants A and B, andbetween the time instants B and C, we will derive the relationshipsexisting between the time instants A and C. The transitive compo-sition is expressed through the formulae represented in Fig. 13.

For example, if A occurs on the same day of B (A = DB) and B occurson the same month of C (B = MoC), we can derive that A occurs onthe same month of C, as stated by the formula in Derivation 1.

The proposed algorithm uses the equivalences and the deriva-tions listed in Figs. 12 and 13 to identify possible inconsistenciesduring the definition of a granular abstraction. The main ideabehind the algorithm is to search for inconsistencies by exhaus-tively applying the derivations involving three different abstractionbounds, and comparing the obtained relationships with the rela-tionships obtained in the previous step for any pair of abstractionbounds: if these relationships are inconsistent, the algorithm termi-nates returning the boolean value FALSE. Otherwise, the algorithmderives a single set of relationships between two abstractionbounds and runs until the relationships found between any twoabstraction bounds do not change; in this case the boolean valueTRUE is returned, meaning that the overall granular abstractionis consistent. Fig. 14 illustrates the proposed algorithm: the threeinternal for cycles are similar to the core of the Floyd–Warshallalgorithm for computing the minimal paths on a weighted, directedgraph; an external while cycle is needed to check the convergenceof the algorithm. Indeed, as our algorithm works on a matrix Mcontaining sets of symbolic values representing the relationshipsholding for any two abstraction bounds, the least path lemma [70]used in the Floyd–Warshall algorithm does not hold anymore. Weproved that the algorithm terminates. Intuitively, the derivationsapplied for abstraction bounds converge to find strict relationshipsinvolving only either greatest or smallest granularities, for inequal-ities and equalities, respectively; as the granularities are finite, weare guaranteed that the algorithm terminates in a finite number ofsteps. The external while cycle in the algorithm of Fig. 14 couldbe executed a finite number of times for all the matrix cells: in theworst case, i.e., that of a single cell modifying its content for eachiteration, we have to consider 2n2 iterations (being the matrix sym-metric and n the number of involved abstraction intervals). Fromthe fact that the Floyd–Warshall algorithm, corresponding to thethree internal for cycles, is O(n3), the overall complexity (worstcase) is O(n5), while the mean complexity for the practical caseswe checked is O(n3).

Now, let us consider a simple example of the application ofthe algorithm we propose for checking consistency when defininggranular abstractions. Fig. 15 depicts the definition of an abstrac-
tion at the granularity seconds (i.e., the basic timeline). In thisabstraction, the user specifies that fever (17) may start eitherbefore, or simultaneously to, or after abdominal pain (21), andmust finish before headache (18). Moreover, the user specifies


aring

tp

rmyyub

amF

Faiapgardwb

TT

Fig. 12. Equivalence properties for comp

hat headache may start either simultaneously or after abdominalain.

After defining the abstraction on the basic timeline, the userefines it at the granularity years as shown in Fig. 16. In this refine-ent the user specifies that headache must hold at the end of an

ear (i.e., the headache start must precede the end at the granularityears) and headache must finish before abdominal pain at the gran-larity years, while she does not specify any constraint between theeginning of headache and abdominal pain.

Finally, the user refines the abstraction at the granularity monthss shown in Fig. 17. In this refinement the user specifies that feverust finish on the same month where abdominal pain finishes.

urthermore, start and end of fever must be in different months.Table 1 represents matrix M given in input to the algorithm of

ig. 14. In this table we represent the start and end times of anbstraction interval i by using “i−” and “i+” respectively, and reportn the cells the relationships implicitly specified by the user in thebstraction definition and in the abstraction refinements. For exam-le, the fact that fever (17) must finish before headache (18) at theranularity seconds (S) is represented in the cell related to row 17+

nd column 18+ by means of the relationship <S. When the algo-ithm runs, the 32nd time that it applies derivations involving three
ifferent abstraction bounds (i.e., when the algorithm variables s, t,
take the values 1, 3, 5, respectively), it first derives the relationshipetween 18+ and 21+ by concatenating >S (between 18+ and 17+)

able 1he matrix M given in input to the algorithm.

17− 17+ 18− 18+ 21− 21+

17− – <S , <Mo <S <S , < Y

17+ >S , >Mo – >S <S >S <S , = Mo

18− <S – <S , <Y ≥S <S , <Y

18+ >S >S >S , > Y – >S <S , <Y

21− <S ≤S <S – <S

21+ >S , >Y >S , = Mo >S , >Y >S , >Y >S –

time instants at different granularities.

and = Mo (between 17+ and 21+), obtaining ≥Mo. Then, it comparesthe obtained relationship with the one in the matrix for bounds18+ and 21+, i.e., <Y, and detects the inconsistency, as highlightedin Table 2.

3.2. Visually querying temporal abstractions: the logical area

After being defined, temporal abstractions can be combined toproperly compose queries on the database of patient histories. Inthe following, we first introduce the metaphor we propose for log-ical expressions; then, we describe how to visually specify queriesinvolving several abstractions, possibly sharing some componentabstraction.

3.2.1. The logical notationIn the logical area, the user can specify logical conditions on

the defined abstractions. The main idea of the formalism we pro-pose tries to overcome the graphical limitations of Venn diagramswhen there are several (i.e., more than three) query terms [63].The proposed formalism is based on the usual set representation,where objects in sets are the atomic query terms (in our case, the
considered abstractions). Objects are visually represented as smallcolored tokens, labeled by the abstraction identifier. Sets, visuallydepicted through closed lines (e.g., circles or boxes), represent log-ical connectives among the objects of sets. Different contours (e.g.,
Table 2The matrix M with the identified inconsistency ( ≥ Mo ∧ < Y ).

17− 17+ 18− 18+ 21− 21+

17− – <S , <Mo <S <S , < Y

17+ >S , >Mo – >S <S >S <S , = Mo

18− <S – <S , <Y ≥S <S , <Y

18+ >S >S >S , > Y – >S ≥Mo , <Y

21− <S ≤S <S – <S

21+ >S , >Y >S , = Mo >S , >Y >S , >Y >S –


lation

cflpstmtlnt

Fig. 13. Rules for deriving temporal re

ontinuous, dashed, and so on) and lines are used to specify dif-erent logical connectives. This way the user may freely composeogical conditions by drawing the corresponding connective andutting in it the suitable tokens (i.e., abstractions). The free compo-ition may support an arbitrary number of parentheses, to specifyhe order of application of the specified connectives: connectives

ay be simply nested to obtain the required complex logical condi-
ion. The goal behind this visual formalism is twofold: (i) we want toet the user freely compose logical conditions without forcing anyormal form, as, for example, the disjunctive normal form, wherehe only allowed conditions are composed by the OR of groups of
ships through transitive composition.

several ANDed terms; (ii) we try to use a formalism resemblingthe graphical symbols for sets, where, in this case, sets collect theterms (i.e., abstractions or other sets) that have to be properlyconnected.

In the proposed visual query system, logical connectives are rep-resented by means of different kinds of boxes; a box contains all thetokens associated to the considered abstractions. More precisely, a
box with a continuous border represents the AND connective, asshown in Fig. 18a, where the colored tokens representing abstrac-tions (i.e., flu (0) and viral gastroenteritis (20)) are considered inconjunction (0 AND 20).


sisten

sd

tireafiiiae

Fig. 14. The algorithm for checking con

A box with a dashed border represents the OR connective, ashown in Fig. 18b, where the above abstractions are contained in aashed box (0 OR 20).

As for the negation (NOT), there are two different possibili-ies: when a single abstraction needs to be negated, it can be putnside a crossed box, as depicted in Fig. 18c, where the negationefers to the abstraction 0 (i.e., we are looking for patients who hadither viral gastroenteritis or not flu; when the negation involves

complex expression, as conjunction, disjunction, or an arbitraryormula composed by conjunctions, disjunctions, and negations, its visually represented by crossing the most external box, contain-ng all the other boxes and tokens of the expression. For example,
n Fig. 18d, the negation involves the conjunctions of the tokens 0nd 20 (i.e., NOT(0 AND 20): we are looking for patients who hadither only flu or only viral gastroenteritis or no one of them).
Fig. 15. Defining an abstractio

cy when defining granular abstractions.

The user can specify conditions by composing the graphicalconnectives described above, and by duplicating the filled colored(numbered) tokens. Fig. 18e illustrates an example representing adisjunction of two conjunctions (i.e., ((0 AND 12) OR (12 AND 20))).In this case, we need to duplicate the colored token identified by(12) (representing the abstraction pneumonia), then we use the boxwith a continuous border for representing (0 AND 12) and (12 AND20), and finally we use the box with a dashed border to includethe two created boxes for representing the disjunction OR. In thisexample, we are looking for patients having had viral gastroenteritis(20) and pneumonia (12) or flu (0) and pneumonia (12).

Another example of complex condition is illustrated in Fig. 18f:
here, we are looking for patients who had pneumonia but not viralgastroenteritis, or pneumonia but not flu (i.e., (((NOT(0) AND 12)OR (NOT(20) AND 12)))).
n on the basic timeline.


raction

3

Pstia

tTitoetumoi(tdtApsS

t

Fig. 16. Refining the abst

.2.2. Composing queries with temporal abstractionsAfter defining (or opening) composite abstractions through the

aint Strips metaphor available in the temporal area, the user canpecify a logical condition on the available abstractions throughhe token-based notation previously discussed. The composed log-cal condition will be used to query the patient database, wherebstractions represent patient histories.

Tokens are available, corresponding to the considered abstrac-ions: colors of tokens correspond to the colors of abstractions.okens and boxes for connectives can be inserted and managedn the logical area. During the composition of the logical condition,he corresponding logical expression is visualized in the upper partf the logical area: abstraction identifiers are used to represent thexpression, to provide the user with short expression. However,o improve the readability of the composed logical conditions, theser can switch from the compact textual representation to theore extended representation obtained using the complete names

f abstractions. As an example, with regard to the scenario depictedn Fig. 1 and in Fig. 19, tokens are available for abstractions flu0), pneumonia (12), and viral gastroenteritis (20). Fig. 1 depictshe logical expression using abstraction identifiers, while Fig. 19,isplaying in the temporal area the abstraction pneumonia, depictshe same logical expression by explicitly using abstraction names.fter the definition of the logical condition, the system looks foratients having a history, i.e., a set of abstractions, satisfying the
pecified condition (the query specification is translated into anQL statement according to the approach discussed in [7]).
Composite abstractions have been defined in isolation and thushere are no connections between the components of different

Fig. 17. Refining the abstraction

at the granularity years.

abstractions. For example, considering the logical condition inFig. 18e, we are looking for patients who had viral gastroenteri-tis (20) and pneumonia (12) or flu (0) and pneumonia (12); in thiscase, we are not interested in the period when these abstractionsoccurred (i.e., a patient suffering from intestinal flu in 1980 andfrom pneumonia in 1990 would be selected as well as a patient suf-fering from intestinal flu and from pneumonia in the same month).When we need to identify patients who are involved in all the con-sidered abstractions in the same period, we could use a suitablefixed time frame, restricting the query evaluation to only a portionof patient histories. Moreover, the physician could be interestedin identifying patients having abstractions sharing some of theircomponents. For example, let us consider the query depicted inFigs. 1 and 19 involving the abstractions flu (0), viral gastroenteritis(20), and pneumonia (12). Here, we are looking for patients whohad pneumonia but not viral gastroenteritis, or pneumonia but notflu (i.e., (((NOT(20) AND 12) OR (NOT(0) AND 12)))). However, weare interested in selecting only those patients who had pneumoniaand not episodes of flu related to the same symptoms and/or signs;even patients who had pneumonia without flu in May 2005 andflu in July 2005 should be selected. In this case, we have to connectthe components of the considered abstractions: more precisely, weneed that the query has to be evaluated with the fever componentset to be the same for the three abstractions. This is specified inthe Connection area, in the left part of the Logical area, as shown
in Fig. 19, where the component fever (23) of abstraction viral gas-troenteritis has to be the same of component fever (1) of abstractionflu and the component fever (13) of abstraction pneumonia has tobe the same of component fever (1) of abstraction flu.
at the granularity months.


F ationc

4

aaJowd

4

ptpAmTaabp

ig. 18. Logical area: AND (a) AND, (b) OR, (c) negation of an abstraction, (d) negonjunctions with negations.

. Prototype design, implementation and evaluation

In this section we briefly discuss the design, implementation,nd evaluation of the proposed prototype for the visual definitionnd query of temporal clinical abstractions. We developed it inAVA; a preliminary evaluation of the interface has been carriedut involving some end users from the medical domain, and, finally,e used the system prototype for the management of hemodialysisata.

.1. Prototype architecture

Fig. 20 illustrates the architecture of the system by depicting theackage diagram [71]. The package Graphical User Interface allowshe user to interact with the system. Information management isrovided by the other packages shown in the diagram. In particular,bstractionsContainer, which is the core of the system, handles infor-ation introduced by the user through the Graphical User Interface.

he package AbstractionsContainer stores data when it represents
bstractions, or passes it to the other packages. Constraints associ-ted to the defined abstractions, such as time distance, are managedy the package ConstraintsContainer, while connections among com-onents in different abstractions are dealt with by the package
of a conjunction, (e) disjunction of two conjunctions, and (f) disjunction of two

SameElementsContainer. The consistency of abstractions is checkedby the package ConsistencyContainer. The logical area is managedby LogicContainer, which deals with the logical aspects representedin the graphical interface. Information managed by the system isstored in a database, through a DBMS. The interaction with thedatabase system is managed by the package DatabaseManager.

4.2. Evaluation of the proposed approach

The evaluation of the proposed system has been carried out isseveral parts and steps. The first part deals with completing theevaluation of the metaphors involved in the temporal area, afterthe detailed evaluation of Paint Strips described in [7], by consid-ering the constructs related to the specification of abstractions atdifferent granularities. The second part of the evaluation deals withthe visual definition of queries involving different composite gran-ularities and focus on the capability of specifying complex logicalconditions through the logical notation provided in the logical area.

4.2.1. Evaluating the paint strip metaphor extended withgranularities

In a previous work [7], the authors evaluated the Paint Stripsmetaphor against other metaphors based on real world objects.


Fig. 19. Logical area: connecting abstractions.

Graphi cal User I nter fac e

SameElementCon taine r ConstraintsCon taine r

Abstraction sCon taine r

ConsistencyCon taine r DatabaseManage r

LogicContaine r Database

Fig. 20. The architecture of the prototype for the visual definition and querying of clinical abstractions. Dashed lines represent how the user interface interacts with the otherpackages to support the interaction with the user, while continuous lines represent how data inserted by the user through the user interface are shared by the packages.

lligence in Medicine 54 (2012) 75– 101 95

Pip[soc

ptwatwtTawsctwlTtctnhFleobdtm

ee

btoaupttDwiojtsjwmiict2tl

the starting (ending) instant of a granule, and thus do not under-stand that the position of a strip with respect to a brick does notmatter. On the other side, some other users find that vertical lines


aint Strips received a statistically significant preference from usersnvolved in the evaluation (significance of this result has been com-uted with Pearson’s Chi Square test for one-way tables (p < 0.025)7]). In this further evaluation, we mainly considered the issue ofpecifying abstractions at different granularities together with anverall evaluation of the provided interface for abstraction specifi-ation.

Experimental task: The experimental task consisted in twoarts: in the first one subjects were asked to solve 30 interpre-ation exercises provided through the real interface. Each exerciseas composed by a screenshot representing a possible scenario

t a given granularity and by three different possible interpreta-ions, of which only one was correct. In the second part, subjectsere required to solve a complex exercise, by visually specifying

hrough the interface a scenario described only in natural language.he interpretation exercises showed simple abstraction patternst some granularity: no more than three component abstractionsere considered. The specification exercise of the second part con-

idered a more difficult scenario, where subjects had to define aomplex abstraction at the basic granularity, and then to refinehe definition at two coarser granularities. The complex abstractionas composed by four component abstractions at the basic granu-

arity, related by using all the constructs of the temporal metaphor.he two refinements at different granularities involved two andhree component abstractions, respectively. The specification exer-ise was expressed through a natural language description similaro the following: “Find all patients having the following clinical sce-ario: an episode of fever started before headache appeared. Aftereadache started, patients had an episode of low blood pressure.ever could have started before, after or together with a decrease ofeukocytes. Fever ended before the end of headache that in its turnnded 2–4 h before the end of low blood pressure. The decreasef leukocytes could have ended before or together the end of lowlood pressure. Fever started the same day headache started. Theecrease of leukocytes has to have started at least the month beforehe start of low blood pressure and has to have ended the same

onth headache ended.”The overall purpose of the task was to assess whether differ-

nt backgrounds help users to understand the granularities used inxpressing composite abstractions.

Experiment design and procedure: The exercises were doney 13 subjects (five females and eight males). Age ranged from 20o 22, averaging at 21. Subjects were university students (2nd yearf the technical degree in Medical Radiology Techniques, Imagingnd Radiotherapy). All the subjects were familiar with computersage and had computer-related university courses. Each subjecterformed the task in a single session. Each session began with araining phase, lasting about 20 min, where subjects were shownhe interface and told about the meaning of its graphic elements.uring training, the subject was first guided to directly interactith the interface: for each graphic object, (s)he learned how to

nsert, modify, and delete it. Then, the meaning of each graphicbject in the context of temporal patterns was explained. The sub-ect was invited to ask for any clarifications (s)he wanted duringhe training phase, because it was not possible to do it in theubsequent parts of the session. After the training phase, sub-ects were introduced to the first part of the experimental task,

here 30 interpretation exercises were proposed for two differentetaphors (15 exercises for each metaphor). For each screenshot of

nterpretation exercises, subjects were asked to identify the rightnterpretation within 3 min. Then, subjects performed the specifi-ation exercise (about 15 min) with a metaphor not considered in
he interpretation exercises. After task completion, subjects filled a8-questions user’s satisfaction questionnaire inspired to the ques-ionnaire for user interaction satisfaction (QUIS) [72]. To minimizeearning effects on the experiment results, different users took
Fig. 21. Correct answers for interpretation exercises.

different metaphors in different orders and executed different exer-cises according to different orders. In each session, we collected thefollowing quantitative data: number of correct answers to inter-pretation exercises and number of correct answers (for questionsrelated to backgrounds for granularity) to specification exercises.Qualitative impressions were recorded with the user’s satisfac-tion questionnaire. The hardware used for the experiment was aWindows-based PC with a standard 17 inch monitor.

Analysis and results: The results of the interpretation exercisesare summarized in Fig. 21: the percentage of correct answers are54.4, 73.3, 61, and 61 for striped wall, plastered wall, brick wall,and absence of wall, respectively. The number of correct answersto exercises were analyzed using the t test for two unpaired sam-ples, by comparing each metaphor with each other. Mean values(by subject) for correct answers are reported in Fig. 22: post hocstatistical analysis on means confirmed that differences betweenresults obtained through different metaphors are not statisticallysignificant. On the other side, specification exercises pointed outthat for complex queries subjects had more issues in managing thespecification at the basic granularity than in refining it at differentgranularities.

The user’s satisfaction questionnaire confirmed that there areno preferences for one of the granularity-related metaphors. Fromthe qualitative point of view, questionnaires helped us in focusingon pros and cons of the different proposals. The most importantcomments reported by the subjects may be summarized as in thefollowing:

• some users associate the starting (ending) point of a brick with

Fig. 22. Means of correct answers for interpretation exercises.

9 elligen

•

•

•

dqics

ctbtfgaot

wieoapf

tifttdbgiup

4

pbkafsowttn


of bricks help in the comprehension of the temporal relationsbetween different paint strips.The plastered wall makes the comprehension of informationrelated to the granularity easier, and is pleasant from an estheticpoint of view. The main drawback pointed out from people is thatthe differently grained backgrounds are hard to distinguish onefrom another.The use of only labels seems to be easy to understand, but unin-teresting from an esthetic point of view.The striped wall interferes with the comprehension of informa-tion related to the granularity, and does not have any positiveaspect. Users find that this solution does not improve the com-prehension of information related to the granularity, and doesnot present advantages with respect to other solutions.

Final think-aloud sessions: On the base of the previously intro-uced results, we decided to involve more people in a deeperualitative evaluation and analysis of the proposed metaphors: we

nvolved seven more people (three physicians, one medical physi-ist and four computer scientists) in three different think-aloudessions.

In the first think-aloud session with a physician and the medi-al physicist, we did not explain to the involved people the aim ofhe evaluation: i.e., we did not mention the need of evaluating theetter metaphor for representing the different granularities. Fromhe comments we received in this session, a negative evaluationor the proposed backgrounds emerged. People found that back-rounds are useless for providing information about granularities,nd moreover backgrounds are perceived as misleading. The use ofnly labels for representing the different granularities seems to behe better solution.

In the second think-aloud session, with two young physicians,e introduced people to the basic concepts needed for understand-

ng the graphical interface, and explained them the aim of thevaluation. The comments collected in this session are similar to thenes of the previous session. People preferred the use of only labels,nd found backgrounds difficult to understand. In this session, peo-le suggested us to place an image or an icon in the temporal areaor representing different granularities.

In the last think-aloud session, we involved four computer scien-ists, thus the discussion concerned possible alternative solutions,nstead of the evaluation of the proposed ones. Involved people pre-erred the use of only labels because backgrounds seemed makinghe interface too heavy. In this case, the comment is related moreo the usability of the system then to the comprehension of theifferent granularities. Moreover, people suggested us other possi-le approaches, such that different colors for representing differentranularities, transparent icons on the background, and, suitablecons placed in the temporal area. The last solution has been eval-ated as the best one, according also with the suggestion from therevious think-aloud session.

.2.2. Evaluating the logical notationIn evaluating the logical notation we compared it (using a

revious version of the logical notation based on more roundedoxes, i.e., elliptical boxes) with a suitable adaptation of the wellnown tabular query form proposed in [14] (hereinafter TQF): itllows one the specify logical expressions in disjunctive normalorm, as discussed in Section 2.4. The implemented TQF has beenuitably adapted to the context: for example, instead of matching-bject selection, the interface use the sentence “Find abstraction(s)
here:”; moreover, a colored and labeled token is used to refer to
he considered abstraction. The interface allows one to introduceokens, to delete or duplicate tokens, to negate a token, to introduceew match forms. Fig. 23 depicts an example of query expressed

ce in Medicine 54 (2012) 75– 101

through the implemented TQF: in particular, it refers to the samequery expressed in Fig. 18f.

Experimental task: The experimental task consisted in twoparts: in the first one subjects were asked to solve five interpre-tation exercises provided through the real interface. Each exercisewas composed by a screenshot representing a possible scenarioand by three different possible interpretations, of which only onewas correct. The exercises had an increasing difficulty and wererepresenting different logical expressions: the easier expressionswere composed only by AND and OR connectives, respectively.Then, we increased the difficulty by introducing the NOT connec-tive with either AND or OR connectives, and then with both theconsidered connectives. Moreover, correct answers were some-times expressed by simple sentences not exactly corresponding tothe logical expression, specified in disjunctive normal form: thegoal of these exercises was also verifying that TQF does not pre-vent the user from understanding queries not directly expressed asdisjunctions of conjunctions.

In the second part, subjects were required to solve five exer-cises, by visually defining through the interface a scenario describedonly in natural language. As for interpretation exercises, also spec-ification exercises were proposed according to their (increasing)complexity. Even for the specification exercises, two exercises havethe scenario expressed not in disjunctive normal form, to verifywhether TQF has some limitation when users have to deal withnatural language expressions not directly related to a query in dis-junctive normal form. As an example of the complexity of exercises,the five specification exercises had the right answers similar to thefollowing logical formulas, respectively:

1. fever AND headache AND sore throat;2. fever OR headache OR NOT(sore throat);3. sore throat AND (fever OR headache);4. NOT(fever AND headache AND sore throat);5. (headache OR sore throat) AND NOT fever.

Experiment design and procedure: Subjects were recruited atthe Medical Clinic of the University of Udine and performed boththe evaluation on metaphors for the temporal area, discussed in [7],and the evaluation of the logical notation. A total of 31 people (15males and 16 females) were recruited: six students in Medicine,four medical doctors (MDs) who had just earned their degree, 18MDs specializing in various subfields of medicine (two in ClinicalPharmacology, 10 in Public Health, five in Clinical Psychiatry, one inSurgery), one psychologist, one pharmacologist, and one physicianemployed in the Public Health Department. Age ranged from 23to 44, averaging at 30. With respect to computer usage, only onesubject never used a computer; the other subjects were equallysplit among those that use it for only a couple of hours per week andthose who use it for five or more hours per week. All subjects wererequired to do the same tasks with both the interfaces, during twosessions in different days. To minimize learning effects, we did notfollow a predefined order for the used interface. For each interfacesubjects performed the following steps:

1. training with the interface;2. interpretation exercises;3. specification exercises;4. user’s satisfaction questionnaire filling.

During the training step, all explanations and examples were givenabout the meaning of graphical symbols and about their usage:
subjects were asked to directly use the interface both for the tem-poral part and for the logical one: this way the meaning of logicalnotations was explained and understood with respect to the spe-cific context of temporal clinical abstractions. Subjects were invited


Fig. 23. Logical area: disjunction of two conjunctions with negations in tabular query form [14].

w FIN

ttrtawfwn

Fig. 24. Defining the abstraction QBlo

o ask questions and to solve some training exercises to confirmheir familiarity with the introduced concepts. When subjects wereeady, they were introduced to the second step, where they hado solve for each interface five interpretation exercises (one rightnswer among three) with increasing complexity. Then, subjects
ere asked to solve five specification exercises with each inter-
ace; even in this case, the complexity of the described scenariosas increasing. At the end, subjects answered a short question-aire about their preference and their feeling with respect to the

Fig. 25. Completing the definition of the abstraction QB

ISHES SBPdecr on the basic timeline.

graphical notations. All the experiments were driven by the sameexperimenter. The hardware used for the experiment was a stan-dard PC with a 17 inch monitor.

Analysis and results: We first checked whether TQF is perceivedas less difficult to learn with respect to the proposed logical nota-
tion. Moreover, we verified whether the subjects did less errorsby using TQF for queries expressed in disjunctive normal form. Onthe other hand, we verified whether subjects are more comfort-able with the proposed logical notation when they need to specify
low FINISHES SBPdecr at the granularity minutes.

9 elligen

qcrettesaTtwahpcdiTectrirtmlwGtscpc

fwctohpwiS

4

•

•

•

• having had either a decrease of SBP or a low DBP finished by a period


ueries not expressed in disjunctive normal form: indeed, in thisase the translation in disjunctive normal form is assumed to beelatively hard for users not familiar with concepts from math-matical logic. Suitable statistical tests were performed on timesaken for the different tasks and on correct answers for interpre-ation and composition tasks. The number of correct answers toxercises were analyzed using the Wilcoxon test for two relatedamples; results concerning time spent to complete tasks werenalyzed using the paired-samples t test. Results confirmed thatQF required significant (p=0,0001 - paired-samples t test) lessime for the training phase: the mean times for the training phaseere about 21 min and 14 min for the proposed logical notation

nd for TQF, respectively. This result could be explained by theigher number and combination of objects introduced by the pro-osed logical notation with respect to TQF. This result, however,ould be also related to the ability of the experimenter in intro-ucing all the different concepts of the two interfaces. As for the

nterpretation exercises, our logical notation performed better thanQF with regard to the mean number of correct answers both forxercises expressed in disjunctive normal form and for the exer-ises not expressed in disjunctive form; however, subject took lessime to solve interpretation exercises with TQF. Both for the cor-ect answers and for times, the result differences between the twonterfaces failed to reach the statistical significance. A more soundesult to this regard could be related to the times taken by subjectso solve the composition exercises expressed in disjunctive nor-

al form: in this case, subjects employed significantly (p = 0.018)ess time to solve them with TQF. Interestingly, the correct answers

ere not significantly different for the two considered interfaces.oing to the (relatively simple) exercises not expressed in disjunc-

ive normal form, the two interfaces did not provide statisticallyignificant differences both for the time taken to solve the exer-ises and for the correct answers, even though in this case subjectserformed better with our logical notation both for times and fororrect answers.

The final questionnaire confirmed that no one of the two inter-aces is preferred to the other one (significance has been computedith Pearson’s Chi-Square test for one-way tables): the only result

lose to the statistical significance (p = 0.065 – Wilcoxon test forwo related samples) was related to the visual (esthetic) qualityf the two interfaces. In this case, the proposed logical notationas been preferred to TQF. In summary, the evaluation of the pro-osed logical notation confirms its soundness with respect to aell-known interface as TQF, especially considering the complex

ssue of visually representing logical formulas, as discussed also inection 2.4.

.2.3. Some final comments about the preliminary evaluationsThe obtained evaluation results may be summarized as follows:

both the metaphors for granularities and the logical notation con-firmed the soundness of the proposed solutions, even though noone of the proposed solutions emerged as the (significantly) bestone;as for metaphors for granularities, it seems that users prefer someicon-based solution, instead of adding new objects to the existingPaint Strips metaphor;as for the logical notation, the proposed solution obtained resultscomparable with the ones of a well-known tabular query form,even in the presence of less constrained, and thus more complex,constructs;the obtained results confirm that both the tasks of specifying
complex temporal abstractions and of defining temporal queriesover a set of temporal abstractions need to be oriented to specificgroups of users: indeed, the refinement of this kind of pro-posals needs to be oriented to skilled clinical users, aware of
ce in Medicine 54 (2012) 75– 101

non-elementary concepts as those of temporal abstraction, gran-ularity, and (temporal) logical expressions. To this regard, someanalogy arises with respect to graphical tools supporting On LineAnalytical Processing (OLAP) analysis of temporal data [73].

• As the evaluation was preliminary, the fact that the proposedsolutions are not significantly different is not a negative result.The overall goal of the evaluation was to discover pros and consof the proposed solutions and to check their overall soundnessand acceptance by users: the evaluation allowed us to reach thisgoal.

5. A real world application: defining and queryingabstractions for hemodialysis data

Hemodialysis is a well-known periodic treatment for patientswith acute or chronic end-stage renal failure. During an hemodial-ysis session, a machine (the hemodialyzer) removes metabolites(e.g., urea) from patient’s blood, re-establishes the acid-base equi-librium, and removes water in excess. This process is performedthrough extracorporeal blood circulation. Usually, hemodialysispatients are treated three times a week and each session lasts about4 h. Unfortunately, the number of patients that need hemodialysisis constantly increasing [74]. Since this treatment is very expen-sive as well as critical for the patient’s quality of life, it is veryimportant to be able to evaluate the quality of (i) each singlehemodialysis session and (ii) all the sessions concerning the samepatient over a period of time, for the early detection of qualityproblems in the hemodialytic treatment. Modern hemodialyzersare able to acquire a huge number of parameters from the patient(e.g., heart rate, blood pressure, weight loss due to lost liquids,. . .) and from the process (e.g., pressures in the extra-corporealcircuit, incoming blood flow, . . .), with a configurable samplingtime.

As we already underlined, our system assumes that basicabstractions on clinical data have already been obtained: in thiscase, the main basic abstractions are related to the behavior ofthe numerical parameters with respect to time. In particular, thebasic abstractions we considered are related to QB (the blood flowentering the hemodialyzer), SBP (Systolic Blood Pressure), and DBP(Diastolic Blood Pressure). QB is set by nurses below the optimalvalue when the patient’s blood pressures either decrease or are ata low level. Although this QB suboptimal setting is needed becauseof the patient’s state, a level of QB below the optimal value nega-tively affects the overall quality of the provided care. In this context,it means defining the abstractions that identify different interest-ing situations: the first one, QBlow FINISHES SBPdecr, specifies thatan interval where QB is below the optimal value starts after, butin the same minute, and finishes together with an episode of SBPdecreasing, as depicted in Figs. 24 and 25. The second abstrac-tion, QBlow FINISHES DBPlow, specifies that the period when QBis below the optimal value starts after and finishes together witha period when the patient has DBP below her usual value, as inFig. 26.

The final abstraction, QBlow MEETS QBopt, deals with a periodwhen the patient has a QB below the optimal value immedi-ately followed by a period when QB is set at the right value,as depicted in Fig. 27. The query involving the three previousabstractions looks for patients having had QBlow FINISHES DBPlowor QBlow FINISHES SBPdecr and QBlow MEETS QBopt, as depicted inthe logical area of Fig. 27. In other words, we are looking for patients

of suboptimal QB, immediately followed by a period with QB set tothe right value: this way, we can distinguish patients who had theirQB changed, due to worsening of their status, and then set to theoriginal value before the end of the hemodialysis session.


Fig. 26. Defining the abstraction QBlow FINISHES DBPlow.

d que

6

tTitpar

Fig. 27. Defining the abstraction QBlow MEETS QBopt an

. Conclusions

In this paper, we proposed a visual framework for composingemporal abstractions and querying clinical abstraction databases.o make our proposal more flexible with respect to time, wentroduced the possibility of visually defining temporal abstrac-
ions involving different granularities. Moreover, we consideredroblems related to consistency checking in defining temporalbstractions by using different granularities, and proposed an algo-ithm for identifying inconsistencies. The first evaluation of the
rying the database by using all the defined abstractions.

proposed metaphor objects for granularities allowed us to high-light pros and cons of each solution and to focus on the proposal ofsome specific graphical icons for granularities. We then proposedand evaluated a visual language for expressing boolean queriesinvolving different abstractions: the proposed notation showedsome interesting preliminary results as the evaluation did not high-
lighted significant differences with respect to another well knownformalism, namely TQF [14]. An application of the visual frameworkhas been described, dealing with the analysis of hemodialysis data.The overall evaluation of the visual framework confirmed us that

1 elligen

btnsass

tptirAawwpcmt

A

DfvVhpteeattocmtcs

R

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[


oth the specification of abstractions involving different granulari-ies and the definition of boolean queries are complex tasks both foraive and for expert users; both of them need to be supported byophisticated visual systems. Moreover, a more exhaustive evalu-tion is needed for the overall visual framework, considering moreubjects and more focused and specific tasks, possibly consideringome real world clinical application.

In this paper, we did not discuss how query results, i.e., the foundemporal abstractions, are visually represented: to this regard, ourroposal seems to be complementary to other research efforts inhe visualization of temporal data and could be straightforwardlyntegrated with some of the well known visual systems for theepresentation/exploration of temporal clinical data [42,9,43,10].lthough our research has focused on definition of temporalbstractions and queries by users, an interesting future line of workill be to investigate how our visual language could be integratedith automated analytical methods which are able to identify tem-oral patterns. In this way, the temporal analysis task could beooperatively carried out by the human user and the automaticethods, using the visual language as a common form of represen-

ation and communication.

cknowledgements

This research was partially funded by several grants from theepartment of Computer Science of the University of Verona and

rom the Department of Mathematics and Computer Science, Uni-ersity of Udine. We would like to thank the students Claudiaicenzotto and Andrea Visentini, from the University of Udine, whoelped in developing the proposed model and the implementedrototype during their master thesis work. We would thank alsohe student Fabio Salzano, from the University of Verona, whovaluated the metaphors for different granularities during his bach-lor thesis. Thanks are due to people from Universities of Udinend Verona (Faculty of Medicine) who helped us in evaluatinghe developed metaphors and interfaces. Special thanks are dueo Dr. Roberto Bellazzi and to his team at the Unit of Nephrol-gy and Dialysis of the Hospital of Mede, PV, Italy, for the kindooperation offered during the application of the framework toanage hemodialysis data. Finally, we acknowledge here the con-

ribution of prof. Luca Chittaro, from the University of Udine, whoontributed to the early stages of this research and suggested useveral improvements to previous versions of this paper.

eferences

[1] Chittaro L. Information visualization and its application to medicine. ArtificialIntelligence in Medicine 2001;22(2):81–8.

[2] Combi C, Keravnou-Papailiou E, Shahar Y. Temporal information systems inmedicine. New York: Springer Publishing Company, Incorporated; 2010.

[3] Aigner W, Miksch S. Carevis: integrated visualization of computerizedprotocols and temporal patient data. Artificial Intelligence in Medicine2006;37(3):203–18.

[4] Aigner W, Miksch S, Müller W, Schumann H, Tominski C. Visual methods foranalyzing time-oriented data. IEEE Transactions on Visualization and ComputerGraphics 2008;14(1):47–60.

[5] Combi C, Portoni L, Pinciroli F. Visualizing temporal clinical data on the WWW.In: Horn W, Shahar Y, Lindberg G, Andreassen S, Wyatt JC, editors. Proceedingsof the joint European conference on artificial intelligence in medicine and med-ical decision making, Vol. 1620 of Lecture Notes in Computer Science. London,UK: Springer-Verlag; 1999. p. 301–14.

[6] Plaisant C, Mushlin R, Snyder A, Li J, Heller D, Shneiderman B, et al. Lifelines:Using visualization to enhance navigation and analysis of patient records. In:Proceedings of the 1998 American Medical Informatic Association annual fallsymposium. Philadelphia, PA, USA: Hanley & Belfus; 1998. p. 76–80.

[7] Chittaro L, Combi C. Visualizing queries on databases of temporal histo-
ries: new metaphors and their evaluation. Data & Knowledge Engineering2003;44(2):239–64.
[8] Chittaro L, Combi C, Trapasso G. Data mining on temporal data: a visualapproach and its clinical application to hemodialysis. Journal of Visual Lan-guages and Computing 2003;14(6):591–620.

[

ce in Medicine 54 (2012) 75– 101

[9] Klimov D, Shahar Y, Taieb-Maimon M. Intelligent visualization and explorationof time-oriented data of multiple patients. Artificial Intelligence in Medicine2010;49(1):11–31.

10] Wang TD, Plaisant C, Shneiderman B, Spring N, Roseman D, Marchand G, et al.Temporal summaries: supporting temporal categorical searching, aggregationand comparison. IEEE Transactions on Visualization and Computer Graphics2009;15(6):1049–56.

11] Shahar Y, Musen MA. Knowledge-based temporal abstraction in clinicaldomains. Artificial Intelligence in Medicine 1996;8(3):267–98.

12] Combi C, Chittaro L. Abstraction on clinical data sequences: an object-orienteddata model and a query language based on the event calculus. Artificial Intelli-gence in Medicine 1999;17(3):271–301.

13] Shahar Y, Cheng C. Model-based visualization of temporal abstractions. Com-putational Intelligence 2000;16(2):279–306.

14] Pane JF, Myers BA. Tabular and textual methods for selecting objects froma group. In: Proceedings of the 2000 IEEE international symposium onvisual languages (VL’00). Washington, DC, USA: IEEE Computer Society; 2000.p. 157–64.

15] Combi C, Franceschet M, Peron A. Representing and reasoning about temporalgranularities. Journal of Logic and Computation 2004;14(1):51–77.

16] Bettini C, Wang XS, Jajodia S. A general framework for time granularity andits application to temporal reasoning. Annals of Mathematics and ArtificialIntelligence 1998;22(1–2):29–58.

17] Bettini C, Wang XS, Jajodia S. Temporal granularity. In: Liu L, Özsu MT, editors.Encyclopedia of database systems. Springer US; 2009. p. 2968–73.

18] Goralwalla IA, Leontiev Y, Özsu MT, Szafron D, Combi C. Temporal gran-ularity: completing the puzzle. Journal of Intelligent Information Systems2001;16(1):41–63.

19] Combi C, Cucchi G, Pinciroli F. Applying object-oriented technologies in model-ing and querying temporally oriented clinical databases dealing with temporalgranularity and indeterminacy. IEEE Transactions on Information Technologyin Biomedicine 1997;1(2):100–27.

20] Combi C, Montanari A, Pozzi G. The t4sql temporal query language. In: SilvaMJ, Laender AHF, Baeza-Yates RA, McGuinness DL, Olstad B, Olsen ØH, FalcãoAO, editors. Proceedings of the sixteenth ACM conference on information andknowledge management. New York, NY, USA: ACM; 2007. p. 193–202.

21] Combi C, Pozzi G. Hmap – a temporal data model managing intervals with dif-ferent granularities and indeterminacy from natural language sentences. VeryLarge Data Bases Journal 2001;9(4):294–311.

22] Shahar Y. A framework for knowledge-based temporal abstraction. ArtificialIntelligence 1997;90(1–2):79–133.

23] Shneiderman B. The eyes have it: a task by data type taxonomy for informationvisualizations. In: Proceedings of the 1996 IEEE symposium on visual languages.IEEE Computer Society; 1996. p. 336–43.

24] Combi C, Pinciroli F, Musazzi G, Ponti C. Managing and displaying different timegranularities of clinical information. In: Ozbolt J, editor. 18th annual sympo-sium on computer applications in medical care. Philadelphia, PA, USA: Hanley& Belfus; 1994. p. 954–8.

25] Cousins SB, Kahn MG. The visual display of temporal information. ArtificialIntelligence in Medicine 1991;3:341–57.

26] Wohlfart E, Aigner W, Bertone A, Miksch S. Comparing information visualiza-tion tools focusing on the temporal dimensions. In: Proceedings of the 200812th international conference information visualisation. Washington, DC, USA:IEEE Computer Society; 2008. p. 69–74.

27] Bueno R, Razente HL, Kaster DS, Barioni MCN, Traina AJM, C.T. Jr. Metric dataanalysis enhanced through temporal visualization. In: Banissi E, Bertschi S,Burkhard RA, Counsell J, Dastbaz M, Eppler MJ, Forsell C, Grinstein GG, Johans-son J, Jern M, Khosrowshahi F, Marchese FT, Maple C, Laing R, Cvek U, Trutschl M,Sarfraz M, Stuart LJ, Ursyn A, Wyeld TG, editors. 14th international conferenceon information visualisation, IV 2010. Washington, DC, USA: IEEE ComputerSociety; 2010. p. 116–21.

28] Noah SA, Yaakob S, Shahar S. Application of information visualization tech-niques in representing patients’ temporal personal history data. In: ZamanHB, Robinson P, Petrou M, Olivier P, Schröder H, Shih TK, editors. Proceedingsof visual informatics: bridging research and practice, first international visualinformatics conference, vol. 5857 of Lecture Notes in Computer Science. 2009.p. 168–79.

29] Ropinski T, Oeltze S, Preim B. Survey of glyph-based visualization techniques forspatial multivariate medical data. Computers & Graphics 2011;35(2):392–401.

30] Aigner W, Miksch S, Schumann H, Tominski C. Visualization of time-orienteddata. London: Springer Publishing Company, Incorporated; 2011.

31] Pinciroli F, Portoni L, Combi C, Violante F. WWW-based access to object-oriented clinical databases: the KHOSPAD project. Computers in Biology andMedicine 1998;28(5):531–52.

32] Aigner W, Miksch S, Thurnher B, Biffl S. Planninglines: novel glyphs for rep-resenting temporal uncertainties and their evaluation. In: 9th internationalconference on information visualisation, IV. Washington, DC, USA: IEEE Com-puter Society; 2005. p. 457–63.

33] Plaisant C, Milash B, Rose A, Widoff S, Shneiderman B. Lifelines: visualizingpersonal histories. In: Proceedings of the SIGCHI conference on Human factorsin computing systems: common ground, CHI ’96. ACM; 1996. p. 221–7.

34] Tatu A, Albuquerque G, Eisemann M, Schneidewind J, Theisel H, Magnor MA,et al. Combining automated analysis and visualization techniques for effectiveexploration of high-dimensional data. In: Proceedings of the IEEE symposiumon visual analytics science and technology. IEEE VAST 2009, part of VisWeek2009. Washington, DC, USA: IEEE Computer Society.

lligen

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[


35] Woodring J, Shen H-W. Multiscale time activity data exploration via temporalclustering visualization spreadsheet. IEEE Transactions on Visualization andComputer Graphics 2009;15(1):123–37.

36] Javed W, Elmqvist N. Stack zooming for multi-focus interaction in time-seriesdata visualization. In: IEEE pacific visualization symposium PacificVis 2010.Washington, DC, USA: IEEE Computer Society; 2010.

37] Hinum K, Miksch S, Aigner W, Ohmann S, Popow C, Pohl M, et al. Gravi++: inter-active information visualization to explore highly structured temporal data.Journal of Universal Computer Science 2005;11(11):1792–805.

38] Lowe A, Jones RW, Harrison MJ. The graphical presentation of decision sup-port information in an intelligent anaesthesia monitor. Artificial Intelligence inMedicine 2001;22(2):173–91.

39] Aigner W, Miksch S, Müller W, Schumann H, Tominski C. Visualizing time-oriented data – a systematic view. Computers & Graphics 2007;31(3):401–9.

40] Aigner W, Miksch S, Müller W, Schumann H, Tominski C. Visual methods foranalyzing time-oriented data. IEEE Transactions on Visualization and ComputerGraphics 2008;14(1):47–60.

41] Shahar Y, Goren-Bar D, Boaz D, Tahan G. Distributed, intelligent, interactivevisualization and exploration of time-oriented clinical data and their abstrac-tions. Artificial Intelligence in Medicine 2006;38(2):115–35.

42] Klimov D, Shahar Y, Taieb-Maimon M. Intelligent selection and retrieval ofmultiple time-oriented records. Journal of Intelligent Information Systems2010;35(2):261–300.

43] Wang TD, Plaisant C, Quinn AJ, Stanchak R, Murphy S, Shneiderman B. Align-ing temporal data by sentinel events: discovering patterns in electronic healthrecords. In: Czerwinski M, Lund AM, Tan DS, editors. Proceedings of the 2008conference on human factors in computing systems. New York, NY, USA: ACM;2008. p. 457–66.

44] Bade R, Schlechtweg S, Miksch S. Connecting time-oriented data and informa-tion to a coherent interactive visualization. In: Dykstra-Erickson E, Tscheligi M,editors. Proceedings of the 2004 conference on human factors in computingsystems. New York, NY, USA: ACM; 2004. p. 105–12.

45] Hochheiser H, Shneiderman B. Dynamic query tools for time series datasets: timebox widgets for interactive exploration. Information Visualization2004;3(1):1–18.

46] Fails JA, Karlson AK, Shahamat L, Shneiderman B. A visual interface for multi-variate temporal data: finding patterns of events across multiple histories. In:Wong PC, Keim DA, editors. IEEE symposium on visual analytics science andtechnology. IEEE VAST 2006. Washington, DC, USA: IEEE Computer Society;2006. p. 167–74.

47] Wongsuphasawat K, Shneiderman B. Finding comparable temporal categoricalrecords: a similarity measure with an interactive visualization. In: Proceedingsof the IEEE symposium on visual analytics science and technology. IEEE VAST2009. 2009. p. 27–34.

48] Wang TD, Wongsuphasawat K, Plaisant C, Shneiderman B. Visual informationseeking in multiple electronic health records: design recommendations and aprocess model. In: Veinot TC, C atalyürek ÜV, Luo G, Andrade H, Smalheiser NR,editors. Proceedings of the ACM international health informatics symposium.New York, NY, USA: ACM Press; 2010. p. 46–55.

49] Wang TD, Deshpande A, Shneiderman B. A temporal patternsearch algorithm for personal history event visualization. IEEETransactions on Knowledge and Data Engineering 99 (PrePrints).doi:http://doi.ieeecomputersociety.org/10.1109/TKDE.2010.257.

50] Lammarsch T, Aigner W, Bertone A, Gärtner J, Mayr E, Miksch S, et al. Hier-archical temporal patterns and interactive aggregated views for pixel-basedvisualizations. In: Banissi E, Stuart LJ, Wyeld TG, Jern M, Andrienko GL, MemonN, Alhajj R, Burkhard RA, Grinstein GG, Groth DP, Ursyn A, Johansson J, ForsellC, Cvek U, Trutschl M, Marchese FT, Maple C, Cowell AJ, Moere AV, editors.13th international conference on information visualisation, IV. Washington,DC, USA: IEEE Computer Society; 2009. p. 44–50.

51] Lee T-Y, Shen H-W. Visualization and exploration of temporal trend relation-ships in multivariate time-varying data. IEEE Transactions on Visualization andComputer Graphics 2009;15(6):1359–66.

52] Dudzic S, Godwin JA, Kilgore RM. Visualization of temporal relationships within
coordinated views. In: Proceedings of the IEEE conference on visual analyticsscience and technology. IEEE VAST 2010. Washington, DC, USA: IEEE ComputerSociety; 2010. p. 219–20.
53] Jin J, Szekely PA. Interactive querying of temporal data using a comic stripmetaphor. In: Proceedings of the IEEE conference on visual analytics science

[

[

ce in Medicine 54 (2012) 75– 101 101

and technology. IEEE VAST 2010, part of VisWeek 2010. Washington, DC, USA:IEEE Computer Society.

54] Chu WW, Hsu C-C, Cardenas AF, Taira RK. Knowledge-based image retrievalwith spatial and temporal constructs. IEEE Transactions on Knowledge andData Engineering 1998;10(6):872–88.

55] Dionisio JDN, Cardenas AF. Unified data model for representing multimedia,timeline, and simulation data. IEEE Transactions on Knowledge and Data Engi-neering 1998;10(5):746–67.

56] Dionisio JDN, Cardenas AF. Mquery: a visual query language for multime-dia, timeline and simulation data. Journal of Visual Languages and Computing1996;7(4):377–401.

57] Cardenas AF, Ieong IT, Taira RK, Barker R, Breant CM. The knowledge-basedobject-oriented picquery+language. IEEE Transactions on Knowledge and DataEngineering 1993;5(4):644–57.

58] Bui AAT, Aberle DR, Kangarloo H. Timeline: visualizing integrated patientrecords. IEEE Transactions on Information Technology in Biomedicine2007;11(4):462–73.

59] Aoyama DA, Hsiao J-TT, Cardenas AF, Pon RK. Timeline and visualization ofmultiple-data sets and the visualization querying challenge. Journal of VisualLanguages and Computing 2007;18(1):1–21.

60] Ding J, Hughes LM, Berleant D, Fulmer AW, Wurtele ES. Pubmed assistant:a biologist-friendly interface for enhanced pubmed search. Bioinformatics2006;22(3):378–80.

61] Murray N, Paton NW, Goble CA. Kaleidoquery: a visual query language for objectdatabases. In: Catarci T, Costabile MF, Santucci G, Tarantino L, editors. Proceed-ings of the working conference on advanced visual interfaces 1998. New York,NY, USA: ACM Press; 1998. p. 247–57.

62] Polyviou S, Samaras G, Evripidou P. A relationally complete visual query lan-guage for heterogeneous data sources and pervasive querying. In: Proceedingsof the 21st international conference on data engineering, ICDE. Washington,DC, USA: IEEE Computer Society; 2005. p. 471–82.

63] Jones S, McInnes S, Staveley MS. A graphical user interface for booleanquery specification. International Journal on Digital Libraries 1999;2(2–3):207–23.

64] Spoerri A. Infocrystal: a visual tool for information retrieval & management.In: Bhargava BK, Finin TW, Yesha Y, editors. Proceedings of the second interna-tional conference on information and knowledge management. New York, NY,USA: ACM Press; 1993. p. 11–20.

65] Huo J. KMVQL: a visual query interface based on karnaugh map. In: Levialdi S,editor. Proceedings of the working conference on advanced visual interfaces.New York, NY, USA: ACM Press; 2008. p. 243–50.

66] Paolino L, Sebillo M, Tortora G, Vitiello G. The predicate tree – a metaphorfor visually describing complex boolean queries. In: Qiu G, Leung C, Xue X,Laurini R, editors. Proceedings of the 9th international conference on advancesin visual information systems, vol. 4781 of Lecture Notes in Computer Science.Berlin/Heidelberg: Springer-Verlag; 2007. p. 524–36.

67] Huo J, Cowan W. Comprehending boolean queries. In: Creem-Regehr SH,Myszkowski K, editors. Proceedings of the 5th symposium on applied percep-tion in graphics and visualization. ACM international conference proceedingseries. New York, NY, USA: ACM; 2008. p. 179–86.

68] Chui M, Dillon A. Speed and accuracy using four boolean query systems. In: PrissU, editor. Tenth midwest artificial intelligence and cognitive science conference(MAICS 99). CA, USA: AAAI Press; 1999. p. 36–42.

69] Bettini C, Wang XS, Jajodia S. Solving multi-granularity tem-poral constraint networks. Artificial Intelligence 2002;140(1/2):107–52.

70] Cormen TH, Leiserson CE, Rivest RL. Introduction to algorithms. Cambridge:The MIT Press/McGraw-Hill Book Company; 1989.

71] Booch G, Rumbaugh J, Jacobson I. The unified modeling language referencemanual. Boston, MA: Addison-Wesley; 2004.

72] Chin JP, Diehl VA, Norman KL. Development of an instrument measuring usersatisfaction of the human–computer interface. In: Proceedings of the SIGCHIconference on Human factors in computing systems. New York, NY, USA: ACM;1988. p. 213–8.

73] Golfarelli M, Rizzi S. A survey on temporal data warehousing. InternationalJournal of Data Warehousing and Mining 2009;5(1):1–17.

74] Bellazzi R, Larizza C, Magni P, Bellazzi R. Temporal data mining for the qual-ity assessment of hemodialysis services. Artificial Intelligence in Medicine2005;34(1):25–39.

visually defining and querying consistent multi-granular clinical temporal abstractions

Documents