Merging multimedia presentations andsemistructured temporal data: a graph-basedmodel and its application to clinical information

Artificial Intelligence in Medicine (2005) 34, 89—112

http://www.intl.elsevierhealth.com/journals/aiim

Carlo Combi �, Barbara Oliboni, Rosalba Rossato

Dipartimento di Informatica, Universita degli studi di Verona, Ca’ Vignal 2, Strada le Grazie 15,I-37134 Verona, Italy

Received 3 November 2003; received in revised form 15 November 2004; accepted 22 November 2004

KEYWORDSTemporal clinical data;Semistructured data;Multimedia;Multimedia

presentations;Data model;Cardiology data

Summary

Objective: In this paper, we focus on the issue of providing physicians with thecapability of representing in a seamless way both temporal aspects of multimediasemistructured data and their temporal presentation requirements.Background: Semistructured data are data having some structure, that may beirregular or incomplete and does not necessarily conform to a fixed schema. Semi-structured data often contain the description of histories of the considered realworld. The eXtensible Markup Language (XML) is becoming a cross compatible andstandardized means for representing semistructured clinical data. In the field ofmedical informatics, there are many ongoing activities concerning XML. In the field ofmultimedia database systems, the topic related to the integration of several mediaobjects (with their temporal aspects) have been considered both for data modelingand querying issues and for modeling multimedia presentations.Methodology: We first propose the Multimedia Temporal Graphical Model (MTGM), byrepresenting a clinical database for cardiology patients undergoing cardiac angio-graphies and then describe it in a formal way. We deal with the problem of expressingMTGM data by XML and of managing MTGM clinical data through an XML-based system.We provide both a technique for translating (a part of) an MTGM database into an XMLdocument and some techniques allowing us to obtain presentations defined by meansof the Synchronized Multimedia Integration Language (SMIL) from MTGM presenta-tions.Results: MTGM allows one to represent and store clinical information in a semistruc-tured, temporal, and multimedia database. The physician can define multimediapresentations based on the stored data. Multimedia presentations are then stored inthe same MTGM database together with temporal clinical information and are thus

� Corresponding author. Tel.: +39 045 802 7985; fax: +39 045 802 7068.E-mail address: carlo.combi@univr.it (C. Combi).

0933-3657/$ — see front matter # 2004 Elsevier B.V. All rights reserved.doi:10.1016/j.artmed.2004.11.003

90 C. Combi et al.

represented according to the same data model. A prototype based on an XML nativedatabase system has been designed and implemented.Discussion and conclusions: In this work we have considered the theoretical andmethodological issues concerning the definition of a general data model for describingtemporal and multimedia features of semistructured clinical information. Otherresearch and application oriented features, which have not been considered in MTGM,could be investigated for completing MTGM with regard to its applicability to clinicaldomains: MTGM does not allow one to express times at different levels of granula-rities, i.e. with different time units, or with indeterminacy; besides the consideredvalid time, it could be interesting to manage also other temporal dimensions such asthe transaction and availability times. Besides being useful for managing multimediadata stored according to widely accepted standards as MPEG and DICOM, nowadayssemistructured data, and XML in particular, are becoming the most important way forexpressing and exchanging medical knowledge and data: MTGM can be considered as adata model allowing the seamless representation of both (multimedia and temporal)clinical data and knowledge.# 2004 Elsevier B.V. All rights reserved.

1. Introduction

During the last years, the amount of multimediaclinical data (as, for example, magnetic resonanceand computed tomography images, echographies,angiographies, related reports, medical records,and so on) available electronically has been growingup and the need of an integrated management ofthese heterogeneous data is widely acknowledged[1,2]. The above-mentioned clinical data are het-erogeneous both (i) for the software tools which areneeded to access/display them, and (ii) for thestructure according which they are interpreted/encoded. Indeed, multimedia data are accessedby physicians through different systems, like webbrowsers, database systems usually supporting SQL-based query languages, application-specific inter-faces, or general-purpose multimedia systems,which can support different data exchange formats[2—6]. On the other hand, the structure of multi-media clinical data can be irregular: this informa-tion can consist either of images, sounds, andmovies or of structured reports or of documentshaving different and partial structures, as thosecoming from the Web.

All this kind of information must be suitablymanaged in a seamless way, as it is used in mostknowledge-based clinical activities [1,2,7]. To thisend several different research topics should beconsidered and suitably integrated:

� M

odeling semistructured clinical information.Besides structured and raw data, there is a lotof information which is represented by semistruc-tured data [8]: indeed, sometimes the structureexists but has to be extracted from the data, asfor clinical information published on the Web; inother cases the structure is left implicit, as in the

case of discharge letters, which follow a prede-fined structure, but without any explicit refer-ence to it; in other cases, structured information,as that from clinical databases, has to be mergedwith other information structured according todifferent schemata. In general, we can say thatsemistructured data have a structure which canbe partial (i.e., not completely specified), impli-cit (i.e., represented within the data), and irre-gular (i.e., specific for any different data item).To this regard, the eXtensible Mark-up Language(XML) is spreading out as a general format forrepresenting, exchanging, and publishing infor-mation on the Web and more generally as astandard for representing semistructured data[8,9]. As for clinical data, XML has been exten-sively considered as a means for data exchangeamong clinical applications [7,10,11]; consideringmore specifically medical knowledge-based sys-tems, XML has been proposed for the specifica-tion, through a standard language, of widelyaccepted medical ontologies and taxonomies[12], and for the definition of suitable languagesfor clinical domains [13,14]. Thus, the interest inXML-related technologies and methodologies inthe medical informatics community can be eval-uated as relevant; nevertheless, several theore-tical and methodological issues related to theadoption of semistructured data models for med-ical data have not yet been completely consid-ered with the same accuracy used in recent pastyears for relational and object-oriented datamodels [1,2,15—17].

� R

epresenting multimedia and temporal aspects inclinical databases. In several clinical areas, multi-media and temporal information are often pre-sent and intertwined, such as in the descriptionsof patient symptoms, in angiographic and echo-

Merging multimedia presentations and semistructured temporal data 91

graphic movies, in X-ray images, in the clinicalparameters periodically acquired, and need to bemanaged in a homogeneous way. Indeed, theobjects stored in a database often represent realclinical entities that are characterized by tem-poral aspects: for example, it could be importantto describe the interval during which an angio-graphy movie has been acquired or the intervalduring which a vocal annotation, associated to theparticular clinical situation highlighted by themovie, has been recorded. Multimedia and tem-poral aspects of medical information have beenstudied in some details and suitable data models,query languages, and systems have been proposedand applied to several clinical domains as cardi-ology [2,15,18], radiology, and oncology [1]. Theconsidered temporal dimension is the valid time(VT), which is generally defined as the time atwhich a fact is true in the considered domain [19].

� A

uthoring and modeling multimedia presenta-tions. Multimedia data need often to be presentedaccording to several spatial and temporal rela-tionships, specified during the presentationdesign [20]. A multimedia presentation could bedefined, for example, for medical students, dis-playing all the X-ray images showing pelvis frac-tures; it could be required that during thevisualizations of the frames with specific detailsof the fracture, both the audio and a text in aspecified position of the screen enrich thedescription of the fracture; moreover, it couldbe required that the text must disappear fromthe screen as the visualized frames do not repre-sent the detail related to the text content. In thisresearch area, several proposals deal with thespecification of synchronization constraintsbetween different multimedia data items as wellas with the definition of suitable tools (i) allowingthe specification of complex presentations, (ii)verifying the consistency of the designed presen-tation, and (iii) playing the presentations accord-ing to the specified synchronization relationships[20]. As for multimedia presentations, it is impor-tant to mention here the Synchronized Multime-dia Integration Language (SMIL), a new XML-basedlanguage specifically designed for defining distrib-uted multimedia presentations [21].

With respect to the above issues, the first goal ofthis paper is to focus on the definition of a generaldata model for representing both complex semi-structured information, having multimedia and te-mporal features, and the related multimediapresentations in a seamless way. More specifically,the main aims of this paper regarding the modelingissues are:

� d

efining a general data model, based on labeledgraphs, for representing semistructured clinicalinformation having multimedia and temporal fea-tures;

� r

epresenting in a seamless way both temporalaspects of multimedia data and their temporalpresentation requirements;

� p

roposing some techniques for the managementof semistructured, multimedia, and temporaldatabases through XML database systems.

The data model, we will describe in this paper, isnamed Multimedia Temporal Graphical Model (MTG-M), and is based on the Temporal Graphical Model(TGM) presented in [22]. MTGM allows one to repre-sent semistructured, temporal,andmultimediadata;multimedia presentations based on multimedia obj-ects stored into the database are managed by MTGM,too. MTGM can possibly be considered as a logicalmodel for semistructured data: to complete the pic-ture for its application to clinical data, we show thatMTGM databases can be suitably translated into XMLdocuments, which can be managed either by XML-enabled database systems [23—26] or by XML-nativedatabase systems [27—29]. Moreover, SMIL presenta-tions can be derived from MTGM presentations.

The second goal of this paper is related to themanagement of real world clinical data through ourapproach. To this end, as a proof of concept, weconsidered the clinical information about patientsundergoing cardiac angioplasty [30], according totwo different perspectives:

� m

odeling multimedia temporal clinical data andrelated presentations through MTGM;

� m

anaging a semistructured clinical databasethrough a complete prototype system designedaccording to the previously defined approach,based on an XML native database system.

The structure of the paper is as follows. In Section2 we describe some interesting proposals, dealingwith temporal aspects of semistructured data, mul-timedia data presentations, and the management ofmedical information using XML. Section 3 introducesthe main features of MTGM, by representing a clin-ical database for cardiology patients undergoingcardiac angiographies; the formal definition of ourgraphical data model is given in Section 4. In Section5 we deal with the problem of expressing MTGM databy XML and of managing MTGM clinical data throughan XML-based system. Section 6 contains some finaloutlines, some discussion about the limitations ofMTGM, its applicability to different clinical domai-ns, and its integration with other technologies spe-cifically oriented to the medical domain; some

92 C. Combi et al.

sketches on future research directions conclude thesection.

2. Related work

In this section we summarize some proposals pre-sented in the literature for representing semistruc-tured data and their temporal dimension, describesome work on XML-based applications for managingmedical information, and report some proposals forrepresenting multimedia databases and multimediapresentations.

2.1. Semistructured temporal data

Semistructured data are data having some struc-ture, that may be irregular or incomplete and doesnot necessarily conform to a fixed schema [8]. Thestructure and the content of semistructured datacan change with the time;moreover, semistructureddata can contain the description of histories of theconsidered real world. For these reasons, also forthis type of information it is important to considerthe temporal aspect of data.

In this context, the models proposed in the lit-erature allow one to represent and query changes insemistructured data. Among these, we cite here theDelta Object Exchange Model (DOEM) [31], a tem-poral data model representing the transaction time[19] (i.e. the time at which an information is storedin the database), and TGM [22], a graph-based datamodel, which is able to represent static and dynamicaspects of semistructured data, considering thealready mentioned valid time [19].

It is usual for semistructured data models torepresent data as labeled graphs, where nodesand edges suitably represent different (related)data items. In particular, a TGM graph is a labeledgraph with two types of nodes: complex nodes,which describe the objects of the represented rea-lity, and atomic nodes, which represent propertiesof the objects. The relationships between objects inthe reality are represented by labeled relationaledges in the graph.

2.2. XML and clinical information

Due to the fast evolution of Internet, the number ofXML-based applications is growing up rapidly. In thefield of medical informatics, there are many ongoingactivities concerning XML. The increasing interest inXML as a cross compatible and standardized meansfor representing semistructured clinical data has ledto new developments. In most of these works, XML isused to specify the (often partial and implicit)

structure and the semantics of clinical documentsfor the purposes of sharing and exchanging clinicalinformation [7,10,12—14,32—36].

In [34], the authors examine some potential usesof XML in the healthcare industry, and present alsoseveral high-level areas in which either an XML-based protocol or an XML-based data standard wouldprove to be useful in the healthcare field. In [13] ispresented an XML-based patient record architec-ture, to create a common data architecture forthe interoperability of healthcare documents. Inthe radiology field, one of the main goals is tocommunicate imaging information to aid in patientmanagement. Information standards facilitate com-munication and help to realize this goal. An XMLnotation provides a compact document representa-tion schema; in this way, complex radiologicalreports can be transmitted over the Web as univer-sally understandable, self-defining documents [7].

XML can also be used to represent reports deriv-ing from decision support systems. In [14], the use ofXML is explored to mediate between components ofthe computer-based patient record. The authorssought to integrate existing Web-based systemsfor structured reporting (SPIDER) and a probabilisticdecision support (BANTER). From the data enteredthrough the Web interface, SPIDER can create tex-tual reports or XML documents. The XML documentsinclude a report-specific Document Type Definition,that defines the allowable data fields and values;the resulting report is thus both portable and self-defining [11].

Another important use of XML is related to therepresentation of complex linguistic processingstages [12] and, in particular, of the heterogeneousknowledge contained in practice guidelines [35]. In[12] XML is used for the analysis of a medical lan-guage which involves a number of complex linguisticprocessing stages. The main aim of this work is toacquire lexical semantics information from MedLine(the collection of biomedical abstracts maintainedand supported by the U.S. National Library of Med-icine) through parsing. In [35], the authors havedeveloped the Guideline Elements Model usingXML to better represent the heterogeneous knowl-edge contained in practice guidelines. The devel-oped approach has been used to representknowledge deriving from guidelines that addressdifferent topics, including diagnosis of appendicitis,risks of hypercholesterolemia, indication for coron-ary artery bypass graft, and management of asthma.

2.3. Multimedia temporal information

In the field of multimedia database systems, thetopics related to the integration of several media

Merging multimedia presentations and semistructured temporal data 93

objects (with their temporal aspects) have beenconsidered in several works. Each of them high-lighted different aspects of the problem dependingon the modeling needs and the focus of the con-sidered applications: some works consider only sta-tic images [37], while others consider also video[1,4,38], and even audio data [1]; some data modelsallow the association between visual data and sim-ple unstructured notes [39], while some other datamodels consider the capability of modeling informa-tion related to visual data as structured entities[1,38]. Moreover, some proposals are based on (gen-eric or ad-hoc) temporal data models, allowing oneto manage temporal aspects related to multimediadata [1,4,37].

2.4. Multimedia presentations

A multimedia presentation can be defined as thecomposition of a set of multimedia objects whichhave to satisfy some given spatio-temporal con-straints during their playback: several proposalsdeal with the specification of synchronization con-straints between different multimedia data items[20]. For example, in [40] the authors present amethodological approach for checking the temporalintegrity of interactive multimedia documents,focusing on temporal properties of presentations.In [41], a system is presented, which is able to checkinteractively the consistency of a presentation, dur-ing its definition. Consistency is checked both for(synchronous and asynchronous) spatial and tem-poral requirements.

3. MTGM at work

In this section we describe the main features ofMTGM, by showing how it works with a real, evensimple, example taken from a clinical scenario. Inparticular, we will represent a database containinginformation on cardiology patients, undergoing car-diac angiographies. In MTGM the considered timedimension is the VT.

3.1. An MTGM clinical database

Cardiac angiography is a technique adopted to studythe condition of coronary vessels (coronary angiogra-phy) and the heart functionalities (left ventriculo-graphy). The result of a cardiac angiography consistsof an X-raymovie, displaying, in different parts, bothheart and coronary vessels functionalities [30].

Cardiac angiography is, for example, performedwith the purpose of defining the coronary anatomyand the degree of luminal obstruction of the cor-

onary arteries. It is mainly used to diagnose thepresence of coronary artery disease (CAD) inpatients which present typical CAD-related symp-toms.

In this context,weanalyze theclinical informationabout patients undergoing cardiac angioplasty: cor-onary revascularization is performed to reinstateblood supply by a catheter fitted with a small infla-table balloon placed in the coronary artery affectedby stenosis (a partial obstruction of the vessel); bysuccessive inflations, the catheter blows up andcauses the vessel to dilate and letmoreblood through(percutaneous transluminal coronary artery angio-plasty). These patients, after the angioplasty, arefollowed up to immediately manage further pro-blems, such as re-stenoses, of the patients them-selves [42].

The information collected for each patient are:demographic data, data on risk factors, diagnoses,therapies, data acquired during the follow-up visits,multimedia data acquired during the angiocardio-graphic exams and the related clinical data. Let usnowconsider anexampleof a patientmedical record.

Example 1. On October 10, 2001, at 10:00 a.m.the physician visits for the first time Ed Bawer whobecomes, from this moment, his/her patient. After-wards, Ed Bawer reports that from October 15, 2001,at 8:40 a.m. to October 15, 2001, at 11:20 a.m. andfrom October 15, 2001, at 4:00 p.m. to October 15,2001, at 4:50 p.m. he suffered from light chesttrouble and on October 22, 2001, at 10:00 a.m. thephysician diagnoses this symptom as related to CAD.From October 22, 2001, at 10:30 a.m to October 22,2001, at 12:45 Ed Bawer undergoes a cardiac angio-plasty. Clinical parameters, as well as data related tothe evaluation of the stenosis observed during thecoronarography, are acquired during the angioplasty.The diagnosis of the patient is CAD pathology and, forthis reason, the physicianprescribes to thepatient ananticoagulating drug (Warfarin).

Figs. 1 and 2 show respectively the schema andthe instance of the MTGM database containing infor-mation about the patient and his clinical informa-tion. As usually in semistructured data models, (partof) the schema and (part of) the instance of adatabase are represented in a similar way throughgraphs. More particularly an MTGM graph is a direc-ted, labeled graph, with a single root.

MTGM has complex nodes and atomic nodes; inthe example, shown in Fig. 2, we have 10 complexnodes (depicted as rectangles) and twenty-fouratomic nodes (depicted as ovals). Complex nodesare Patient, RiskFactor, Symptom, Diagnosis, Ther-apy, AngioVisit, CoronaroExam, ExamReport and

94 C. Combi et al.

Figure 1 A portion of the MTGM database schema.

Merging multimedia presentations and semistructured temporal data 95

Figure 2 A portion of an MTGM database.

96 C. Combi et al.

Stenosis and represent entities, while atomic nodesare, for example, Name, Surname, Pathology, andDrug and represent attributes of entities. Moreover,MTGM has the specific node type stream: streamnodes contain multimedia information as unstruc-tured texts, movies, and sounds; they are depictedas thick ovals and are a particular kind of atomicnode. For example, in Fig. 2 the atomic nodeStreamFile (property of the node CoronaroExam)is a stream node and contains the string‘‘xa12.mpg’’, which refers to the file that encodesthe movie of the coronarography of the patient.

Nodes are connected through labeled edges hav-ing a direction (graphically specified by an arrow),called relational edges. Relational edges connectingcomplex nodes to their atomic nodes have labelscontaining the name ‘‘HasProperty’’. Complex nodesare related to their streamnodes by edges having thelabel containing the name ‘‘HasMMProperty’’.

VT is explicitly managed by MTGM both for nodesand edges. The VT1 of a complex node is representedin its label. The label of a relational edge is com-posed by the name of the relationship and its VT. TheVTof an atomic (stream) node is represented in thelabel of the edge between the atomic node and itsparent. For example, as shown in Fig. 2, VT of thecomplex node Patient is [10/10/01 10:00, now]where the special value ‘‘now’’ indicates (accordingto the standard meaning given by the temporaldatabase community [43]) that the represented factis currently true (i.e., Ed Bawer is still a patient),while the VT of the atomic node SBP is [22/10/0111:30, 22/10/01 11:30] (i.e., the SBP was 190 mmHgfrom a measurement performed on the specifiedinterval). As reported in Fig. 2, the label of therelational edge between Patient and Therapy ishPatient � Therapy; ½23=10=0110 : 00; now�i andrepresents the fact that the therapy with Warfarinfor Ed Bawer started on October 23, 2001 and is stillongoing. In general, the VT of a complex node andthose of its ingoing relational edges can be differ-ent: for example, we could represent the fact that apatient stops a prescribed therapy by a VT on therelational edge starting together with and endingbefore the VT of the connected node Therapy.

Even though values for VT are inserted by theusers, some constraints exist between VTs of con-nected nodes and edges in an MTGM graph. These(few) constraints are related to some basic consis-tency in the description of the considered part ofreal world through an MTGM graph: for example,considering the graph depicted in Fig. 2, it would notbe meaningful to have the node AngioVisit con-

1 Note that in the MTGM graph, time-related information arerepresented in the format DD=MM=YY HH : mm.

nected to the atomic node SBP with an edgeHasProperty having a VT preceding the VT of Angio-Visit. Indeed, we suppose that a complex object(represented by a complex node) is related to itsproperties (atomic nodes) while it is valid. Thus, ingeneral, the VT of edges HasProperty and HasMM-Property cannot start before and cannot end afterthe VT of the complex node, they originate from.

Besides these basic constraints for VTs of nodesand edges, several other domain-dependent con-straints could be defined for VTof specific nodes andedges. For example, let us assume that, besides thenode Patient depicted in Fig. 2, the MTGM graphcontains several nodes Patient, referring to otherpatients; in this case, we could have that severalnodes Patient are connected by several edgesPatient-Diagnosis to the same node Diagnosis, con-taining the description ‘‘CAD’’ for the pathology. Inthis case, it is quite obvious to require that the VTofeach edge Patient-Diagnosis must be contained intothe VTof the pointed node Diagnosis (in other words,the VT of each edge Patient-Diagnosis must be dur-ing the VTof the pointed Diagnosis). In Section 4, wewill present in a more formal way the basic temporalconstraints between VTs of MTGM nodes and edges,while the definition and the management of user-defined domain-dependent temporal constraints forMTGM graphs are beyond the scope of this paper andis left for following investigations.

In anMTGMgraph, besides the name ‘‘HasMMProp-erty’’, the label of the edge relating a complex nodeto a stream node contains the VTof the relationship,and the specific subpart of the stream object thecomplex node is related to. For example, as reportedin Fig. 2, the label of the edge between Stenosis withVT [22/10/01 11:30, 22/10/01 12:45] (i.e., the noderepresenting the stenosis after the intervention), andthe stream node StreamFile is hHasMMProperty, [22/10/01 11:35, 22/10/01 11:36], [10,50] i. This edgerepresents the fact that the subpart of the movie ofthe overall angiography associated to the Coronar-oExam and represented by the stream node Stream-File with value ‘‘xa12.mpg’’, is the sequence offrames 10; 11; . . . ; 50. The VT of the HasMMPropertyedge is the time during which the subpart of themovie has been recorded.

In general, the considered subpart of a streamnode is specified according to the type of the repre-sented stream object. For example, if we consider amovie, the subpart is described by [ frame_s, fra-me_e],2 where frame_s and frame_e are respec-tively the first and the last frame of the playedsequence, while for an audio the subpart is

2 The special frame interval ½0; infinity� denotes that all themovie is associated to the related complex node (see Fig. 2).

Merging multimedia presentations and semistructured temporal data 97

described by [time_s, time_e], where time_s andtime_e are respectively the start time and the endtime (expressed in seconds) of the considered sub-part of the audio. In the case of an image, thesubpart is specified by [x; y;width; height] thatrepresents the portion of the image with origin in(x, y) and dimensions width and height. Differenttechniques for identifying subparts into a streamnode can be easily supported, such as for compositestream objects consisting of DICOM files [44], con-taining radiological images and related patientinformation.

3.2. Defining multimedia presentations onclinical data

As previously outlined, stored data need to be pre-sented according to different requirements andpoints of view [2]: to this end, MTGM allows us tocompose a multimedia presentation starting fromthe nodes stored in an MTGM database. Storedmultimedia information can be used in differentways: as an example, we could suppose that physi-cians are interested in the composition of videos andother clinical information about some patients intoa unique presentation summarizing the most impor-tant cases. On the other hand, it could be requiredthat a different presentation is composed, based onsome information already present in the previouspresentation, dealing only with the information of asingle given patient: for example, with respect tothe data of Example 3.1, let us suppose to define, forthe given patient, a multimedia presentation con-sisting of three parts.

� T

he first part contains the name of the patient, hissurname, the birth date, the sex of the patient,and the risk type. All these data are shown simul-taneously.

� T

he second part reports the complete descriptionof the clinical data for the patient showing thediagnosed pathology, the prescribed drug, and theassumption modality (represented by the atomicnodes Amount and Frequency). These data areshown simultaneously.

� F

inally, the name and surname of the patient, theparameter values acquired during the visit(weight, heart-rate, SBP, and DBP) are shownsimultaneously with the complete description innatural language of the coronarography (repre-sented by the stream node StreamFile with value‘‘report.rt’’) and with the portion of the coronar-ography video (represented by the stream nodeStreamFile with value ‘‘xa12.mpg’’ and child ofthe complex node Stenosis) that reveals the pre-sence of stenosis before the angioplasty. The

playback of the portion of the video must berepeated four times; the portion of the videoand the report are to be shown 5 s after thebeginning of the visualization of the other infor-mation.

In order to define this multimedia presentation,we have to solve some specific issues, due to the factthat (i) each presentation is based on data storedinto a database, and (ii) the presentation itself hasto be stored into the same semistructured database.

3.2.1. Storing presentations as semistructureddataIn an MTGM database, several multimedia presenta-tions can be stored: thus, we need to identify eachdefined presentation involving clinical data. To thispurpose, we introduce a special complex node,named Presentation, which represents the startingpoint of each presentation and has an atomic nodePresName representing its unique name withrespect to the considered MTGM database. Thisway, each presentation stored in the MTGM data-base is identified by its name. Moreover, the value ofPresName will be used to identify all the edgeswhich will connect the objects composing the pre-sentation itself. To this regard, each complex nodePresentation is a specific access point to the semi-structured database, besides the (unique) root ofthe MTGM database, with respect to objects belong-ing to the considered presentation.

The Presentation node is connected by means ofan edge to the complex node containing the firstmedia object of the presentation: a media object isany atomic or stream node, which is part of a multi-media presentation. Fig. 4 depicts the nodes and theedges of the overall MTGM graph related to theconsidered presentation, having name ‘‘P1’’. Thicklines highlight the (multimedia) nodes and edgescomposing a multimedia presentation.

A piece of multimedia information, representedby a media object, can be inserted several times inthe same presentation and, in each visualization ofthe same media object, different spatial and tem-poral coordinates can be required. Thus, we need torecognize different instances of the same mediaobject for a given presentation, in order to specifythe right coordinates. To this regard, we introducethe concept of media presentation identifier (mpi).For example, in the presentation previouslydefined, the object Name appears in the first andin the last part; we could require that in the firstpart it has to be shown in the center of the screen,while in the last one it has to be visualized in theupper part of the screen. In this case the objectName has two instances: the first has mpi ¼ 1 and

98 C. Combi et al.

Figure 3 The timeline of the multimedia presentationwith its mpis.

3 We assume that each setting node has a unique ingoing edgeand, for this reason, in Fig. 4, there are, for example, two settingnodes Type related to the object Name by means of two differentedges with label h P1,1 i and h P1, 10 i, respectively.

the second hasmpi ¼ 10. Fig. 3 depicts the timelineof the considered presentation together with theassigned mpis.

An mpi identifies a specific instance of a mediaobjectwithin a presentation and, thus, it allows us todescribe the suitable information related to eachplayback of a media object in a particular multi-media presentation. More precisely, the media pre-sentation identifiers are ordered according to thestarting time of the corresponding playback. In thefollowing, we will use the ordered set of naturalnumbers to specifympis: for example, if we considerthe timeline depicted in Fig. 3, we can see that theobjects Name, Surname, Birthday, Sex, and RiskTypeare shown in the first part of the multimedia pre-sentation.We assign the value 1 to thempi related tothe object Name; the othermpis have values 2, 3, 4,and 5, respectively. In the second part of the con-sidered multimedia presentation, the objectsPathology, Drug, Amount, and Frequency are shownsimultaneously; thuswe assign the value 6 to thempicorresponding to the object Pathology, and thevalues 7, 8, and 9 to the other objects.

3.2.2. Specifying spatio-temporal settingsTogether with mpis, we introduce a new kind ofnode, named setting node and depicted as a square,to contain the specific information related to eachplayback of a media object. Setting nodes can berelated only to atomic and stream nodes composing

a multimedia presentation. They are used, forexample, to represent the position of the objectson the screen during the presentation or the dimen-sion of the area filled during their visualization.

For each instance of an object we need to defineand represent the right setting nodes. In our exam-ple, the object Name is played into two differentplaces and thus we need two different couples ofspatial coordinates: the first one is related to thefirst instance of Name (withmpi ¼ 1) and is (20, 20),while the second one is related to the secondinstance of Name (with mpi ¼ 10) and is (1, 1).For the first instance of Name, we will have thesetting nodes C_x (which represents the coordinateon the x-axis on the screen) and C_y (which repre-sents the coordinate on the y-axis on the screen)having both value 20. Moreover, in order to connectthe media object to its instance-related settingnodes, we label the edges between atomic (stream)and setting nodes by means of the associated mpi.Thus, the label between the object Name and thesetting node C_x, is labeled h P1, 1 i, where P1 is thename of the presentation we are defining3 and 1 isthe considered mpi.

The media objects, which compose the multime-dia presentation, must be played according to spe-cific temporal relationships, to be compliantwith thetimeline associated to the presentation itself. Forexample, in the first part of the multimedia presen-tation shown in Fig. 3, the media objects Name,Surname, Birthday, Sex and RiskType must be playedsimultaneously. In order to represent a multimediarelationship between these media objects, we intro-duce a particular kind of edge: the multimedia edge(depicted as thick line). The label of a multimediaedge reports the identifier of the considered presen-tation, thempis of theconsideredmedia objects, andthe description of the temporal relationship. Amulti-media edge is a directed edge that connects theobject with mpii and the object with mpij(mpii <mpij); the edge starts from the object withmpii and ends to the object with mpij.

In the following,wewill use thenotationT relationin order to describe a given presentation relationshipbetween mpis; relation stands for one of the follow-ing Allen’s relationships [45]: Starts, Before, Meets,Overlaps, Equal, Finished_by, andContains. Thisway,we are allowed to express any kind of qualitativetemporal relationships between intervals while con-sidering, at the same time, the direction of multi-media edges. Indeed, it is enough to represent only

Merging multimedia presentations and semistructured temporal data 99

those Allen’s relationships, in which the first operand(interval) starts before, or together with, the secondone: in fact, the object with the first mpi (i.e., thempi related to the object, the edge originates from)starts before, or together with, the object with thesecond mpi (i.e., the mpi related to the object, theedge terminates on).

Let us consider a first simple temporal scenario: inFig. 4, the multimedia edge between the complexnode Patient and the complex node RiskFactor withthe T_Equal label represents the fact that the infor-mation about the sexof thepatient and that about hisrisk factors must be played simultaneously. In parti-cular, the T_Equal relationship is defined betweenthe media object with mpi ¼ 4, which is the atomicnode Sex of the complex node Patient, and themediaobject with mpi ¼ 5, which is the atomic node Risk-Type of the complex object RiskFactor.

Let us now focus on a more complex temporalscenario: in the last part of the multimedia presen-tation, the video of the coronarography (represen-ted by the stream node StreamFile with value‘‘xa12.mpg’’), which has to be played four times,and the report of this exam (represented by thestreamnodeStreamFilewith value ‘‘report.rt’’) haveto be visualized together, but 5 s after the start timeof the visualization of the objects Name, Surname,SBP, DBP,Weight, andHeartRate (Fig. 3 highlights theduration of time interval ½t2; t3�, representing thedelay). This type of synchronization requirement isrepresented through several multimedia edges andsuitable setting nodes. Indeed, we have to introducemultimedia edges between themedia objects Name,Surname, SBP, DBP, Weight, and HeartRate, specify-ing the temporal relationshipT_Equal; then,wehaveto introduceamultimediaedgebetween thecomplexnodes Stenosis and ExamReport, specifying the tem-poral relationship T_Equal, to constrain the twostreamnodes StreamFile tobe contemporary. Finally,we insert a multimedia edge between the complexnode AngioVisit and the complex node Stenosis, asshown in Fig. 4. The label of this edge containsa relation TFinishedby between the object withmpi ¼ 15 (i.e. HeartRate) and the object withmpi ¼ 16 (i.e. StreamFile).

By these multimedia edges we have specified therequired qualitative temporal relationships betweenthe considered media objects. Setting nodes help usto specify the quantitative temporal requirements:to describe the delay of 5 s between the visualizationof the video with respect to the visualization of theother textual information, we introduce the settingnode Delay. In this case, the edge between thestream node and the Delay setting node is labeledh P1, 15, 16 i: it represents the delay of the visualiza-tion of the media object withmpi ¼ 16 (StreamFile)

with respect to the start time of the visualization ofthe object with mpi ¼ 15 (HeartRate). Thus, thesetting node Delay with value ‘‘5 s’’ represents therequired delay between the visualization of the clin-ical parameters of the patient (SBP, DBP, and so on)and the visualization of the video of the coronaro-graphy and of the associated textual report. More-over, to conclude the temporal specification of thepresentation, as shown in Fig. 4, the setting node Repof node StreamFile (with value ‘‘x12a.mpg’’) hasvalue ‘‘4’’, specifying that the video ‘‘xa12.mpg’’will be played four times together with the visualiza-tion of the other textual information.

Note that, if the related media objects are chil-dren of different complex nodes, we insert themultimedia edge between the two parents; instead,if the media objects are children of the same par-ent, we insert the multimedia edge directlybetween them. For example, in the first part ofthe presentation P1, the objects Name, Surname,Birthday, Sex must be played simultaneously withthe object RiskType. In this case, as shown in Fig. 4,we insert the multimedia edge with label h P1, 1,T_Equal, 2, [. . .] i between the atomic nodes Name(mpi ¼ 1) and Surname (mpi ¼ 2), the multimediaedge with label h P1, 2, T_Equal, 3, [. . .] i betweenthe atomic nodes Surname (mpi ¼ 2) and Birthday(mpi ¼ 3), and the multimedia edge with label h P1,3, T_Equal, 4, [. . .] i between the atomic nodesBirthday (mpi ¼ 3) and Sex (mpi ¼ 4). The multi-media edge between the complex nodes Patient andRiskFactor describes the fact that the atomic nodeRiskType has to be visualized together with thepreviously mentioned objects. The synchronizationrelationships between the media objects SBP, DBP,Weight, and HeartRate, which have to be visualizedsimultaneously in the last part of the presentation,are represented in a similar way.

All the edges and nodes introduced to modelmultimedia presentations are part of the multime-dia temporal database and each of them has a VT.The VT of the Presentation node represents thetimes when the presentation has been visualized.This way, it is possible to store into the semistruc-tured database the temporal information on theplayback of the defined multimedia presentations.VTs on multimedia edges represent the time duringwhich the part of presentation modeled by theconsidered edges has been played.

4. The multimedia temporal graphicalmodel

In this section, we provide a formal description ofMTGM, which is based on the Temporal Graphical

100 C. Combi et al.

Figure 4 The portion of the MTGM database that describe the multimedia presentation.

Merging multimedia presentations and semistructured temporal data 101

Model [22]. More specifically, MTGM deals with twonew aspects, i.e., the modeling of multimedia infor-mation and the modeling of presentations. More-over, further constraints are specified for VTs ofnodes and edges of MTGM graphs.

Definition 1. A multimedia temporal graph is adirected labeled graph hN; E; ‘i, where:

� N

is a finite set of nodes; � E is a set of labeled edges of the form hm; label; ni,

with m; n2N;

� ‘ : N!ðT n [f?gÞ � ðLn [f?gÞ � ðS [ f?gÞ�ðV [ f?gÞ;

� la

bel2 ðT e � LeÞ � ðMP [f?gÞ�ðLss [Lst [f?gÞ � ðMP [f?gÞ�ðLMMProp [f?gÞ � ðV [ f?gÞ.

? means ‘undefined’, and in the following, whenthe context is clear, if an element of a label is m-apped in ? , it will be omitted.

� N

odesThe set of types for nodes is:

T n ¼ fcomplex; atomic; stream; settinggand Ln is the set of labels to be used as namefor complex, stream and setting nodes (objects).S is a set of strings to be used as atomicvalues.

� E

dgesThe set of types for edges is:

T e ¼ frelational;multimediagand Le is the set of labels to be used as name foredges.

LMMProp ¼f½frame s; frame e�; ½time s; time e�;½x; y;width; height�g

is the set of descriptions to define a subpart of astream object: [frame_s , frame_e ] for video,[time_s , time_e ] for audio, and [x, y, width,height] for image objects.

� M

ultimedia edgesLst and Lss are sets of temporal and spatial

synchronization labels for multimedia edges. Inparticular:

Lst ¼ fT Before;T Contains;T Equal;T Finished by;T Meets;T Overlaps;T Startsg

Lss ¼ fS Disjoint; S Equal; S Meets;S Overlaps; S Includesg

The ordered set MP contains the values for thempis.

We require that:

Ln \Le \Lst \Lss ¼ ?

Temporal elements

� V is a set of temporal elements, such as

½t1; t2� [ ½t3; t4� [ � � � [ ½tn�1; tn�, with n> 1, tobe used as temporal elements for nodes andedges.

We use a temporal element to keep trace ofdifferent time intervals when an object exists inthe reality.

� L

abels‘ can be seen as composed by four single-

valued functions ‘T n ; ‘Ln ; ‘S ; ‘V.

� C onstraints

A multimedia temporal graph must satisfy thefollowing constraints:1. Besides presentation nodes, there is only

another access point to the graph; in otherwords, a multimedia temporal graph withoutcomplex nodes ‘‘Presentation’’ has a singleroot.

9 !x 2Nð‘T nðxÞ ¼ complex^

‘LnðmÞ 6¼ ‘‘Presentation’’^

@ hm; hrelational;RelName; Ii; xi 2 E ^@ hm; hmultimedia; PresDeti; xi 2 EÞ

where PresDet stands for all the differentstructures related to multimedia edges.

2. Values are associated to atomic, stream andsetting nodes only.

8 x 2Nð‘T nðxÞ ¼ complex! ‘SðxÞ ¼ ? Þ

3. Temporal elements are not associated toatomic, stream, and setting nodes becausein the database the temporal element of avalue coincides with the temporal element ofits ingoing edge.

8 x 2Nðð‘T nðxÞ ¼ atomic_

‘T nðxÞ ¼ stream_

‘T nðxÞ ¼ settingÞ! ‘VðxÞ ¼ ? Þ

4. The label of a relational edge is composedby

hrelational;RelName; Iiwhere relational is the type of the edge,RelName the name associated to the edgeand I its temporal element.(a) Relational edges can be defined only

between complex nodes or between a

102 C. Combi et al.

8 hm; hmultimedia; PresName;mpii;

R Sincr;mpij; Ii; ni 2 E

ð‘T nðmÞ ¼ ‘T n

ðnÞ ¼ complex_ðð‘T n

ðmÞ¼stream_ ‘T nðmÞ¼atomicÞ ^

ð‘T nðnÞ ¼ stream_ ‘T n

ðnÞ ¼ atomicÞ ^9 p2Nð‘T n

ðpÞ ¼ complex^9 hp; hmultimedia; PresName;mpii; Ii;m; i 2 E ^

9 hp; hmultimedia; PresName;mpij; Ii;

complex node and its atomic or streamnodes.

8 hm; hrelational;RelName; Ii; ni 2 E

ð‘T nðmÞ ¼ ‘T n

ðnÞ ¼ complex_ð‘T n

ðmÞ ¼ complex^ð‘T n

ðnÞ ¼ atomic_ ‘T nðnÞ ¼ streamÞÞÞ

(b) Each atomic node is connected to itsparent by an edge labeled ‘‘HasProperty’’and with temporal element I.

8 hm; hrelational;RelName; Ii; ni 2 E

ð‘T nðnÞ ¼ atomic!

RelName ¼ ‘‘HasProperty’’Þ

5. The VT of an edge between a complex nodeand an atomic node must be related to theVT of the complex node. Intuitively, therelation between a complex node and anatomic node cannot survive the complexnode, thus the VTof the edge cannot containtime points which do not belong also to theVT of the complex node. This is due to thefact that we suppose that a complex node isrelated to its properties (atomic nodes)while it is valid.

8 hm; hrelational;RelName; Ii; ni 2 E

ðð‘T nðnÞ ¼ atomic_

‘T nðnÞ ¼ streamÞ! ‘Vn

ðmÞ IÞ6. At a specific time instant, between two nodes

it cannot exist more than one edge with thesame name RelName. Note that all the edgeshave the same name RelName. This is due tothe fact that an edge represents a relation-ship between two nodes, thus it makes nosense representing with two edges the samerelationship.

8 hm; hrelational;RelName; I1i; ni 2 E

ð@ hm; hrelational;RelName; I2i; ni 2 E

ðI1 \I2 6¼?ÞÞ

7. Each presentation stored in an MTGMgraph will have a complex initial noden2N with label of the form hcomplex;Presentation; ? ; Ii. The complex nodePresentation will be the root of an MTGMsub-graph representing the multimediainformation associated to a presentation.Each complex node Presentation has a uniquename described by the value of its atomic

node PresName.

8m2Nðð‘T nðmÞ ¼ complex^

‘LnðmÞ ¼ ‘‘Presentation’’Þ!

ð 9 !n2N

ð 9 hm; hrelational;RelName; Ii; ni 2 E

ð‘T nðnÞ ¼ atomic^ ‘Ln

ðnÞ ¼ ‘‘PresName’’^RelName ¼ ‘‘HasProperty’’^@ hx; hrelational;RelName; Ii;mi 2 E ^@ hy; hmultimedia; PresDeti;mi 2 EÞÞÞÞ

8. The label of a multimedia edge between twocomplex nodes or between twomedia objectswhich are children of the same parent, iscomposed by

hmultimedia; PresName;mpij;R Sincr;mpik; Iiwhere PresName is the presentation name,R_Sincr the spatio/temporal synchronizationrelationship ðRSincr2 ðLss [Lst [f?gÞÞ,mpij;mpik 2MP, and I the temporal ele-ment related to edge.

The label of a multimedia edge between acomplex node and its stream or atomic nodeis composed by

hmultimedia; PresName;mpij; Iiwherempij is themediapresentation identifierassociatedtothestreamoratomicobjectandIthe temporal element related to the edge.(a) Each complex node Presentation has a

unique multimedia outgoing edge, thatis connected to the first media object inthe presentation.

8m2Nð‘LnðmÞ ¼ ‘‘Presentation’’!

9 !hm; hmultimedia; ?;?;?;?;?i; ni 2 E

ð‘T nðnÞ ¼ complexÞÞ

(b) Multimedia edges with a specific spatio/temporal synchronization relationship canconnect only complex nodes or stream/atomic nodes children of the same parent.

n; i 2 EÞÞÞ

Merging multimedia presentations and semistructured temporal data 103

(c) Multimedia edges can connect acomplex node with its stream nodeswhich represent links to media objects.The label of an ingoing edge of a streamnode is composed by the name ‘‘HasMM-Property’’ and the value p, whichdescribes a specific subpart of the mediaobject.

8 hm; hmultimedia; PresName;mpij; Ii; ni2 E ðð‘T n

ðmÞ ¼ complex^‘T nðnÞ ¼ streamÞ!

9 hm; hrelational;RelName; p; Ii; ni 2 E

ðRelName ¼ ‘‘HasMMProperty’’ÞÞ

9. The label of the unique multimedia edgebetween a stream or atomic node and itssetting node, but for Delay, is composedby

hmultimedia; PresName;mpijiwhere mpij is the media presentationidentifier related to the stream or atomicnode.

8 n2Nðð‘T nðnÞ ¼ setting^ ‘Ln

ðnÞ 6¼ ‘‘Delay’’Þ! ð 9 !m2Nðð‘T n

ðmÞ ¼ atomic _‘T nðmÞ ¼ streamÞ ^

9!hm;hmultimedia; PresName;mpiji; ni2EÞÞÞ10. In the case of a multimedia edge between a

stream node and the setting node Delay, thelabel of the unique multimedia edge betweenthem is composed by:

hmultimedia; PresName;mpij;mpikiwhere mpij and mpik are media presentationidentifiers and mpij<mpik.

8 n2Nðð‘T nðnÞ ¼ setting^ ‘Ln

ðnÞ ¼ ‘‘Delay’’Þ! ð 9 !m2Nðð‘T n

ðmÞ ¼ atomic_‘T nðmÞ ¼ streamÞ ^

9 !hm; hmultimedia; PresName;mpij;

mpiki; ni 2 EÞÞÞWe also require the following conditions.

11. The qualitative temporal relationship,defined in the synchronization label of a mul-timedia temporal edge between two streamor atomic nodesm and n, children of the sameparent p, can be indefinite if and only if thesetting nodes Cs and Ce describe the start andthe end of the visualization for the mediaobject n with respect to the start

time of the presentation PresName.

8m2Nð 9 hm; hmultimedia; PresName;mpij;

T Sincrjk;mpik; Ii; ni 2 E

ððT Sincrjk ¼ ? ^ ð‘T nðmÞ ¼ stream _

‘T nðmÞ ¼ atomicÞ ^

ð‘T nðnÞ ¼ stream_ ‘T n

ðnÞ ¼ atomicÞÞ$9p;Cs;Ce2Nð‘T n

ðpÞ ¼ complex^‘T nðCsÞ ¼ ‘T n

ðCeÞ ¼ setting^ð‘SðCsÞ 6¼ ? ^ ‘SðCeÞ 6¼ ? Þ^

9 hp; hmultimedia; PresName;mpij; Ii;mi 2E ^ 9 hp; hmultimedia; PresName;

mpik; Ii; ni2 E ^9 hn;hmultimedia; PresName;mpiki;Csi2E ^9 hn; hmultimedia; PresName;

mpiki;Cei 2 EÞÞÞ12. The qualitative temporal relationship,

defined in the synchronization label of amultimedia temporal edge between twocomplex nodes m and n, can be indefiniteif and only if the setting nodes Cs and Ceassociated to the stream (or atomic) nodeq (child of the node n), describe the startand the end time of its visualization withrespect to the start time of the presentationPresName.

8m2Nð9hm;hmultimedia;PresName;mpij;

T Sincrjk;mpik;Ii;ni2EððT Sincrjk¼? ‘T n

ðmÞ¼‘T nðnÞ¼complexÞ

$9p;q;Cs;Ce2Nðð‘T nðpÞ¼ stream_

‘T nðpÞ¼atomicÞ^

ð‘T nðqÞ¼ stream_‘T n

ðqÞ¼atomicÞ‘T nðCsÞ¼‘T n

ðCeÞ¼setting^9hm;hmultimedia;PresName;mpij;Ii;pi2E^9hn;hmultimedia;PresName;mpik;Ii;qi2E^9hq;hmultimedia;PresName;mpiki;Csi2E^

9hq;hmultimedia;PresName;mpiki;Cei2EÞÞÞ

Further constraints, similar to the constraints11 and 12, hold also with respect to qualitativespatial relationships and setting nodes, as detailedin [46].

Definition 2. Let G be an MTGM graph that containsa multimedia presentation P (where P is the value ofthe atomic node PresName). The multimedia viewon P of G is an MTGM graph G0 where:

104 C. Combi et al.

� t

he root is the complex node named ‘‘Presenta-tion’’ with its properties (atomic nodes);

� t

he complex nodes of G0 are the multimediacomplex nodes that compose the multimedia pre-sentation with their stream and setting nodes;

� t

he edges ofG0 are themultimedia edges of G withlabel containing P.

For example, Fig. 4 shows themultimedia view onP1 of the MTGM graph shown in Fig. 2.

5. Designing and implementing MTGM byan XML native database system

MTGM can be seen as a logical model for semistruc-tured data: for its application to clinical data it isimportant to consider how (parts of) an MTGM data-base can be physically realized through the emergingXML technology [9]. To do that, we have to provideboth a technique for translating (a part of) an MTGMdatabase into an XML document and some techni-ques allowing us to obtain SMIL presentations fromMTGM presentations. Moreover, before implement-ing a prototype MTGM database system, we have todesign the overall architecture of the system.

5.1. Translating MTGM graphs into XMLdocuments

Among the main differences between MTGM and thedata model underlying XML, we have to consider (i)the fact that MTGM considers labeled graphs, whileXML mainly deals with trees, and (ii) that, while inMTGM both nodes and edges are labeled, in XML onlylabeled nodes are allowed. Another important differ-ence is that XML node labels are in some way atomic,while in MTGM we deal with compound labels.

The overall, main ideas underlying the designedtranslation technique can be summarized as follows.

� C

omplex nodes are translated into complex ele-ments (i.e., elements which contain other ele-ments); in particular, they have a (nested)element for the corresponding VTand an elementfor each outgoing edge.

� A

tomic nodes are translated into mixed elements(i.e., elements containing both text and otherelements); in particular, they contain the stringrepresenting their values and an element for theirVTs (which are contained, in the correspondingMTGM graph, in the label of the ingoing relationaledge).

� E

dges between complex nodes are representedthrough complex elements nested into the ele-ment corresponding to the complex node, which

the edge originates from. The element corre-sponding to the node the edge points to, isreferred through a suitable attribute in the ele-ment representing the edge.

� E

dges between a complex node and an atomic(stream) one are not translated into an XML ele-ment but are represented by nesting the atomic(stream node) within the complex one.

� C

ompound labels are managed by introducingsuitable sub-elements (i.e., nested elements),as for representing VTs of nodes and edges.

� E

lements corresponding to MTGM nodes have anattribute (of type ID), which allows one to refer tothem in an unambiguous way.

� A

root element for the XML document is explicitlycreated: it contains the elements corresponding tothe nodes Presentation and the single elementcorresponding to the root of the multimedia data-base.

In the following, we will describe the proposedtranslation for three cases, starting from the trans-lation of a complex node with an atomic propertyand proceeding in order to consider its relationaland multimedia edges, to give some examples of theproposed approach. A detailed description of thetranslation of MTGM graphs can be found in [46].

� T

ranslation of an MTGM complex node with anatomic property.

Fig. 5 shows the translation of the complexnode Patient, with identifier obj1, and its atomicnode Name, with identifier obj2 and value Ed.

� T

ranslation of an MTGM relational edge betweentwo complex nodes.

Fig. 6 shows the translation into XML code of thecomplex node Patient, with identifier obj1, and ofthe complex node RiskFactor, with identifier obj2.The relational edge Patient-RiskFactor betweenthem is nested into the hPatienti XML element.

� T

ranslation of an MTGMmultimedia edge betweentwo atomic nodes.

Fig. 7 shows the translation into XML code of amultimedia edge between the atomic nodes SBPwith identifier obj6 and DBP with identifier obj7.

5.2. Translating an MTGM presentation into aSMIL document

SMIL is emerging as the XML-based language for thedefinition of multimedia distributed audiovisualpresentations and some (even free) commercialplayers for video and audio data over the Websupport media presentations written according tothe SMIL syntax [47]. After the mapping of MTGMdatabases into XML documents, it is thus important

Merging multimedia presentations and semistructured temporal data 105

Figure 5 (a) The complex node Patient, its atomic property Name and (b) their translation into XML.

to address the mapping of MTGM presentations intoSMIL documents: it would allow physicians to have apowerful way of exchanging (suitable views on)multimedia clinical data, without the need of mana-ging ad-hoc software and file formats.

A SMIL presentation is a structured composition ofautonomous media objects [21]. The root of a SMILdocument is the element hsmili. SMIL documentshave two parts: head and body. Each of them musthave hsmili as parent. The head of a SMIL document

Figure 6 (a) The complex nodes Patient and RiskFactor, the rinto XML.

describes the meta-information about the visualiza-tion environment, while the body describes theinformation about the synchronization. Thus, theinformation about temporal relationships isdescribed in the body part.

The SMIL hregioni element is part of the head andallows one to specify the position, size, and scalingof media objects. The position of a region is speci-fied by its top, bottom, left, and right attributes.Each SMIL-region is related to a unique id: in the

elational edge Patient-RiskFactor and (b) their translation

106 C. Combi et al.

Figure 7 (a) The atomic nodes SBP and DBP, their multimedia edges and (b) their translation into XML.

4 Type allows one to specify the type of the media object, Csand Ce allow one to specify the starting and ending times of theobject presentation, Delay allows one to specify the delay for thebeginning of the object presentation, BeginKey and EndKey allowone to specify the key the user has to press for beginning andending, respectively, the object presentation, Repeat specifiesthe number of times the object has to be presented, Fill allowsone to specify whether the final state of the object presentationshould be maintained by freezing it, Restart allows one to specifyhow the object presentation has to deal with recurrent objectpresentations.

following, we use the media presentation identifier(mpi) of a media object as its SMIL-region’s id.

For each media object represented in an MTGMgraph G0, we define its ‘‘smil-region’’. Let be mi amedia object, mpij one of the media presentationidentifiers of mi and { sij} the set of setting nodes ofmi, related to the media presentation instancempij: the set of setting nodes related to the nodemi, allowing the description of its region element, isfCx;Cy;Width;Height; Priority;BgColor; SdLevelg(see Section 3.2), where the new setting nodesPriority and SdLevel specify the stacking order,when the region overlaps another region, and theaudio volume (in case of audio data). We insert inthe tag hregioni only those setting attributes havinga specified value. For example, in Fig. 4 the settingnodes of the node Name related to the mpi withvalue 1 are Cx, Cy, and Type with values 1, 1, and‘‘text’’, respectively.

The hbodyi element of a SMIL document containsinformation related to the temporal behavior of thedocument. In particular, in a SMIL presentation wecan use three basic timing policies through theelements [21]: hseqi or sequential time container;hpari or parallel time container; hexcli, or exclusivetime container.

We can nest the three basic time container typesin a presentation hierarchy; i.e., each child of a par,seq or excl container can be a simple media object

or an embedded par, seq, or excl container. In thebody part, we use the himgi, haudioi, hvideoi, orhtexti tags to insert into its region an image, anaudio, a video, or a text, respectively. Moreover,the temporal behavior of a generic node mi canbe specified by setting nodes belonging to the setfType; Cs; Ce;Delay; Duration; BeginKey; EndKey;Repeat; Fill;Restartg.4

The content of a multimedia presentation is notincluded in the body part of a SMIL document:according to the approach underlying SMIL [47], aSMIL document contains only references to themedia objects to be played. Media objects arereferred through suitable URLs. If the media objectmi is a stream node then its value is used as value ofthe src attribute, which describes the full or relativeURL for the object; if the media object is an atomicnode, then we use the function representing the

Merging multimedia presentations and semistructured temporal data 107

Figure 8 The set of multimedia edges for the MTGMgraph in Fig. 4.

content of the atomic node. For example, the func-tion Nodetostringfile applied to the atomic nodeName (NodetostringfileðNameÞ) returns the name ofthe text file ‘‘Name.rt’’ that contains the informa-tion ‘‘Name: Ed’’.

In order to describe the SMIL document thatrepresents the multimedia presentation shown inFig. 3, we consider the labels of the multimediaedges according to the order induced by mpis. Forexample, the set E of the multimedia edges in Fig. 4is ordered as in Fig. 8.

We create the basic time container type (par andseq) for each media object included in the set E. Inparticular, we distinguish two cases:

� if

thempij of a nodemi is related to thempik of anode ml with a temporal relation T_Overlaps,T_Contains, T_Starts, T_Finished_by, or T_Equal,then we insert these objects in the same hpariSMIL tag;

� if

thempij of a nodemi is related to thempik of anode ml with a temporal relation T_Before orT_Meets, then we insert these objects in the same

Figure 9 The SMIL code fragment of

hseqi SMIL tag.

For example, the edges a, b, c, d require a T_-Equal relationship between the objects withmpis 1,2, 3, 4 and 5; in this case Name (mpi ¼ 1), Surname(mpi ¼ 2), Birthday (mpi ¼ 3), Sex (mpi ¼ 4) andRiskType (mpi ¼ 5) of a patient are shown simulta-neously, and thus they must be inserted in the samehpari SMIL element.

The edges l, m, n, o, p, q, and r require that themedia objects Name, Surname, SBP, DBP, Weight,HeartRate, StreamFile with value ‘‘report.rt’’, andStreamFile with value ‘‘xa12.mpg’’must be insertedin the same hpari SMIL element.

The edge q requires a T_Finished_by relationshipbetween the objects with mpis 15 and 16. Thesetting node Delay related to the stream nodeStreamFile with mpi ¼ 16 and value ‘‘xa12.mpg’’,describes the required delay between the visualiza-tion of the clinical parameters of the patient (likeSBP, DBP, etc.) and the visualization of the video ofthe exam. Thus, the SMIL elements which describethe visualization of the video (and of the report),must define the required delay by means of a beginSMIL attribute with value 5 s (i.e. begin ¼ ‘‘5s00).

The edge e requires a T_Meets relationshipbetween the objects identified by mpis 5 and 6.In this case the hpari element that contains theobjects with mpis 1, 2, 3, 4, and 5 must be insertedin a hseqi SMIL element with the hpari element thatcontains the objects with mpis 6, 7, 8, and 9. As inthe previous case, for the label of the edge i, thehpari element that contains the objects withmpis 6,7, 8, and 9 must be inserted in the hseqi elementwith the hpari element that contains the objectswith mpis 10, 11, 12, 13, 14, 15, 16, and 17.

the presentation specified in Fig. 4.

108 C. Combi et al.

Figure 10 The architecture of the system.

The SMIL code fragment in Fig. 9 describes theoverall body part of the presentation depicted inFig. 4.

5.3. System architecture

The system prototype has been designed accordingto the architecture depicted in Fig. 10. The system iscomposed by the following modules:

� M

TGM engine. The MTGM engine provides all theprimitives for managing MTGM graphs. An MTGMgraph is stored as sets of related objects, repre-senting (complex, atomic, stream or setting)nodes and (relational or multimedia) edges.

� G

UI (graphic user interface). This module allowsthe interaction between the user and the system,mainly in creating and updating MTGM graphs, andin producing XML and SMIL files. It provides a tree-based data representation, common to otherwidely used interfaces as DataGuides [48]. Suita-ble graphical solutions are provided to managethe representation of MTGM graphs as trees. Awizard guides the user to build multimedia pre-sentations. Even though the GUI guides the user tobuild consistent MTGM graphs, according to theconstraints described in Section 4, it is worthnoting that the design of tools for the on-lineverification of the presentations consistency(i.e., during their definition) is the main focusof specific research efforts [20,40,41], which arecomplementary to the modeling issues consideredin this paper.

� D

OM interface. The DOM interface allows both thetranslation of a (part of) MTGM graph into an XMLdocument and the translation of the stored XMLdocuments into MTGM graphs.

� C

ache. During the update operations, it may hap-pen that the system has to deal with huge filescontaining multimedia data as movies, images,and sounds: this module allows the usage of thevirtual memory to store temporary files contain-ing multimedia data during the data processing,avoiding several (slow) I/O operations on themassstorage device.

� N

XD interface. It manages the access to the storeddata through a native XML database system (NXD).The basic entity managed by the database systemis the XML document.

� F

ile manager. It manages the operations on files:more precisely, it manages both the access to thefiles containing multimedia data, which have tobe stored/exported into/from the database, andthe creation of XML and SMIL files.

� Q

uery engine. It manages both the queries neededby the other system modules and the queriesdefined by the user. Queries are expressed inXPATH.

5.4. The system prototype

The system prototype has been developed by theJava technology, and is based on the native XMLdatabase system eXist [49].

Fig. 11 depicts as an example how the systemdisplays the MTGM graph of Fig. 2 to the final user:on the left part of the window, the MTGM graph isdisplayed as a tree with different icons for complex,atomic, and stream nodes, respectively. Complexnodes pointed by several relational edges are dis-played several times, to allow a tree-like represen-tation. All the defined presentations arerepresented in the left, below the tree. Detailson nodes and edges appear in suitable windows in

Merging multimedia presentations and semistructured temporal data 109

Figure 11 An example of the prototype GUI.

the right part: Fig. 11 depicts a window for a com-plex node (CoronaroExam), a stream node (Stream-File), an atomic node (DBP), and an edge (Exam),respectively.

As previously described, SMIL presentations canbe created by the system: Fig. 12 shows the thirdpart of the multimedia presentation (specifiedwithin the third element par in the code fragment)

Figure 12 Visualization of the Ed

displayed with RealOnePlayer [50], one of the soft-ware systems currently supporting the display ofSMIL presentations. At the top of the left part ofthe window, the name of the patient and his clinicalparameters are shown. After 5 s from the start ofvisualization of this information, the video of thecoronarography and the report are shown. The videois shown four times in the right part of the window,

Bawer’s coronary angiography.

110 C. Combi et al.

while the textual report ‘‘Mild stenosis on the prox-imal right coronary artery’’, is shown in the bottomof the left part.

6. Discussion and conclusions

In this work we have considered the theoretical andmethodological issues concerning the definition of ageneral data model for describing temporal andmultimedia features of semistructured clinicalinformation. We have proposed MTGM, a semistruc-tured data model based on labeled graphs for repre-senting and storing both multimedia and temporalaspects of semistructured data in a seamless way,and we have shown its applicability to medicaldomains. In particular, we have highlighted howMTGM allows the expression of several differentmultimedia presentations, related to different con-texts, based on the same data stored in an MTGMclinical database. The temporal concept of validtime has been extended also to the multimediaobjects and to the synchronization relationshipscomposing a multimedia presentation. Moreover,we have shown that MTGM databases can be suitablytranslated into XML documents, and that SMIL pre-sentations can be derived from MTGM presenta-tions; in this direction we have designed andimplemented an XML-based database system, allow-ing one to manage and store MTGM graphs.

Other both research and application orientedfeatures, which have not been considered in MTGM,could be investigated for completing MTGM withregard to its applicability to clinical domains.Indeed, MTGM does not allow one to express timesat different levels of granularities, i.e. with differ-ent time units, or with indeterminacy [43]; eventhough this issue has been largely considered alsofor clinical databases and some solutions have beenproposed [16,17,42,51,52], to the best of our knowl-edge, there are no clinical information systemscurrently in a daily use that can support differenttemporal granularities or indeterminacy in medicalinformation.

MTGM deals only with the VT dimension: in amedical setting it would be useful to be able tomanage other temporal dimensions, as the transac-tion time [43], or the recently proposed availabilitytime, which is time during which the fact is knownand believed correct by the information system[53]. Indeed, by these two temporal dimensions itis possible to analyze medical decisions according tothe information stored into the database and avail-able to the physician, respectively, in order, forexample, to assess in an unambiguous way thequality of the provided care.

MTGM presentations can be considered as a wayto access (and hide) patient data: it could be pos-sible, at the application level, to associate MTGMviews to specific users, both for the definition andthe playing of multimedia presentations. Thus,MTGM presentations could be integrated with theusual policies for allowing the right data access toauthorized users [54].

As a final point of discussion, let us focus on theapplicability of the MTGM data model to real worldclinical data and, more precisely, to knowledge-based medical applications providing a support tosome decision making task. In this paper, weshowed, as a proof of concept, that MTGM can beused to represent data from the cardiac angiographydomain [30]; moreover, we described both thetranslation of MTGM graphs into XML data and theirmanagement through an XML native database sys-tem. Besides being useful for managing multimediadata stored according to widely accepted standardsas MPEG and DICOM, for radiological and, moregenerally, clinical multimedia data, it is importantto underline that nowadays semistructured data,and XML in particular, are becoming the most impor-tant way for expressing and exchanging medicalknowledge and data. To this regard, for example,the DICOM standard has been considered and stu-died for an XML-based representation [55]; on theother hand, medical ontologies will be representedaccording to specific XML languages [56,57]. In thisscenario, MTGM can be considered as a data modelallowing the seamless representation of both (multi-media) data and knowledge; while the same (struc-tural) constraints are directly managed by MTGM,suitablemechanisms could be designed, to allow theuser to couple in a consistent way acquired knowl-edge and patients data.

In this paper, we did not consider the design of aspecific query language for MTGM data, even thoughthe user is allowed to query by XPATH the XMLdatabase storing the MTGM data: as a future work,we plan to design a suitable query language for thedefinition of complex multimedia views on clinicaldata: indeed multimedia presentations can be con-sidered as a special kind of (interactive) views ondata. To this regard, it would be interesting to allowthe physician to define multimedia presentationswhich would be evaluated in a dynamic way, i.e.,according to the content of the database when thepresentation is played.

Acknowledgments

We are grateful to Andrea Allegri for his work indesigning and implementing the prototype. Wewant

Merging multimedia presentations and semistructured temporal data 111

also to thank the anonymous reviewers for theirhelpful and detailed suggestions. This work has beenpartially supported by contributions from the ItalianMinistry of University and Research (MIUR) underprogram ‘‘COFIN-PRIN 2003 Representing and mana-ging spatial and geographical data on the Web’’.

References

[1] Dionisio JDN, Cardenas AF. A unified data model for repre-senting multimedia, timeline and simulation data. IEEETrans Knowledge Data Eng 1998;10(5):746—67.

[2] Portoni L, Combi C, Pinciroli F. User-oriented views in healthcare information systems. IEEE Trans Biomed Eng 2002;49(12):1387—98.

[3] Arya M, Cody WF, Faloutsos C, Richardson J, Toya A. Designand implementation of QBISM, a 3D medical image databasesystem. In: Subrahmanian VS, Jajodia S, editors. Multimediadatabase system: issues and research direction. Berlin:Springer; 1996. p. 79—100.

[4] Jiang H, Elmagarmid HK. Spatial and temporal content-based access to hypervideo databases. VLDB J 1998;7:226—38.

[5] Subrahmanian VS, Jajodia S. Multimedia database system:issues and research direction. Berlin: Springer, 1996.

[6] Pinciroli F, Combi C, Pozzi G. ARCADIA: a system for theintegration of angiocardiographic data and images by anobject-oriented DBMS. Comput Biomed Res 1995;28:5—23.

[7] Kahn CE, Wang C. Potential use of exstensible markuplanguage for radiology reporting: a tutorial. Radiographics2000;20:287—93.

[8] Abiteboul S, Buneman P, Suciu D. Data on the web: fromrelations to semistrucutred data and XML. San Francisco:Morgan Kaufmann Publishers, 1999.

[9] World Wide Web Consortium. XML http://www.w3.org/xml/1.0. http://www.w3.org/xml/.

[10] Dugas M, Hoffmann E, Janko S, Hahnewald S, Matis T, UberlaK. XML-based visual data mining in medicine. In: Patel VL,Rogers R, Haux R, editors. Proceedings of the 10th worldcongress on health informatics (Medinfo 2001), vol. 10.Amsterdam: IOS Press; 2001. p. 1324—8.

[11] Kahn CE. Self-documenting structured reports using openinformation standards. In: Scherrer JR, Cesnik B, McCray AT,editors. Proceedings of the ninth world congress on medicalinformatics (Medinfo ’98), vol. 52. Amsterdam: IOS Press;1998. p. 403—7.

[12] Grover C, Klein E, Lapata M, Lascarides A. XML-based NLPtools for analysing and annotating medical language. In:Proceedings of the second international workshop on NLPand XML (NLPXML-2002); 2002.

[13] Dolin RH, Alschuler L, Behlen F, Biron PV, Essin D, Boyer S,Harding L, Lincoln T, Mattison JE, Sokolowski R, Rishel W,Spinosa J, Williams JP. HL7 document patient record archi-tecture: an XML document architecture based on a sharedinformation model. In: Lorenzi NM, editor. Proceedings ofthe AMIA annual symposium. Philadelfia: Hanley and Belfus;1999. p. 52—6.

[14] Kahn CE, de la Cruz N. Extensible markup language (XML)in health care: integration of structured reporting anddecision support. In: Chute CG, editor. Proceedings ofthe AMIA annual fall symposium. Philadelfia: Hanley andBelfus; 1998. p. 725—9.

[15] Combi C, Cucchi G, Pinciroli F. Applying object-orientedtechnologies in modeling and querying temporally-oriented

clinical databases dealing with temporal granularity andindeterminacy. IEEE Trans Inform Technol Biomed 1997;1(2):100—27.

[16] Nguyen JH, Shahar Y, Tu SW, Das AK, Musen MA. Integrationof temporal reasoning and temporal-data maintenance intoa reusable database mediator to answer abstract, time-oriented queries: the Tzolkin system. J Intell Inform Syst1999;13(1/2):121—45.

[17] Shahar Y, Combi C. Temporal reasoning and temporal datamaintenance in medicine: issues and challenges. ComputBiol Med 1997;27(5):353—68 (special issue: Time-OrientedSystems in Medicine).

[18] Combi C, Portoni L, Pinciroli F. Visualizing temporal clinicaldata on the WWW. In: Horn W, Shahar Y, Lindberg G,Andreassen S, Wyatt JC, editors. Proceedings of the jointEuropean conference on artificial intelligence in medicineand medical decision making (AIMDM’99), vol. 1620. LectureNotes in Artificial Intelligence. Berlin: Springer; 1999. p.301—311.

[19] Jensen CS, Snodgrass RT. Temporal data management. IEEETrans Knowledge Data Eng 1999;11(1):36—44.

[20] Bertino E, Ferrari E. Temporal synchronization models formultimedia data. IEEE Trans Knowledge Data Eng 1998;10(2):612—31.

[21] Bulterman DCA. SMIL 2.0 Part 1: overview, concepts andstructure. IEEE Multimedia 2001;8(4):82—8.

[22] Oliboni B, Quintarelli E, Tanca L. Temporal aspects of semi-structured data. In: Proceedings of the eighth internationalsymposium on temporal representation and reasoning (TIME-01). Los Alamitos: IEEE Computer Society Press; 2001 . p.119—27.

[23] Oracle. Oracle 8i. 2002. http://otn.oracle.com/tech/xml/xmldb/content.html.

[24] Microsoft. Access. 2002. http://msdn.microsoft.com/library/.

[25] Chaudhri A, Rashid A, Zicari R. XML data management:native XML and XML-enabled database systems. Boston:Addison Wesley Professional, 2003.

[26] IBM. DB2. 2003. http://www-3.ibm.com/software/data/db2/extenders/xmlext/.

[27] Fiebig T, Helmer S, Kanne CC, Moerkotte G, Neumann J,Schiele R, Westmann T. Anatomy of a native XML basemanagement system. VLDB J 2002;11(4):292—314.

[28] Software AG. Tamino, 2002. http://www.softwareag.com/tamino/architecture.htm.

[29] Apache Software Foundation. Xindice, 2002. http://xml.a-pache.org/xindice/.

[30] Scanlon PJ, Faxon DP, Ritchie JL, Gibbons RJ, Audet AM,Carabello B, Dehmer GJ, Eagle KA, Legako RD, Leon DF,Murray JA, Nissen SE, Pepine CJ, Watson RM, Cheitlin MD,Eagle KA, Gardner TJ, Garson A, Russell RO, Ryan TJ, SmithSC. ACC/AHA guidelines for coronary angiography. J AmCollege Cardiol 1999;33(6):1756—824.

[31] Chawathe S, Abiteboul H, Widom J. Managing historicalsemistructured data. Theory Practice Object Syst 1999;5(3):143—62.

[32] Bui AAT, Winger GS, Barretta SJ, Dionisio JD, Kangarloo H. AnXML gateway to patient data for medical research applica-tions. Ann New York Acad Sci 2002;980:236—46.

[33] Schweiger R, Hoelzer S, Rudolf D, Rieger J, Dudeck J. Linkingclinical data using XML topic maps. Artif Intell Med 2003;28(1):105—15.

[34] Seals M. The use of XML in healthcare information manage-ment. J Healthcare Inform Manage 2000;14(2):85—95.

[35] Sheth AP, Larson JA. GEM: a proposal for a more compre-hensive guideline document model using XML. J Am MedInform Assoc 2000;7(5):488—98.

112 C. Combi et al.

[36] Yamazaki S, Satomura Y. Standard method for describing anelectronic patient record template: application of XML toshare domain knowledge. Meth Inform Med 2000;39(1):50—5.

[37] Cardenas AF, Ieong IT, Taira RK, Barcher R, Breant CM. Theknowledge-based object-oriented PIQUERY+ language. IEEETrans Knowledge Data Eng 1993;5:644—57.

[38] Tanaka K, Oomoto E. Ovid: design and implementation of avideo-object database system. IEEE Trans Knowledge DataEng 1993;5:629—43.

[39] Roussopoulos N, Faloutsos C. An efficient pictorial databasesystem for PSQL. IEEE Trans Knowledge Data Eng 1988;5:639—50.

[40] Mirbel I, Pernici B, Sellis TK, Tserkezoglou S, Vazirgiannis M.Checking the temporal integrity of interactive multimediadocuments. VLDB J 2000;9(2):111—30.

[41] Bertino E, Ferrari E, Stolf M. MPGS: an interactive tool forthe specification and generation of multimedia presenta-tions. IEEE Trans Knowledge Data Eng 2000;12(1):612—31.

[42] Combi C, Pinciroli F, Pozzi G. ARCADIA: a system for theintegration of angiocadiographic data and images by anobject-oriented DBMS. Comput Biomed Res 1995;28(1):5—23.

[43] Jensen CS, Dyreson CE, Bohlen M, Clifford J, Gadia SK,Elmasri R, Grandi F, Hayes P, Jajodia S, Kafer W, Kline N,Lorentzos N, Mitsopoulos Y, Montanari A, Nonen D, Peressi E,Pernici B, Roddick JF, Sarda NL, Scalas MR, Segev A, Snod-grass RT, Soo MD, Tansel A, Tiberio P, Wiederhold G. Theconsensus glossary of temporal database concepts–—Febru-ary 1998 Version. In: Etzion O, Jajodia S, Sripada SM, editors.Temporal databases research and practice Lecture notes incomputer science, vol. 1399. Berlin: Springer; 1998. p. 367—405.

[44] Mildenberger P, Eichelberg M, Martin E. Introduction to theDICOM standard. Eur Radiol 2002;12(4):920—7.

[45] Allen JF. Maintaining knowledge about temporal intervals.Commun ACM 1983;26(11):832—43.

[46] Combi C, Oliboni B, Rossato R. A semistructured data modelfor multimedia and temporal clinical information. Technical

Report RR 22/2004. Department of Computer Science, Uni-versity of Verona, Italy; 2004.

[47] World Wide Web Consortium. Syncronized Multimedia Inte-gration Language. SMIL 2.0, 2000. http://www.w3c.org/tr.

[48] Goldman R, Widom J. DataGuides: enabling query formula-tion and optimization in semistructured databases. In: JarkeM, Carey MJ, Dittrich KR, Lochovsky FH, Loucopoulos P,Jeusfeld MA, editors. VLDB’97, Proceedings of the 23rdinternational conference on very large data bases. SanFrancisco: Morgan Kaufmann; 1997. p. 436—45.

[49] Meier W. Exist: an open source native XML database. In:Chaudhri AB, Jeckle M, Rahm E, Unland R, editors. Webweb-services and database systems Lecture notes in com-puter science, vol. 2593. Berlin: Springer; 2003. p. 169—83.

[50] RealNetworks. RealOnePlayer. http://www.real.com/player/.

[51] Combi C, Pozzi G. HMAP–—a temporal data model managingintervals with different granularities and indeterminacyfrom natural language sentences. VLDB J 2001;9:294—311.

[52] Keravnou ET. Amultidimensional andmultigranular model oftime for medical knowledge-based systems. J Intell InformSyst 1999;13:73—120.

[53] Combi C, Montanari A. Data models with multiple temporaldimensions: completing the picture. In: Dittrich KR, GeppertA, Norrie MC, editors. Proceedings of the 13th internationalconference CAiSE 2001–—advanced information systemsengineering Lecture Notes in Computer Science, vol.2068. Berlin: Springer; 2001. p. 187—202.

[54] Damiani E, DeCapitani Di Vimercati S, Paraboschi S, SamaratiP. A fine-grained access control system for XML documents.ACM Trans Inform Syst Security (TISSEC) 2002;5(2):169—202.

[55] Noumeir R. DICOM structured report document type defini-tion. IEEE Trans Inform Technol Biomed 2003;7(4):318—28.

[56] Grutter R, Eikemeier C, Fierz W, Steurer J. Conception of anXML-based ontology for a Web-based medical informationservice. Stud Health Technol Inform 2000;77:832—6.

[57] Philippi S, Kohler J. Using XML technology for the ontology-based semantic integration of life science databases. IEEETrans Inform Technol Biomed 2004;8(2):154—60.