information visualization for information fusion
TRANSCRIPT
Research proposal
Information Visualization for Information Fusion
byMaria Riveiro
Supervisor:Tom Ziemke
Technical Report HS- IKI -TR-07-004
School of Humanities and Informatics,University of Skovde
SE-541 28 Skovde, Sweden
June 2007
Abstract
Information fusion is a field of research that strives to establish theories, techniques andtools that exploit synergies in data retrieved from multiple sources. In many real-worldapplications huge amounts of data need to be gathered, evaluated and analyzed in order tomake the right decisions. An important key element of information fusion is the adequatepresentation of the data that guides decision-making processes efficiently. This is wheretheories and tools developed in information visualization, visual data mining and humancomputer interaction (HCI) research can be of great support.
This report presents an overview of information fusion and information visualization,highlighting the importance of the latter in information fusion research. Information vi-sualization techniques that can be used in information fusion are presented and analyzedproviding insights into its strengths and weakness. Problems and challenges regarding thepresentation of information that the decision maker faces in the ground situation awarenessscenario (GSA) lead to open questions that are assumed to be the focus of further research.
Keywords: information visualization, information fusion, decision support, situation aware-ness, uncertainty, human computer interaction, visual data mining, visual data exploration
Acknowledgment
This work was supported by the Information Fusion Research Program (University of Skovde,Sweden) in partnership with the Swedish Knowledge Foundation under grant 2003/0104(http://www.infofusion.se) and carried out in collaboration with Saab Microwave Systems(Gothenburg, Sweden). I would like to thank my supervisor Tom Ziemke for his usefulcomments and the other members of the ground situation awareness scenario for fruitfuldiscussions and valuable feedback.
1
Publications
Riveiro, M. 2007. Evaluation of Uncertainty Visualization Techniques for Information Fu-sion. Proceedings of the 10th International Conference on Information Fusion (ICIF ’07),Quebec, Canada, July 912, 2007, pp. 1-8. IEEE. Catalog Number: 07EX1591. ISBN: 978-0-662-45804-3.
Niklasson, L., Riveiro, M., Johansson, F., Dahlbom, A., Falkman, G., Ziemke, T., Brax,C., Kronhamn, T., Smedberg, M., Warston, H. and Gustavsson, P. 2007. A Unified SituationAnalysis Model for Human and Machine Situation Awareness. Proceedings of the 3rd Ger-man Workshop on Sensor Data Fusion: Trends, Solutions, Applications (SDF 2007), Bremen,Germany, September 27, 2007.
Riveiro, M. 2007. Cognitive Evaluation of Uncertainty Visualization Methods for DecisionMaking. Symposium on Applied Perception in Graphics and Visualization (APGV 2007),Tubingen, Germany, July 25-27, pp. 133. ACM SIGGRAPH. ISBN: 978-1-59593-670-7.
2
Contents
1 Introduction 71.1 Organization of the report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 Information fusion 92.1 Information and data fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.1 The JDL model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102.1.2 OODA loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Other models in information fusion . . . . . . . . . . . . . . . . . . . . . . . . 132.3 HCI aspects in information fusion . . . . . . . . . . . . . . . . . . . . . . . . . 142.4 The importance of information visualization in information fusion . . . . . . . 16
2.4.1 The influence of information presentation on decision-making . . . . . 162.4.2 Visualization and the OODA loop: observe and orient . . . . . . . . . 17
3 An introduction to information visualization 203.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.1.1 Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.1.2 Information visualization . . . . . . . . . . . . . . . . . . . . . . . . . 213.1.3 Visualization and cognition . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2 Tools, systems and applications . . . . . . . . . . . . . . . . . . . . . . . . . . 233.2.1 Visualization of large documents and software . . . . . . . . . . . . . . 243.2.2 Visualization of hierarchies . . . . . . . . . . . . . . . . . . . . . . . . 253.2.3 Focus + Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.2.4 Visualizing and Exploring the Web . . . . . . . . . . . . . . . . . . . . 263.2.5 Integrating information across multiple applications . . . . . . . . . . 26
3.3 Challenges in information visualization . . . . . . . . . . . . . . . . . . . . . . 26
4 The ground situation awareness scenario 314.1 Ground situation awareness scenario . . . . . . . . . . . . . . . . . . . . . . . 314.2 Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
5 Uncertainty Visualization 345.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.1.1 Uncertainty definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.1.2 Sources of Uncertainty . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3
5.2 Previous work on uncertainty visualization . . . . . . . . . . . . . . . . . . . . 375.3 Cognitive theories for the analysis of uncertainty visualizations . . . . . . . . 385.4 Insights from user tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415.5 Examples: cognitive and theoretical analysis of uncertain visualizations in
information fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425.5.1 Probabilistic demand prediction for traffic flow decision support . . . 425.5.2 Graphical formats to convey uncertainty in a decision making task . . 435.5.3 DSS prototype display: a critical decision analysis of aspects of naval
anti-air warfare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445.6 Conclusions and future work . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Bibliography 47
4
List of Figures
2.1 Information fusion from databases, sensors and simulations . . . . . . . . . . 102.2 Revised JDL model (1998) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.3 OODA loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.4 Waterfall model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.5 The importance of information presentation through the OODA loop . . . . . 18
3.1 Diagram of the visualization process . . . . . . . . . . . . . . . . . . . . . . . 233.2 Reference model or framework for visualization . . . . . . . . . . . . . . . . . 243.3 SeeSoft: visualizing software . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.4 Tree Map example: SmartMoney . . . . . . . . . . . . . . . . . . . . . . . . . 253.5 Cone Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.6 WebForager (site map application) . . . . . . . . . . . . . . . . . . . . . . . . 263.7 WebBook (web pages organizer) . . . . . . . . . . . . . . . . . . . . . . . . . . 263.8 Integrating information across multiple applications: historical information . 273.9 Integrating information across multiple applications: routes over maps . . . . 27
5.1 Visualization pipeline: sources of uncertainty . . . . . . . . . . . . . . . . . . 365.2 Color change (saturation) indicating uncertain information . . . . . . . . . . 385.3 Colour indicates uncertain surface . . . . . . . . . . . . . . . . . . . . . . . . 385.4 Uncertain surface indicates uncertainty . . . . . . . . . . . . . . . . . . . . . . 385.5 Box-and-whisker plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385.6 Glyphs representing angular and magnitude uncertainty . . . . . . . . . . . . 395.7 Castle reconstruction. Uncertainty is encoded into transparency . . . . . . . . 395.8 Extension of the 2D box-plot . . . . . . . . . . . . . . . . . . . . . . . . . . . 405.9 Uncertainty glyphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405.10 Hypothetical probabilistic Center Monitor . . . . . . . . . . . . . . . . . . . . 425.11 Center monitor display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435.12 Pairs of icons representing hostile or friendly objects . . . . . . . . . . . . . . 435.13 Blurred icons: probability that an object is hostile or friendly . . . . . . . . . 445.14 Graphical display used in the dynamic decision making experiment . . . . . . 445.15 DSS display features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5
List of Tables
2.1 Comparison of data/information fusion models . . . . . . . . . . . . . . . . . 152.2 Summary of HCI Enabling Technologies . . . . . . . . . . . . . . . . . . . . . 19
3.1 How information visualization amplifies cognition. . . . . . . . . . . . . . . . 30
6
Chapter 1
Introduction
We are drowning in information but starved for knowledge. This level of in-formation is clearly impossible to be handled by present means. Uncontrolled andunorganized information is no longer a resource in an information society, insteadit becomes the enemy
John Naisbitt, Megatrends
In today’s information age, the lack of information is seldom a problem. Rather theproblem is the opposite, the overload of information. The difficulty of processing and handlingvast amounts of information from multiple sources is a common feature of multiple real-lifedomains. Military operations, crisis management or homeland security applications involvea large number of actors with different characteristics, needs and behaviors. The solutionlies in the ability to process and filter the information in a manner that results in knowledge,providing responders and decision makers with improved situation awareness.
Achieving situation awareness is crucial for making effective decisions. Such awarenessin these complex situations may be difficult to achieve due to not only the overload of infor-mation but also other factors like time pressure, high stress or the imperfect and uncertainnature of the information. Hence, there is a need for decision support systems that help thedecision maker to comprehend the situation and anticipate future consequences. Fortunately,techniques, methods and tools used in information fusion can attenuate this problem.
Information fusion has been identified as key enablers for providing decision support. Itincludes theory, techniques and tools for exploiting the synergy in the information acquiredfrom multiple sources, for example sensors observing the environment, databases storingknowledge and simulations predicting future behavior.
The result of applying information fusion techniques can generate vast amounts of com-plex data that need to be analyzed by a decision maker. The presentation of the information,the graphical interface and the availability of interaction methods play a central role in theacquisition of the awareness necessary to make effective decisions. Advances in informationvisualization, interactive computer graphics (software and hardware) and human computerinteraction open new possibilities for the access, analysis, navigation and retrieval of infor-mation.
7
The focus of this report is the role that information visualization plays in decision mak-ing in the ground situation awareness (GSA) scenario that is part of the Information Fusionresearch program at the University of Skovde. The GSA scenario includes military appli-cations in a network centric warfare, surveillance, national security and civilian operations,like catastrophe management. All this applications share common characteristics, like theavailability of huge amounts of data, large number of objects/targets that are hard to identifyand classify, hidden or camouflage entities, uncertain data and hidden patterns and relations.This information is not only handled and processed by a computer system but it is typicallypresented to the decision maker who is normally under time pressure and overwhelmed bythe overload of information. In order to support the decision maker and overcome thesedifficulties, first steps in this research identify tools and techniques developed in informationvisualization, human computer interaction and visual analytics that can help users to synthe-size information; derive insights from massive and often conflicting data; detect the expectedand discover the unexpected.
1.1 Organization of the report
The report is organized as follows: chapter 2 introduces information fusion (definitions, con-cepts and models) and highlights the importance of HCI and visualization within informationfusion. Chapter 3 gives a brief overview on visualization both information and scientific vi-sualization. Visualization tools, techniques and breakthroughs in information visualizationare presented in section 3.2. Chapter 4 describes the ground situation awareness scenarioand its characteristics, HCI research needs and research questions that provide guidance onfuture research. Chapter 5 describes methods to represent uncertainty, introduces cognitivetheories for their evaluation and presents three evaluation examples of information fusionapplications.
8
Chapter 2
Information fusion
This chapter provides an introduction to data and information fusion, basic terminology andconcepts. Widely used models in information fusion, such as the JDL model and OODAloop are briefly described and alternative models are presented as well. The last two sectionsdiscuss the importance of HCI and information presentation (visualization) in informationfusion.
2.1 Information and data fusion
The term data fusion first appeared in the literature around 1960 as mathematical models fordata manipulation (Esteban, Starr, Willetts, Hannah, and Bryanston-Cross, 2005). Variouslycalled multisensor data fusion, sensor data fusion or sensor fusion, early definitions note thatdata fusion seeks to combine data from multiple sensors to perform inferences that may notbe possible from a single sensor or source alone (Hall and McMullen, 2004, chap. 1). Forexample, a definition of data fusion is given in (Steinberg, Bowman, and White, 1998-1999):
Data fusion is the process of combining data or information to estimate orpredict entity states.
Another definition of data fusion is given in (Hall and Llinas, 2000, chap. 1):
Data fusion techniques combine data from multiple sensors and related infor-mation to achieve more specific inferences than could be achieved by using a single,independent sensor.
Information fusion, as used herein, can be considered an important sub-area of data fusion(data fusion is a more general concept given that data is potential information (Meadow andYuan, 1997)). It should be noted that no longer the source of processed data are only sensors(present), but also databases (past information) and predictions (future information) (seefigure 2.1).
Information fusion has emerged as an independent research field in the last two decades(Kokar, Tomasik, and Weyman, 2004). The origins of information fusion can be traced back todevelopments in many areas, specially from the defense arena (Dasarathy, 2000). Examples
9
Figure 2.1: Information fusion from databases, sensors and simulations. Adapted from (An-dler, Niklasson, Olsson, Persson, de Vin, Wangler, Ziemke, and Planstedt, 2005)
of fields that already exploit benefits from information fusion are: robotics, maintenanceengineering, medical diagnosis, information management systems, traffic control, biometricsand military applications such as battle space intelligence, surveillance or crisis management.Dasarathy defines information fusion as follows:
Information fusion encompasses theory, techniques and tools conceived andemployed for exploiting the synergy in the information acquired from multiplesources (sensor, databases, information gathered by human, etc.) such that theresulting decision or action is in some sense better than would be possible, if thesesources were used individually without such a synergy exploitation.(Dasarathy,2001, p. 45)
2.1.1 The JDL model
The most widely used model to categorize data fusion related functions is the Data FusionModel (or JDL model) (Steinberg, Bowman, and White, 1998-1999), developed in 1985 bythe U.S. Joint Directors of Laboratories (JDL) Data Fusion Group. It is a functional model,where functions are divided into levels that relate to the refinement of objects, situations,threats and processes (Hall and Llinas, 2000, chap. 2) (see fig. 2.2). It should be noted thatthe JDL model is not a process model, because it does not specify the interaction amongthese functions within the information system.
Since its development, the JDL model has been revised several times. The first review ofthe model (in (Steinberg, Bowman, and White, 1998-1999)) broadened the definitions of levels1-3 to accommodate fusion problems beyond military and intelligence ones which had beenthe focus of earlier versions of the JDL model (Steinberg and Bowman, 2004). To addressproblems of detecting and characterizing signals a level 0 (Sub-Object Data Assessment) wasproposed. Level 4 (Process Refinement) was also emphasized in that revision as a resourcemanagement function (involving planning and control and not estimation). Nevertheless,it has been argued whether or not the level 4 (process refinement) should be consider anindependent level (Llinas, Bowman, Rogova, Steinberg, Waltz, and White, 2004). Laterrevisions of the model (Hall, Hall, and Tate, 2000) and (Blasch and Plano, 2003) added anew level, level 5, labeled as ‘Cognitive (or User) Refinement’, which addresses cognitiveissues and human computer interaction aspects.
10
Figure 2.2: Revised JDL model (1998). Redraw from (Steinberg, Bowman, and White, 1998-1999)
In this report I will refer to the levels (from level 0 to level 4) and terminology used in(Steinberg, Bowman, and White, 1998-1999). Following (Steinberg, Bowman, and White,1998-1999),(McDaniel, 2001) and (Hall and Llinas, 2000, chap. 2 and 21) the levels aredefined as follows:
• Level 0: Sub-Object Data AssessmentEstimation and prediction of signal or feature states. Examples of assignments in level0 are signal processing (e.g. analog-to-digital converter) and feature extraction.
• Level 1: Object AssessmentLevel 1 processing seeks the detection, identification, location, characterization andtracking of entities. Key functions in level 1 include: data alignment (normalizationof data with respect to time, space or other units), data correlation (determination ofwhether new data relate to existing entities), estimation of the entity state (position,velocity and attributes) and estimation of the entity identity (classification functionsto identify emitters/receivers, low-level military units, etc.)
• Level 2: Situation AssessmentLevel 2 processing seeks to understand the entities’ relationships with other entitiesand with their environment. Functions to achieve that include: object aggregation(temporal relationships, geometrical proximity, communications link and functional de-pendence among entities), event/activity aggregation (relationships in time to identify
11
activities or meaningful events), contextual interpretation (analysis of the data in con-text: weather, terrain, sea state, enemy doctrines and socio-political considerations)and multi-perspective assessment (analysis of the data with respect to the friendly,enemy and neutral forces).
• Level 3: Impact AssessmentEstimation and prediction of threats and potential opportunities. The situation is anal-ysed from a consequences point of view where alternative hypotheses are generated andprojected into the future to determine courses of action. Key functions within level3 include: capability estimation (like size of the enemy forces), predict enemy intent(enemy doctrine), identify threat opportunities (identification of potential enemy threatbased on enemy actions, operation readiness analysis, environmental conditions, etc.),multi-perspective assessment (analyses of enemy, friendly and neutral forces), offen-sive/defensive analysis (prediction of hypothesized enemy engagements, enemy doctrine,weapon models, etc.)
• Level 4: Process RefinementLevel 4 (considered as a meta-process) seeks to monitor and optimize the overall datafusion process. Function examples within this level include: evaluation of the perfor-mance and effectiveness of the fusion process, source requirements and needs, missionmanagement (recommendation for allocation and direction of resources) and data basemanagement functions.
Level 5: Cognitive/User Refinement The need for a level 5 was first proposed in(Hall, Hall, and Tate, 2000) and labeled as Cognitive Refinement. The authors consider thatextensive research in data fusion has focused on the data processing, from sensor data to agraphics display and little has been done in order to support a human decision-maker in theloop. Key functions within level 5 include cognitive aids and human computer interaction(Hall and McMullen, 2004, chap. 9)(Blasch and Plano, 2002) re-labeled Level 5 as User Refinement. In the same line as theprevious authors, Blasch and Plano claim that the JDL model is only for automatic processingof a machine and does not account for human processing. Issues like trust, workload, attentionand situation awareness must be taken into account in the design of a fusion system whichsupports a user. Later publications, (Blasch and Plano, 2004), redefine the User RefinementLevel and propose the JDL-User Model, (Blasch and Plano, 2003).
From my point of view, the functionality included in level 5 should be considered in all theother levels, so there is no need to create an isolated level to group these functions. Moreover,level 5 is at this stage just a proposal and it has not been widely accepted as a level of theJDL model.
2.1.2 OODA loop
One of the most commonly used model to describe the decision making process in informationfusion is Boyd’s OODA (Observe-Orient-Decide-Act) loop, (Boyd, 1987). It has its origins in
12
the military domain, like the JDL model, but it focuses in the human (command and controlgroup) decision process. Boyd considers four main activities in the decision process:
• Observe: the environment
• Orient: position yourself in the environment
• Decide: make a decision
• Act: perform the decision
The model illustrates the ultimate goal of a decision maker, taking the right decision withinthe minimum time, where speed is a condition for winning.
Figure 2.3: OODA loop. Redraw from (Brehmer, 2005)
In spite of being the dominant model for command and control, Boyd’s OODA loop hasbeen criticized from two perspectives: it does not describe the decision making process in themilitary domain nor the decision making process in general (in (Brehmer, 2005) citing otherauthors: Bateman III and Bryant). In order to improve the OODA loop model by Boydand emphasize its dynamic nature, Brehmer in (Brehmer, 2005) developed the DOODA(Dynamic-OODA) loop using cybernetics models for command and control. More detailscan be found in (Brehmer, 2005).
2.2 Other models in information fusion
Even though the JDL model and the OODA loop are widely used models in the informa-tion fusion community one can find alternative models in the literature. For example theOmnibus Model (Bedworth and O’Brien, 2000) which draws together each of the previousmodels (with their advantages and overcoming some of their disadvantages) and presents ageneral terminology of data fusion technology.Dasarathy’s Functional Model defines a natural categorization of data fusion functionsin regard to types of data or information processed (input) and types of results from theprocess (output) (Hall and McMullen, 2004, chap. 2). For example, neural networks or clus-ter algorithms tend to be pattern classification algorithms that transform an input featurevector into an output feature vector (Hall and McMullen, 2004, chap. 2). An expanded viewof Dasarathy’s model and its mapping to the JDL model, levels 0-4, is given in (Hall and
13
Llinas, 2000, chap. 2-15).An example of hierarchical architecture often used by the data fusion community (Esteban,Starr, Willetts, Hannah, and Bryanston-Cross, 2005) is the Waterfall Model, describedin (Harris, Bailey, and Dodd, 1998). The flow of data operates from the data level to thedecision-making level where the sensor system module (level 1) is continuously updated withfeedback from the decision- making module (level-3), see figure 2.4
A comparison among the different models is shown in table 2.1
Figure 2.4: Waterfall model. Redraw from (Esteban, Starr, Willetts, Hannah, and Bryanston-Cross, 2005)
2.3 HCI aspects in information fusion
The development of fully automated data systems has grown in the past decades. In almostany fairly complex systems, like nuclear reactors and aircrafts manual tasks are being replacedby automated functions (Matheus, Kokar, and Baclawski, 2003). It is, as well, a natural pro-cess, where the automatic part of the information fusion process is growing (for example,automatic target recognition applications). Nevertheless, many information fusion applica-tions are design for an human decision maker, and perhaps without enough consideration ofits user. The lack of research in HCI related issues has been acknowledged by many authorsin the information fusion Community. For example (Hall, Hall, and Tate, 2000), (Blaschand Plano, 2002) and (Hall and McMullen, 2004, chap. 9). The traditional approach, seeJDL model, shows that data flows from sensors (source) toward the human (receiver). Thiscould be a very simplistic interpretation, given that the human is actually involved in eacheach step of the fusion process. However, using this basic orientation, rich information fromsensors is compressed for display on a two dimensional computer screen ((Hall and Llinas,2000, chap. 19), referred as the “HCI bottleneck” problem by the authors).
14
Table 2.1: Comparison of data/information fusion models. Adapted from (Hall and Llinas,2000, chap. 2-17)
Activity Waterfall JDL Boyd Intelligencebeing model model loop cycle
undertaken
Command execution Act DisseminateDecision Decision level 4 Decide
making process makingThreat level 3 Orient Evaluate
assessmentSituation Situation level 2
assessment assessmentInformation Pattern level 1 Collateprocessing processing
Featureextraction
Signal Signal level 0processing processing
Source/sensor Sensing Observe Collectacquisition
The effectiveness of a general and non-fully automatic information system highly dependson the human performance. More research is needed (Hall and Llinas, 2000, chap. 21)to understand information access preferences, how users perceive and process information,interact with the system and make decisions. Additionally (Waltz and Llinas, 1990) suggeststhat the overall effectiveness of a data fusion system is affected by the HCI efficacy. Newadvances should enhance the link between effective human cognition and the informationfusion system, considering the human as the centre of the fusion process.
In order to overcome the HCI bottleneck in the information fusion process and accountfor functions for information representation and human machine interaction, Hall, Hall andTate, (Hall, Hall, and Tate, 2000), proposed the introduction of a new level in the JDL model,Level 5: Cognitive Refinement. Level 5 processing involves developing functions to supporta human decision-maker in the loop, users in collaborative environments and cognitive aids.Examples of functions for level 5 processing are (adapted from (Hall and Llinas, 2000, chap.19-9)):
• Cognitive aids: functions to aid and assist human understanding and exploitation ofdata.
• Negative reasoning enhancement: humans have a tendency to seek for informationwhich supports their hypothesis and ignore negative information. Techniques to over-come the tendency to seek confirmatory evidence could be developed.
15
• Uncertainty representation: methods and techniques to improve the representation ofuncertainty.
• Time compression/expansion: time compression and time expansion replay techniquescould assist the understanding of evolving tactical situations, on account of humancapabilities to detect changes.
• Focus/defocus of attention: techniques to assist in directing the attention of an analystto consider different aspects of data.
• Pattern morphing methods: methods to translate patterns of data into forms that aremore easy for an human to interpret
Information fusion can certainly benefit from developments in information visualizationresearch. At the same time information fusion applications provide a valuable source ofcase studies for researchers within the field of information visualization and human-computerinteraction
2.4 The importance of information visualization in informa-tion fusion
Many information fusion applications process and present huge quantities of data in orderto enable an operator to make effective decisions. Commonly, the visual system is the coreof the interface between the operator and the information system. Through the display theoperator perceives data, process information and acquires knowledge (from data to knowledgeand wisdom, (Shedroff, 2001)) in order to achieve some degree of situation awareness whichwill allow decision making. The the temporal limitations, the character uncertain of theinformation handled and how the information is presented clearly influences the decisionprocess.
The following sections highlight the importance of information presentation on decisionmaking.
2.4.1 The influence of information presentation on decision-making
Human decision-making can be seen as a complex information processing task. Regardingthe level of complexity, decisions can be divided in three main groups (Rasmussen, 1987):skill-based sensor-motor behavior (automated or unconscious performance), rule-based behav-ior (simple procedural skills for well-practiced or simple tasks) and on the highest level ofcomplexity, knowledge-based behavior. Knowledge-based behavior represents the most com-plex cognitive processing, used to solve unfamiliar problems or make decisions that requiredealing with huge amounts of information and usually, with its associated uncertainty.
According to Endsley (Endsley, 1995), certain level of situation awareness must be reachedin order to make a complex decision. Endsley defines situation awareness as the perceptionof the elements in the environment within a volume of time and space, the comprehension oftheir meaning and the projection of their status in the near future. Hence, situation awareness
16
involves how to perceive, comprehend and project data/information. First, attributes anddynamics of the elements in the environment are perceived, then multiple pieces of informa-tion are integrated and their relevance to the decision maker’s goals is determined and at theprojection level, future events are predicted. When a decision-maker faces a complex problema mental model of the environment is built. In this process, humans seek for informationthat can help them to understand the situation.
In many real-world applications, the interface between an user and a computer or com-puter system includes a display/-s that connects the environment with the user who perceives,processes and makes a decision. Therefore a key element in the construction of the operator’senvironmental mental model is the adequate presentation of the data that guides the decision-making processes efficiently. Among others, Card (Card, Mackinlay, and Shneiderman, 1999)and Tufte (Tufte, 2001) have highlighted the importance of information presentation on de-cision making.
Past research has investigated the role of information presentation on decision-making.Cognitive fit theory (Vessey, 1991) can be used as a theoretical framework for analyzinghow the presentation of information affects decision making. Cognitive fit theory states thatdecision making is improved when the representation of the information matches the problemsolving task, since decision makers develop a more accurate mental model of the problem.
2.4.2 Visualization and the OODA loop: observe and orient
The OODA loop (see section 2.1.2 for a detailed description) illustrates the ultimate goalof a decision maker, taking the right decision within the minimum time, where speed is acondition for winning. That includes both our own and our opponent’s decisions in theground situation awareness (GSA) scenario. The minimization of the time consumed in theobserve and orient phases of the OODA loop is essential in order to reduce the overallprocessing time. Here is where the visual system plays an extremely important role:
• Observe.There are many sources which provide the user with new information: reports, audio,environment, interaction with other users (organization), etc. but it is the visual displayone of the most valuable information sources, ”more information is acquired throughthe vision than through all of the other senses combined”, (Ware, 2000).
• Orient. As Boyd describes, the orientation phase is affected by factors like geneticheritage, cultural traditions and previous experience, as well as the mental processes ofanalysis and synthesis. Thereby the visual system should act as a cognitive tool in orderto facilitate the mental processes of analysis and synthesis stage: presenting the infor-mation avoiding overload, overcoming human preceptive deficiencies and supportingthe interaction human-machine.
• DecideThe displayed information will not only portray the topographical environment, condi-tion and state of friendly forces, enemy forces, neutral actors or weather conditions. It
17
should increment command and control ability to understand the battle flow of infor-mation and aid them to consider all the possible options and tactics and their impactand consequences in the battle space. Therefore the visualization system should bedesigned as an important part of the decision making support software. It is the bridgebetween data and seeing the picture which will allow the commanders to envision tac-tical alternatives, (Barnes, 2003).
The fig. 2.5 presents an interpretation of the OODA loop. The graphical interface, inthe figure visual display, presents new information to the user and hence will influence theorient and the decide stage. The user interface should be designed taking into account howhumans perceive, process, analyse information and make decisions. The ultimate goal of theuser interface, including the graphical display, is the support of all these activities in orderto speed the loop up.
Figure 2.5: My interpretation of the OODA loop. Orient and Decide takes place in the user/-smind. Observe brings the new information to the user, through e.g. the user interface. Thusthe user interface must be carefully designed to support decision making.
18
Table 2.2: Summary of HCI Enabling Technologies. Adapted from (Hall and McMullen,2004, chap. 9, pp. 322-323)
HCI Description Current TechnologyCompo- Technology Trendsnent
Devices for display -Large, high- fidelity -Proliferation of 3Dof information to a graphic screens full immersionhuman operator. -High density environments (e.g.
Examples: television for collaborativeVisual alphanumeric -High-end 3D devices data analysis)
displays displays, icons and -Personal displays -Drive towardsgraphics devices; associated with increased realityoptical devices wearable computers via gamingto monitor the industryeye motion to -Interactive displays
determine where theuser is looking
SW and HW -COTS voice -Increasedto provide recognition SW interactivityeither aural -Web search (voice recognition
feedback to a engines using and voice feedback)Aural user; voice pseudo natural -Emulation of human
interaction recognition and language facial expressionsnatural language -Syntactic analysis and voice inflections
processing -3D sound
Devices that allow -Standard mechanical -Increased sensitivitya user to interfaces and feedback.
provide commands -Trend towardsto a computer -Experimental haptic full-body haptic
Haptic system (mouse, devices with interfacesdevices joystick, touch emulation -Link between
touch screen) visual interface,acoustic and haptic-Wireless interfaces
19
Chapter 3
An introduction to informationvisualization
3.1 Introduction
3.1.1 Visualization
The term visualization is frequently used ambiguously, referring to graphical representationsthat are normally carried out by a computer. Nevertheless its primary definition states thatvisualization is an activity carried out by a human being and not by a graphic engine. Inorder to exemplify, the following definitions are taken from different dictionaries:
1. visualize: v. to form a mental image of; imagine. (from The Concise Oxford EnglishDictionary)
2. visualize: v. to form a picture of someone or something in your mind, in order toimagine or remember them. (from Cambridge Advanced Learner’s Dictionary)
3. visualization: the display of data with the aim of maximizing comprehension rather thanphotographic realism.(from A Dictionary of Computing. Oxford Reference Online.)
In (MacEachren, 1995) and (Ware, 2000) visualization is referred to as a cognitive activity.An activity in which humans are engaged (Spence, 2001) as an internal construct of the mindand therefore cannot be printed on a paper or displayed on a screen.
However in The Visualization Handbook (Hansen and Johnson, 2005), Hansen and John-son use the definition described in the 1987 National Science Foundations Visualization inScientific Computing Workshop report:
Visualization is a method of computing. It transforms the symbolic into thegeometric, enabling researchers to observe their simulations and computations.Visualization offers a method for seeing the unseen. It enriches the process ofscientific discovery and fosters profound and unexpected insights...The goal ofvisualization is to leverage existing scientific methods by providing new scientificinsight through visual methods.
20
They treat visualization as the process of creating images that convey salient informationabout underlying data. Human perception is considered separately (part X of the book:Perceptual Issues in Visualization).
In (Card, Mackinlay, and Shneiderman, 1999), Card et al. define visualization as follows:”the use of computer-supported, interactive, visual representations of data to amplify cogni-tion.” This definition contains both aspects, visualization as a graphical representation onthe computer display and visualization as a cognitive activity.
3.1.2 Information visualization
Information visualization as a field of research is relatively new, with approximately ten yearsof history (Chen, 2004). It is also an interdisciplinary field, it combines disciplines such ascomputer graphics, HCI, communication theory, cognitive science and graphic design; andhas influences from many domains, for example, World Wide Web, information retrieval,medical and bioinformatics applications, hypertext and virtual reality environments.
In order to reduce the scope of this ground, I will give some remarkable informationvisualization definitions.
In (Card, Mackinlay, and Shneiderman, 1999) Information Visualization is defined asfollows: ”the use of computer-supported, interactive, visual representations of abstract datato amplify cognition”. The only difference between this definition and the above definition ofvisualization given by Card et al. in (Card, Mackinlay, and Shneiderman, 1999) is the use of”abstract data” instead of ”data”.
In the following paragraph, (Ware, 2000), a more detailed definition of information visu-alization linking cognition and graphical engines is given:
What information visualization is really about is external cognition, that is,how resources outside the mind can be used to boost the cognitive capabilities ofthe mind. Hence the study of information visualization involves analysis of boththe machine side and the human side. Almost any interesting task is too difficultto be done purely mentally. Information visualization enables mental operationswith rapid access to large amounts of data outside the mind, enables substitutingof perceptual relation detection for some cognitive inferencing, reduces demandson user working memory, and enables the machine to become a co-participant ina joint task, changing the visualizations dynamically as the work proceeds.
A similar definition is given by the User Interface Research Group in Palo Alto (Parc-Xerox):
Information Visualization is the use of computer-supported interactive visualrepresentations of abstract data to amplify cognition. Whereas scientific visual-ization usually starts with a natural physical representation, Information Visual-ization applies visual processing to abstract information. This area arises becauseof trends in technology and information scale ... Information Visualization is a
21
form of external cognition, using resources in the world outside the mind to am-plify what the mind can do.(http://www2.parc.com/istl/projects/uir/projects/ii.html)
Two major areas have been traditionally considered inside Visualization, InformationVisualization and Scientific Visualization.
(Mackinlay, 2000) argues that scientific visualization focuses on physical data while infor-mation visualization does in abstract data. Examples of physical data are the human body,the earth or 3D medical pictures whereas information visualization treats non-physical datasuch as text, hierarchies or statistical data. Nevertheless the differences and the boundariesbetween information and scientific visualization are not clear. Examples of that are given by(Tory and Moller, 2002) and (Rhyne, 2003). Chen also shares this opinion in (Chen, 2004,Chapter 1, p.1): ”The boundary between information visualization and related fields such asscientific visualization and simulation modeling is becoming increasingly blurred.”
In (Tory and Moller, 2002), the authors suggested a new terminology arguing that theexisting one is vague and ambiguous. They proposed a new classification based on character-istics of models of the data rather than on characteristics of the data itself: continuous modelvisualization and discrete model visualization. Continuous model visualization refers to thevisualization algorithms using continuous models of the data where the phenomenon beingstudied is continuous even though the data is discrete. On the other hand, discrete modelvisualization encompasses visualization algorithms which employ discrete data models.
In (Rhyne, 2003), ”Does the difference between Information and Scientific VisualizationReally Matter?”, Rhyne questions the difference between scientific and information visual-ization. To demonstrate that information visualization is not unscientific and scientific visu-alization is not uninformative, as stated by (Munzner, 2002), the author uses two scenarioswhere this two major areas overlap: geographic information and bioinformatics visualization.
3.1.3 Visualization and cognition
Thinking is not something that goes on entirely, or even mostly, inside the users’ heads(Hutchins, 1995). Most knowledge is acquired or used as an interaction with cognitive toolsand individuals and operating within social networks. More and more is the computer as aninformation system acting as a cognitive tool. The visual system, as a part of the informationsystem functions as well as a cognitive tool (Ware, 2000) and its cognitive nature holds thedifficulty of its study (Spence, 2001).
Visual displays provide the highest bandwidth channel from the computer tothe human: more information is acquired through the vision than through all ofthe other senses combined (Ware, 2000).
But how can visualization amplify cognition? The work by (Larkin and Simon, 1987) isone of the seminal studies analysing why graphical representations are effective. Their resultsshow that diagrams helped reducing the effort to some specific task in three ways:
1. The search is reduced because diagrams can group together information that is usedtogether.
22
Figure 3.1: Diagram of the visualization process. Adapted from (Ware, 2000).
2. Search and working memory is also reduced by using location to group informationabout an element.
3. Graphical representations automatically support a large number of inferences that areeasy for humans.
The ways in which graphical representations can amplify cognition is extended by (Card,Mackinlay, and Shneiderman, 1999) to six:
1. By increasing memory and processing resources available
2. By reducing the time to search for information
3. Enhancing the detections of patterns through visual representations
4. Enabling perceptual inference operations
5. By using perceptual attention mechanisms for monitoring
6. By encoding information in a manipulable medium
Table 3.1 shows different mechanisms that can enhance cognition. The table is takenfrom (Tory and Moller, 2004) but the basic content of the table is the same as in (Card,Mackinlay, and Shneiderman, 1999, table 1.3, p. 16).
3.2 Tools, systems and applications
Visualization provides an interface between two powerful information processing systems: thecomputer and the human mind (Card, Mackinlay, and Shneiderman, 1999). Effective visual
23
Figure 3.2: Reference model or framework for visualization. ”Visualization can be describedas the mapping of data to visual form that supports human interaction in a workspace forvisual sense making.” Adapted from (Card, Mackinlay, and Shneiderman, 1999, p. 17)
interfaces allow the interaction with large volumes of data and the discovery of patterns,hidden characteristics and trends among these data. This section briefly illustrates someinteresting information visualization applications.
3.2.1 Visualization of large documents and software
Large bodies of text and great number of documents need to be analyzed in many applications,like for example in software development. Visualization methods can help the user to browsethrough the data in order to find interesting pieces or relations among the documents. Anexample of visualization tool in this group is SeeSoft. SeeSoft (Eick, Steffen, and Sumner,1992) uses colored rows and columns to represent the frequency of use of certain lines of code(see figure 3.3).
Figure 3.3: SeeSoft visualizing software consisting of 38 files comprising 12037 lines of code.The newest lines are shown in red, the oldest in blue, with a rainbow color scale in between.
24
3.2.2 Visualization of hierarchies
Hierarchies or trees are abstractions of information structures (Chen, 2004). Visualizing treesis a mature branch in information visualization. A number of algorithms and approaches torepresent hierarchies have concentrated on the focus vs context problem. Two well knownexamples in this category are tree maps and cone trees. Tree maps (TreeMaps) are defined by(Johnson and Shneiderman, 1991) as a space-filling approach to the visualization of hierarchi-cal information (see figure 3.4). A three dimensional representation of very large hierarchicalinformation is the cone tree, developed by (Robertson, J., and Card, 1991) (an example canbe seen in figure 3.5).
Figure 3.4: SmartMoney: a market map of USstocks (generated on 17th September 1999),www.smartmoney.com/marketmap/
Figure 3.5: Cone Tree example (Robertson, J.,and Card, 1991)
3.2.3 Focus + Context
In many occasions users need to access low level details and high level contextual informationat the same time or in the same viewing area. The challenge is the compromise between detailand overview on a limited sized display. There are three approaches to overcome this tension(Chen, 2004):
• Overview + detail views: displaying overview and detailed information in multipleviews.
• Zoomable views (or multiscale displays): displaying objects on multiple scales. A goodexample of multiscale displays is Pad++ (Bederson and Meyer, 1998).
• Focus + context views: displaying local detail and global context in integrated butgeometrically distorted views (there is a large group of distortion techniques available,e.g fisheye or bifocal views). Examples of these techniques can be seen in (Furnas, 1986)(Furnas established the foundation of fisheye views) or (Spence and Apperley, 1982) (agood example of a bifocal view).
The potential solutions presented here have different strengths and weakness regarding easyof use, overall effectiveness and simplicity (Chen, 2004). Overview + detail solutions become
25
problematic when the user moves among the split displays of overview and detailed infor-mation. Zoomable views may lose the contextual information temporarily while the thirdapproach, focus + context techniques, may lose the sense of continuity when the focal pointis drastically changed.
3.2.4 Visualizing and Exploring the Web
The visualization of site maps or navigation trails is a challenging problem in informationvisualization. One of the most predominant applications of information visualization on theweb are site maps and there is a great number of site map construction tools. WebForager(Card, Robertson, and York, 1996) (figure 3.6) and MAPA (developed by Dynamic Diagrams)are two of them. Another relevant tool is the WebBook, also presented in (Card, Robertson,and York, 1996). WebBook is a 3D interactive book of HTML pages that organizes pagesand supports web exploration.
Figure 3.6: WebForager from (Card, Robert-son, and York, 1996).
Figure 3.7: WebBook (web pages organizer)from (Card, Robertson, and York, 1996).
3.2.5 Integrating information across multiple applications
Real life situations require the presentation of information from multiple sources while theapplication may require different visualization tools and techniques (Gershon, Eick, and Card,1998). Researches in information visualization have developed a great number of visualizationtechniques for specific data. Nevertheless there is a lack of methods to integrate informationacross multiple applications (Gershon, Eick, and Card, 1998). An exception regarding thismatter is the work of (Kolojejchick, Roth, and Lucas, 1997), where the authors describe asuite of basic tools into an integrated information workspace (see figures 3.8 and 3.9).
3.3 Challenges in information visualization
In this section I summarize the most important unresolved research problems in visualization,from both views, information and scientific visualization. A good orientation of issues thatshould be explored in the future is also given in (Rhyne, Hibbard, Johnson, Chen, and Eick,
26
Figure 3.8: The populated Daily Tag Sum-mary frame shows historical information forthe tags arriving, leaving and remaining dailyat a specified staging area. The data withinthe frame can be specified by dropping dataon it from other frames (Kolojejchick, Roth,and Lucas, 1997).
Figure 3.9: Newly recomposed collections ofshipments are dragged to the original Routesframe where they are shown at their destina-tion (Kolojejchick, Roth, and Lucas, 1997).
2004) (this panel, presented at the IEEE Visualization Conference, includes opinions fromcited authors in Visualization).
1. Visualization as a ScienceFrequently, application developers of visualization technology do not spend enough timetrying to understand the underlying science (Johnson, 2004). Johnson suggests a directcollaboration with application scientists for working side by side with end users.
Therese-Marie Rhyne in (Rhyne, Hibbard, Johnson, Chen, and Eick, 2004) suggestsinter-disciplinary and intra-disciplinary collaboration as a key to solve challenges invisualization. Examples of these challenges are: ”How does our discipline effectivelytransfer its concepts and methods to specific domain scientists and experts who desire toapply visualization techniques? Does the Renaissance Team concept work as we extendvisualization methods to hardware designed for computer games and mobile devices?Can there be an effective interchange between the information visualization and scien-tific visualization communities.”
2. InteractionOne of the top research goals for visualization is effective human-computer interaction
27
(Johnson, 2004). Many authors have addressed this challenge as essential in the futureof visualization research, (Hibbard, 1999), (Johnson, 2004), (Tory and Moller, 2004).Other authors use interaction as a property to praise in their applications, e.g. SimVisdescribed in (Hauser, 2005). Examples of interactive environments (virtual reality),their progress and future challenges can be found in (van Dam, Forsberg, Laidlaw,LaViola, and Simpson, 2000).
Hibbard in (Hibbard, 1999) points out the importance of user manipulation of graphicalrepresentations. He states that new ways of supporting interaction must be developed,for example the integration of gestures and speech parsing to the user interface. Infuture, direct manipulation will be essential for users in a immersing virtual world andthere will be enormous complexity in the way that users manipulate visualization. Inaddition, Hibbard proposes the optimization of physical resources in order to underpininteraction. Solutions which allow interaction should include:
(a) parallel algorithms for common operations,
(b) strategies for data replication and movement in the memory hierarchy and on thenetwork (interactions shared by multiple users).
The graphical representations should provide features to select, group and rearrangethe information.
3. Collaborative and distributed visualizationComputers are mediators of human-to-human and human-to-data interaction. Manyapplications require team work and/or team decision making and support a certaincommon situational picture. In other occasions, domain experts are spread out inspace (and even across time zones) and collaboration has to proceed remotely insteadof face to face. Distributed teams need to collaborate and access shared data whilesome of their team members are outside of their office environments, presenting a newset of challenges in terms of data usage and share. Collaboration involves re-thinkinginterfaces in order to promote and facilitate joint work (Cartwright, Crampton, Gartner,Miller, Mitchell, Siekierska, and Wood, 2001). In military or near military settings, thisaspects are of great interest due to the need of achieving a common situation awarenessthrough a common picture.
4. Represent error and uncertaintyJohnson addresses this challenge in (Johnson and Sanderson, 2003), (Rhyne, Hibbard,Johnson, Chen, and Eick, 2004) and (Johnson, 2004). This unresolved problem is notnew and many many articles have been written suggesting new methods for visualizinguncertainty or surveying existing uncertainty visualization techniques, e.g., the citedclassification by Pang in (Pang, Wittenbrink, and Lodha, 1997). Other examples ad-dressing this problem in general are (Gershon, 1998), (Thomson, Hetzler, MacEachren,Gahegan, and Pavel, 2005) and (Griethe and Schumann, 2006); an example regardingthe geo-spatial area can be found in (MacEachren, Robinson, Hopper, Gardner, Murray,Gahegan, and Hetzler, 2005)
28
The representation of error and uncertain information is crucial when large amountsof data have to be analyzed and evaluated in order to take a decision (Griethe andSchumann, 2005). In (Griethe and Schumann, 2005) the importance of uncertaintyannotation is stated for supporting high level tasks, like decision making. A moredetailed description regarding the representation of uncertainty can be read in chapter5.
5. Perception and cognition-based design.
The effectiveness of a visualization depends on perception, cognition, andthe user’s specific tasks and goals. (Tory and Moller, 2004)
(Tory and Moller, 2004) highlight the importance of human factors in the visualizationprocess. They suggest that how people perceive and interact with a visualization toolshould play a more important role in the design and posterior evaluation of a visualsystem.
In addition Mackinlay in (Mackinlay, 2000) emphasizes human perception as a futurechallenge for information visualization. From his point of view, visualization createsa feedback loop between perceptual stimuli and the user’s cognition but the existingknowledge about human perception and presentation design is still insufficient.
6. Empirical studiesResearch in information visualization has been dominated by refined innovations, elab-orated visual representation, powerful methods and astonishing applications. However,empirical evaluations that validate their usefulness are often overlooked (Chen, 2004).Empirical evidence bring light on what fails, what works and what remains unknown(Chen, 2004). This matter has been acknowledged for many authors (Chen and Czer-winski, 2000), for example in a especial issue on empirical studies of information visu-alizations in Int. Journal of Human Computer Studies (e.g. (Morse, Lewis, and Olsen,2000)). Established methodologies from HCI and psychology can be incorporated ininformation visualization.
From my point of view, empirical evaluations must be performed for newly developedtechniques and cannot be overshadowed for what is possible rather than for what shouldbe done.
29
Table 3.1: How information visualization amplifies cognition. (Tory and Moller, 2004) using(Card, Mackinlay, and Shneiderman, 1999, table 1.3, p.16). Adapted from (Tory and Moller,2004).
Method Description
Increased Resources:Parallel processing Parallel processing by the visual system can increase the
bandwidth of information extraction from the data
Offload work to the With an appropriate visualization, some tasks canperceptual system be done using simple perceptual operations
External memory Visualizations are external data representations that canreduce demands on human memory
Increased storage Visualizations can store large amounts of informationaccessibility in an easily accessible formReduced Search:Grouping Visualizations can group related information for easy
search and access
High data density Visualizations can represent a large quantity of datain a small space
Structure Imposing structure on data and tasks can reducetask complexity
Enhanced Recognition:Recognition instead Recognizing information presented visually can beof recall easier than recalling information
Abstraction and Selective omission and aggregation of data can allowaggregation higher level patterns to be recognized
Perceptual Monitoring: Using pre-attentive visual characteristicsallows monitoring of a large number of potential events
Manipulable Medium: Visualizations can allow interactive explorationthrough manipulation of parameter values
Organization Manipulating the structural organization of data can allowdifferent patterns to be recognized
30
Chapter 4
The ground situation awarenessscenario
This chapter briefly describes the characteristics of the ground situation awareness scenarioand presents general research questions in section 4.2.
4.1 Ground situation awareness scenario
Beside the study of generic aspects of information fusion, infrastructures, methods and algo-rithms, the Information Fusion Research Program at the University of Skovde is involved inmultiple application oriented scenarios. Examples of these scenarios are precision agriculture,bioinformatics, simulation-based for manufacturing decision support and information fusionfor rapid decision making in network-based systems (ground situation awareness, GSA).
The GSA scenario includes military applications in a network centric warfare, surveil-lance, national security and civilian operations, like catastrophe management. The goal ofthe information fusion research within GSA is to support decision making increasing the sit-uation awareness (SAW) by adding automatic and semi-automatic fusion processes (Andler,2005). Before a decision maker can actually decide what to do, the relevant universe of dis-course needs to be observed and analyzed in order for the decision maker to become awareof how the observations relate to each other and influence potential decisions. The process ofachieving SAW is called situation analysis (SA) (Matheus, Kokar, and Baclawski, 2003). SAis the process of examining a given situation, its elements and relations to provide a state ofsituation awareness for decision makers (Roy, Breton, and Paradis, 2001).
What is situation awareness? A general definition of SA is “the upto-the minute cog-nizance required to operate or to maintain a system” (Adams, Tenney, and Pew, 1995).(Endsley, 1995) focused more on the process and defines situation awareness as the percep-tion of elements in the environment, the comprehension of their meaning and the projectionof their status into the near future. The term situation awareness is commonly used by theHCI community referring to a process that occurs in the mind of the operator:
31
It should be clearly noted, however, that technological systems do not provideSA in and of themselves. It takes a human operator to perceive information tomake it useful. (Endsley and Garland, 2000)
SA is, however, a mental state and cannot be directly interacted with by useof technology. (Wallenius, 2004)
Manual tasks are more and more being replaced by automated functions. However, inmany real-world applications human operators are still responsible for managing SAW. Thisraises new kinds of problems due to human limitations in maintaining SAW (Matheus, Kokar,and Baclawski, 2003) (SAW literature presents many examples of incidents and accidentswhich could have been avoided if operators had recognized the situation in time, see (Endsleyand Garland, 2000)).
4.2 Research questions
The GSA scenario is characterized by huge amounts of information, very large number ofobjects/targets that are hard to identify and classify, hidden or camouflage entities, uncertaindata and hidden patterns and relations. This information is not only handled and processedby the fusion system but it is typically visualized and presented to the decision maker whois normally under time pressure and overwhelmed by the overload of information.
In the described domain the information presented to the user should be easily assessedto effectively support high-level analytical tasks like decision making.
The general research question is how to visualize fused situation analysis information incircumstances characterized by information overload and time pressure. The problem domainis characterized by the need for rapid decision making and the presence of many differentinformation sources. The aim within this work is to speed up and improve the decision pro-cess by means of processing and presenting the information: filtering information, enhancinghuman capabilities, supporting user-system interaction, etc.
These are some interesting research aspects that can constitute the focus of further re-search:
• Visualization of uncertainty, information reliability and quality of information. Thisquestion includes how to represent uncertain information to a user in such a way thathe/she is aware of its nature. As example of previous work concerning representation ofuncertainty can be found in (Bisantz, R. Finger, and Llinas, 1999). Preliminary resultsand uncertainty visualization methods can be found in chapter 5.Two general research challenges in uncertainty visualization that are highly relevant toinformation fusion are: (1) the development of representation and evaluation methodsfor depicting multiple forms of uncertainty in the same display and (2) the developmentof methods and tools for interacting with uncertainty representations (MacEachren,Robinson, Hopper, Gardner, Murray, Gahegan, and Hetzler, 2005).
32
• Interactivity of visualization, depending on the user’s needs, his/hers experience andthe level of trust in the system (involving the user in the fusion process).
• Past, present and future (predicted) information. The ultimate purpose of visualizationaids to increase the commander’s ability to understand the battle dynamics, consideroptions and predict outcomes.” (Barnes, 2003). The system should provide with a timeframe picture, showing the past, present and future state reflecting the impacts of theactions.
• Different levels of abstraction or granularity (in time and space).
• Collaborative visualization, enhancing group work (command and control)
The visualization system should be designed with an understanding of how users perceiveand process information, interact with the system and make decisions. Additionally thevisualization system should include the particularities of every task and reflect how usersoperate individually (role-based and user-centred design) and in collaborative environments.
33
Chapter 5
Uncertainty Visualization
As preliminary work, this chapter reviews general ways of representing uncertainty. The sur-vey also includes definitions of uncertainty and a general framework to identify its sources.Perceptual and cognitive theories from Tufte, Bertin and Chambers are described for thetheoretical analysis and evaluation of uncertainty visualizations. These theories can provideinsights into the weakness and strengths of existing and new developed uncertainty visual-ization techniques 1.
5.1 Introduction
Information visualization can provide valuable assistance to decision-makers. In many appli-cations the displayed data is imperfect and has some degree of associated uncertainty. Therecognition of uncertainty and the awareness on the uncertain nature of the information iscrucial in the decision-making process. Therefore, uncertainty should be appropriately repre-sented in order to avoid misinterpretations that may led to inaccurate conclusions. Neverthe-less, most of the developed techniques to represent uncertainty do not include a perceptualand cognitive evaluation that validates its usefulness.
Many applications process and present huge quantity of data in order to enable an op-erator to make effective decisions. Since there could be errors in the acquisition of the data(measuring devices), processing phases (data mining techniques), performance reasons, oreven in the graphical display of the information (Griethe and Schumann, 2005), some degreeof uncertainty is almost always associated (Cedilnik and Rheingans, 2000). If visualizationis used to communicate the content of the data or to explore it, the uncertainty needs to beincluded (Griethe and Schumann, 2006). Moreover, the user should be aware of the natureand the degree of uncertainty of the displayed information, otherwise, there is a danger thatdata can be misinterpreted, potentially leading to inaccurate conclusions.
Even though the need for visualizing uncertainty associated with the data has now gen-eral acceptance (Zuk and Carpendale, 2006), most of the visualization research communityhas ignored or separated the presentation of the uncertainty from the data (Pang, Witten-brink, and Lodha, 1997). Part of the reasons are: it is not easy to include additional un-
1An extended version of this chapter can be read in (Riveiro, 2007)
34
certainty information into an existing visualization while maintaining comprehension ((Zukand Carpendale, 2006),(Cedilnik and Rheingans, 2000)) and there is a lack of methods thatpresent uncertainty along with data (Pang, Wittenbrink, and Lodha, 1997).
Johnson and Sanderson in (Johnson and Sanderson, 2003) suggested that a formal theo-retical framework for visualizing uncertainty and error should be also developed:
We see the need to create a formal, theoretical error and uncertainty visual-ization framework and to investigate and explore new visual representations forcharacterizing error and uncertainty.
This framework will be fundamental to a better understanding of the data with dubious originor quality and as a result it will facilitate the decision making process. Nevertheless, it isworth examining available methods and techniques to visualize uncertainty from a perceptualand cognitive point of view.
In (Finger and Bisantz, 1997) the question of how to represent uncertainty is justifiedfrom two perspectives:
1. It is necessary to determine how different representations impact users and affect thedecision-making process and actions
2. It is also necessary to find the best way of displaying uncertain information, particularlywhen there are a large number of objects associated with uncertainty.
5.1.1 Uncertainty definitions
There are many definitions of uncertainty in the literature. Normally, uncertainty covers abroad range of concepts like inconsistency, doubtfully, reliability, inaccuracy or error (un-known or not quantified error). Thus, it is difficult to give a generally accepted definition ofuncertainty. From my point of view it depends on the context and the possible sources ofuncertainty. The following are some well known definitions of uncertainty in the informationvisualization research field.
In (Pang, Wittenbrink, and Lodha, 1997) uncertainty includes statistical variations orspread, errors and differences, minimum-maximum range values and noisy or missing data.The authors consider three types of uncertainty in their discussion:
1. statistical : distribution of the data or estimated mean and standard deviation (confi-dence interval)
2. error : an absolute valued error among estimates or differences between a known correctdatum and an estimate
3. range: an interval in which the data exists (and cannot be quantified into statisticaleither error definitions)
In ((MacEachren, Robinson, Hopper, Gardner, Murray, Gahegan, and Hetzler, 2005),citing other authors, Hunter and Goodchild, 1993) uncertainty is used meaning inaccuracywhich is not known objectively (otherwise it would be considered as error).
35
In (Foody and Atkinson, 2002, p. 26) it is simply defined as the ”quantitative statementabout the probability of error”, where inaccurate measurements, estimates or predictions areassociated with large uncertainty.
These definitions show that uncertainty has several concepts associated. Depending onwhich concept we refer to, there will be more than one way to quantify and represent uncer-tainty.
5.1.2 Sources of Uncertainty
One key point in the representation of uncertain information for a given application is iden-tifying the sources of uncertainty. A general model is described in (Pang, Wittenbrink, andLodha, 1997). The visualization pipeline shows three major blocks as possible sources ofuncertainty (see fig. 5.1):
1. Introduction of data uncertainty from models and measurements
2. Derived uncertainty from transformation processes
3. Visualization (representation) of uncertainty from the visualization process
Figure 5.1: “This visualization pipeline shows the introduction of data uncertainty from mod-els and measurements, derived uncertainty from transformation processes, and visualizationuncertainty from the visualization process itself” (Pang, Wittenbrink, and Lodha, 1997).Figure adapted from (Wittenbrink, Pang, and Lodha, 1996)
For example, one can identify the following possible sources of uncertainty in a genericinformation system:
• The sensors have limited resolution, its readings contain noise, their positions may beuncertain, sampling is sparse in time and space
36
• The process of converting the raw data into suitable input for numerical models mayinvolve operations like averaging, interpolation, sampling, etc.
• The numerical models are also approximations, further, discrete computation introduceserrors.
• The visualisation of the results introduces quantisation errors (data is interpolated,additional numerical integration may be used).
The level of uncertainty should be displayed in the visualization in order to help inter-preting the results. One should be sure that the operator or user has to do as few mentaltransformations as possible to understand the image (Jaa-Aro, 2006). This is where theoriesof perception and cognitive sciences can be of great support in the evaluation of techniquesused to depict uncertainty (see section 5.3).
5.2 Previous work on uncertainty visualization
Most of the previous work in uncertainty visualizations has been developed in the area ofGeographic Information systems, GIS (for example, see (Hunter and Goodchild, 1993) for asurvey of methods). In (Pang, Wittenbrink, and Lodha, 1997), the authors present a classifi-cation for uncertainty representation techniques. Seven categories are described: add glyphs,add geometry, modify geometry, modify attributes, animation, sonification and psycho-visual.In the following paragraph, examples of proposed techniques to display uncertainty are given(following the classification by (Griethe and Schumann, 2006)):
• Utilization of free graphical variables: colour, size, position, focus, clarity, fuzziness,saturation, transparency (e.g. fig. 5.7) and edge crispness. Figures 5.2 and 5.3 showexamples of the use of colour to display uncertainty.
• Additional objects: labels, images or glyphs. For example, Wittenbrink et al. in (Wit-tenbrink, Pang, and Lodha, 1996), propose the use of glyphs to represent uncertaintyin vector fields. Their approach is to include uncertainty in the magnitude, directionand length in glyphs (see figure 5.6 and 5.9). In (Pang, Wittenbrink, and Lodha, 1997)new ways of modifying glyphs in order to represent uncertainty are presented.
• Animation: the uncertainty is mapped to animation parameters such as speed or du-ration, motion blur, range or extent of motion.
• Interactive representation: e.g. uncertainty can be discovered by mouse interaction.An example can be found in (van der Wel, van der Gaag, and Gorte, 1998).
• Sonification and psycho-visual: incorporation of acoustics, changes in pitch, volume,rhythm, vibration, or flashing textual messages. See e.g. (Fisher, 1994).
Statistical properties can be plotted using a box-and-whisker plot (see figure 5.5). Oneinteresting extension of the use of box-plots over 2D distributions can be found in (Kao, Luo,
37
Dungan, and Pang, 2002) (see fig 5.8). The objective is to reduce the visual clutter when abox-plot is depicted over each grid location on the 2D map. The paper present two differentapproximations: using shape descriptors and parametric statistics.
Figure 5.2: Color change (saturation) indicat-ing uncertain information Figure 5.3: Colour indicates uncertain surface
Figure 5.4: Uncertain surface indicates uncer-tainty Figure 5.5: Box-and-whisker plot
5.3 Cognitive theories for the analysis of uncertainty visual-izations
Many have been the techniques proposed to represent uncertainty. However, almost none ofthem include a perceptual and cognitive evaluation after the development of the technique.An exception is the work by (Wittenbrink, Pang, and Lodha, 1996), which utilizes Tufte’s
38
Figure 5.6: Variety of glyphs representing an-gular and magnitude uncertainty, presented in(Wittenbrink, Pang, and Lodha, 1996).
Figure 5.7: Castle reconstruction. Uncertaintyabout the true architecture is encoded intotransparency (Griethe and Schumann, 2005).
theories to analyse the proposed method.
In (Tory and Moller, 2004), Tory and Moller highlight the need for greater application ofhuman factors research to visualization. This section provides a brief summary of possibleperceptual theories that can be used to evaluate, theoretically, uncertainty representationtechniques.Among all perceptual design theories I will consider Tufte’s, Bertin’s and Chambers’ for beinghighly cited authors in visualization.
1. Tufte (from (Tufte, 2001), (Tufte, 1997), (Tufte, 1994) and (Zuk and Carpendale,2006))Tufte defines two graphical principles that lead to good visualizations: graphical excel-lence and graphical integrity.
Graphical excellence (“give the viewer the greatest number of ideas in the shortesttime with the least ink in the smallest space”). Tufte has specified guidelines toencourage graphical clarity, precision and efficiency and achieve graphical excel-lence. Basically they include: avoid distorting what the data shows; encouragecomparison among the data; present a large amount of data in a small space; re-veal multiple levels of detail; closely integrate statistical and text descriptions intothe data.
Graphical integrity. The principles that ensure graphical integrity are: the repre-sentation of numbers should be directly proportional to the numerical quantitiesrepresented; clear and detailed labelling should be used to defeat ambiguity anddistortion; show data variations and not design variations; the number of infor-mation carrying dimensions should not exceed the data dimensions; show deflatedand standardized units in time-series displays of money; graphics must not presentdata out of context; convincing graphics should demonstrate cause and effect.
Tufte also advices that one should present the largest amount of data with the least
39
Figure 5.8: Extension of the 2D box-plot, by(Kao, Luo, Dungan, and Pang, 2002).
Figure 5.9: Uncertainty glyphs (from(Wittenbrink, Pang, and Lodha, 1996)),www.cse.ucsc.edu/research/slvg/uglyph.html
amount of ink (data-ink maximization principle). Moreover, the ratio amount of dataelements divided by the graphical area must be appropriate. If it is too low, the areashould be reduced (data density principle).
2. Bertin (from (Bertin, J., 1983) and (Zuk and Carpendale, 2006))In (Bertin, J., 1983), Bertin presents a classification of visual variables: planar dimen-sions (x,y) and visible marks over the plane (size, value, grain, color, orientation andshape).The analysis that Bertin suggests is based on the potential of each variable for: imme-diate perceptual group selection, natural perceptual ordering (not learned), perceptualgrouping characteristics, number of discernible elements that can be represented in theset (length) and ability for quantitative comparisons.A variable is called selective when it is perceived immediately without considering in-dividual marks sequentially. The length of a variable must be greatly reduced in orderto use it for selective processing.
3. Chambers (from (J. M. Chambers and W. S. Cleveland and B. Kleiner, and P. A.Tukey, 1983))In (J. M. Chambers and W. S. Cleveland and B. Kleiner, and P. A. Tukey, 1983, chap.8), the authors analyse how the brain, eye and picture interact in order to organize dis-plays to take advantage of the things that the eye and brain do best, so the potentiallymost important patterns are associated with the most easily perceived visual aspectsin the display.Chambers et al. conjecture that the eye is able to perceive: location along an axis moreeasily than other graphical aspects, e.g. size; straight lines more clearly than curves;simple patters more quickly than complex; large or dark objects, or clusters of objects,with greater impact than small, light or isolated ones; symmetry (especially bilateral
40
and circular symmetry). Moreover two more points are added regarding the mentalprocess: we can perceive several different aspects in one plot simply by switching atten-tion from one aspect to another and accumulated visual evidence is roughly additive.Following these ideas, Chambers et al. compile general techniques for plot construction.The guidelines are grouped in three categories: reducing clutter (the amount of unin-formative detail and clutter in a plot should be minimized); removing gross structure(increase the informativeness of a graph by removing structure from the data once wehave identify it) and labelling (effective use of labels).
Another possible set of perceptual theories that can be used in the evaluation are thosecompiled by Ware in (Ware, 2000) (his work includes many explanations from many cognitivescientist, e.g. Gestalt laws). For colour coding and dimensional evaluation, see for example(Keller, Gerjets, Scheiter, and Garsoffky, 2006).
5.4 Insights from user tests
The principles presented in section 5.3 can be complemented with formal studies regardingwhat impact the representation of uncertainty has on users or how the methods compareto each other. Leitner and Buttenfield (Leitner and Buttenfield, 2000) tested saturation,value and texture as means of depicting uncertainty over maps by looking at time, accuracyand confidence within a spatial decision support system. Their results regarding saturationsupport the studies by Schweizer and Goodchild in (Schweizer and Goodchild, 1992), wheresaturation is found to be not especially effective (a more detailed study regarding the use ofcolor to represent uncertainty appears in (Jiang B. and Ormeling, 1996)). On the other hand,they found that if finer texture or lighter value is chosen to depict certainty data the numberof correct responses for an easy decision is increased. They also found that response timesdecreased with the inclusion of certainty information for easy tasks, but no such difference wasobserved for more complex tasks. Four methods of displaying data quality were comparedby Evans (Evans, 1997): static separate maps, static integrated displays, animated non-controllable flicker maps and interactive toggle maps. The results show that users performedbest with the static integrated display and the flicker map. Andre and Cutler (Andre andCutler, 1998) studied the use of rings to display the uncertainty associated with a military-style entity position (the size of the ring indicates the degree of uncertainty). The resultsshow that, using this technique in a collision avoidance task, the performance was improved.
A significant contribution regarding uncertainty representation in information fusion ap-pears in (Bisantz, R. Finger, and Llinas, 1999) and (Finger and Bisantz, 1997). In two userexperiments, degraded and blended icons (figure 5.13) were used to portray uncertainty re-garding the identity of a radar contact as hostile or friendly. The results of these studies showthat people understand uncertainty conveyed in such a manner and that the use of degradedimages may be a viable alternative for representing uncertainty.
41
5.5 Examples: cognitive and theoretical analysis of uncertainvisualizations in information fusion
In this section uncertainty visualization techniques are analyzed based on the perceptualtheories introduced in section 5.3 and the user experiments presented in section 5.4. Thetechniques were selected from information fusion related applications2.
5.5.1 Probabilistic demand prediction for traffic flow decision support
Masalonis, Mulgund, Song, Wanke and Zobell in (Masalonis, Mulgund, Song, Wanke, andZobell, 2004) present visualization concepts and requirements for the display of uncertaintyin decision support systems for air traffic flow management (TFM). Interviews with expertoperators provide guidance for the development of a human machine interface prototypeshown in figure 5.10 and 5.11. Based on probabilistic alert levels (depicted using a three-color scheme: green/yellow/red3) operators must decide. The hypothetical display providesvarying levels of detail and meta-data related to the predicted alert when the mouse ispositioned on a particular cell (see figure 5.10 right side). This work, still in progress, wasnot evaluated, neither theoretically nor empirically by its authors (a theoretical evaluationwas recently presented in (Zuk and Carpendale, 2006) by Zuk and Carpendale).
Figure 5.10: Hypothetical probabilistic Center Monitor (Masalonis, Mulgund, Song, Wanke,and Zobell, 2004). Basic display on the left and sector summary via mouse rollover on theright side. Reproduced with permission, c©2004 MITRE Corporation.
Considering Tufte’s recommendation, the ratio of data elements divided by the graphicalarea is quite low (figure 5.11) and hence, more data could be displayed. Three colors rep-resent the probability of an alert over this display. Color, defined by Bertin as one of thevisible marks over the plane4, has long length so more alert levels could be shown varying it.Similarly, the numbers over the color background could be removed (figure 5.10). The use ofblack text numbers over high saturated color background might cause visual stress. Lowersaturated colors could reduce the luminance ratio of the black text on the used colors, thus
2The selected applications are not representative samples of different uncertainty visualization methods(unfortunately, few applications within information fusion display uncertainty).
3In order to avoid confusion with the overall display (figure 5.11), the scheme green/yellow/red was laterreplaced by magenta/purple/gray (in conversations with Craig Wanke).
4In Bertin’s system of visual variables, color refers only to hue.
42
avoiding visual stress. The display follows Chambers’ recommendations regarding symmetry,simplicity and straightness (figure 5.11). Moreover, the hover queries using the mouse allowcorrect labeling (figure 5.10), although the pop-up window might hide important changes inalert levels on the general display.
Figure 5.11: Center monitor display (Masalonis, Mulgund, Song, Wanke, and Zobell, 2004).Reproduced with permission, c©2004 MITRE Corporation.
5.5.2 Graphical formats to convey uncertainty in a decision making task
Finger and Bisantz (Finger and Bisantz, 1997) presented a novel work on the evaluation ofhow the presentation of the information may affect the decision-maker. Blended and degradedicons were used to represent uncertainty regarding the identity of a radar contact as hostileor friendly. The first part of the study showed that participants could sort, order and rankfive different sets of icons (see figure 5.12) conveying different levels of uncertainty (see figure5.13). In the second part of the study, three of the pairs of icons were used in an applicationin which participants should identify the status of contacts as friendly or hostile (see figure5.14).
Figure 5.12: Pairs of icons representing hostile or friendly objects (Bisantz, R. Finger, andLlinas, 1999). Reproduced with permission.
Blur is a selective (Bertin) visual variable. It gives fuzzy appearance to the icons and forinstance, can depict uncertainty. Not all the selected symbols follow Chambers recommenda-tions of symmetry, simplicity and straightness (e.g. dove and skull). Color (changes in value
43
Figure 5.13: The blurred icons represent the probability that an object is hostile or friendly,from probability 1 of being friendly to probability 1 of being hostile (Bisantz, R. Finger, andLlinas, 1999). Reproduced with permission.
Figure 5.14: Graphical display used in the dynamic decision making experiment (Finger andBisantz, 1997). Reproduced with permission.
and saturation) was found not to be especially effective for displaying uncertainty in userevaluations ((Schweizer and Goodchild, 1992; Leitner and Buttenfield, 2000)). Moreover, itshould be noted that color is defined by Bertin as a variable without implicit order (notlearned) and hence not a good choice when an ordering task must be performed. One sug-gestion is the use of transparency to encode uncertainty. Transparency is considered a colorand value hybrid (redundant encoding) and it facilitates perception (Zuk and Carpendale,2006).
Perceiving icons with very large magnitude of uncertainty (extremely blurred icons) be-comes difficult and its interpretation might be ambiguous. In this case, the integration of textfor the symbols might facilitate perception (Tufte’s principle of close integration of text andgraphics). The data density ratio (Tufte) of the application is quite low (e.g. no backgroundinformation is presented indicating the position of the object) so text labeling and iterativequeries on the application might facilitate comprehension. Following Tufte’s integrity prin-ciple, “do not show data out of context”, a low-saturated color map as a background mighthelp to solve complex tasks (figure 5.14).
5.5.3 DSS prototype display: a critical decision analysis of aspects of navalanti-air warfare
Freeman and Cohen (Freeman and Cohen, 1998) evaluated the effects on tactical decisionmaking of a prototype decision support system (DSS) display developed by the Space andNaval Warfare Systems Center (see figure 5.15). The strategic use of graphics was intended
44
to support rapid decision making based on pattern recognition (e.g. weapons range ringsand task management graph bars). The DSS display improved the ability to think criticallyunder uncertainty. In this case, color coded flags and annotations pop-up on the geographicaldisplay (geoplot) to direct the attention of the officers. The geoplot display allows the officerto concentrate on a specific element, while the overview frame provides a general view of thewhole situation (Roy, Breton, and Paradis, 2001). Threat values assigned to individual tracksare presented to the decision maker as a sorted list from the most threatening, left side, tothe least, right side (see bottom figure 5.15).
Figure 5.15: DSS display features (Freeman and Cohen, 1998). Reproduced with permission.
Tufte’s data density measure appears to be quite high considering the whole display. Mul-tiple displays (physical and functional) account for different granularity on the information,maintaining both background awareness (big picture) and foreground awareness (details andparticular goals) (Tufte’s guideline of excellence revealing data on several levels of detail).Large regions of the display use low saturated colors, e.g gray, avoiding visual stress. In thisapplication, the threat values are sorted and presented to the user as part of a ordered listavoiding ambiguity (highest priority on the left and lowest on the right). Thus, the uncertainthreat values are sorted and easily perceived by the user. However, as situations evolve, thereorganization of the buttons might generate information overload when the decision makeris under stress. Related attention and scanning problems can be analyzed using theories byWickens (Wickens and Hollands, 1999). Following Tufte’s integrity principle of context, theelements are displayed over a geographical map maintaining a general view of the decision-maker’s problem space. Considering the recommendations of Chambers regarding labeling,the buttons display critical identification and kinematic information that allow monitoringwithout any additional interaction with the system.
5.6 Conclusions and future work
Visualization of uncertainty for decision making is an interdisciplinary problem (MacEachren,Robinson, Hopper, Gardner, Murray, Gahegan, and Hetzler, 2005) and many authors have
45
pointed out that further research regarding the representation of uncertain information shouldbe done (Griethe and Schumann, 2005; Pang, Wittenbrink, and Lodha, 1997). Basic questionsremain largely unanswered: whether or not displaying uncertainty helps users, changes theirway of thinking and problem solving or helps to make better decisions. Few studies havebeen done to determine what impact (if any) the display of uncertainty has on users or howvarious methods compare to each other (Harrower, 2003). Neither within information fusionthe representation of uncertainty should be overlooked due to the high degree of uncertaintythat many times is associated with the information handled by a decision maker.
In many cases, the uncertainty representation techniques described in section 5.2 can-not be applied in three dimensional (3D) representations or have not been extended to 3D.According to Johnson (Johnson and Sanderson, 2003), researchers have ignored the depic-tion of uncertainty in 3D representations with few exceptions. Future work should developnew uncertainty visualization techniques and should extend the existing studies in two di-mensions to three. Furthermore, two general research challenges in uncertainty visualizationthat are highly relevant to information fusion are: (1) the development of representationand evaluation methods for depicting multiple forms of uncertainty in the same display and(2) the development of methods and tools for interacting with uncertainty representations(MacEachren, Robinson, Hopper, Gardner, Murray, Gahegan, and Hetzler, 2005).
The evaluation of uncertainty visualization techniques should include theoretical cogni-tive/perceptual analysis and usability tests with actual users. In this report, a small setof cognitive and perceptual theories as well as results from other researchers’ user experi-ments have been presented. They can be used in theoretical evaluations of existing and newtechniques for displaying uncertain information, providing insights into their weakness andstrengths. Posterior user experiments should clarify, for example, whether or not the inclu-sion of uncertainty representations for a given task changes how users evaluate the data anddecide, makes users more cautious or causes them to try to obtain outside information tosupport their decisions.
46
Bibliography
Adams, M., Tenney, Y., Pew, R., 1995. Situation awareness and the cognitive managementof complex systems. Human factors 37(1), 85–104.
Andler, S., 2005. Information fusion from databases, sensors and simulations. Tech. Rep.HS-IKI-TR-06-004, University of Skovde.
Andler, S. F., Niklasson, L., Olsson, B., Persson, A., de Vin, L. J., Wangler, B., Ziemke, T.,Planstedt, T., 2005. Information fusion from databases, sensors and simulations: A collab-orative research program. In: 29th Annual IEEE/NASA Software Engineering Workshop.Vol. 00. IEEE Computer Society, Los Alamitos, CA, USA, pp. 234–244.
Andre, A., Cutler, H., 1998. Displaying uncertainty in advanced navigation systems. In:Human Factors and Ergonomics Society 42nd Annual Meeting. Chicago, IL, USA, pp.31–35.
Barnes, M. J., February 2003. The human dimension of battlespace visualization: Researchand design issues. Tech. Rep. ARL-TR-2885, Army Research Laboratory.
Bederson, B., Meyer, J., 1998. Implementing a zooming user interface: Experience buildingPad++. Software: Practice and Experience 28 (10), 1101–1135.
Bedworth, M., O’Brien, J., 2000. The Omnibus Model: A New Model of Data Fusion. In:Aerospace and Electronics Systems Magazine. Vol. 15(4). pp. 30–36.
Bertin, J., 1983. Semiology of Graphics. University of Wisconsin Press, Madison, WI, (trans.W. Berg).
Bisantz, A. M., R. Finger, Y. S., Llinas, J., 1999. Human performance and data fusion baseddecision aids. In: Proceedings of the FUSION ’99 Conference. Vol. 2. pp. 918–925.
Blasch, E. P., Plano, S., Jul. 2002. JDL level 5 fusion model: user refinement issues andapplications in group tracking. In: Kadar, I. (Ed.), Proc. SPIE, Signal Processing, SensorFusion, and Target Recognition XI. pp. 270–279.
Blasch, E. P., Plano, S., 2003. Level 5: user refinement to aid the fusion process. Multi-sensor, Multisource Information Fusion: Architectures, Algorithms, and Applications 20035099 (1), 288–297.
Blasch, E. P., Plano, S., 2004. Cognitive fusion analysis based on context. In: Dasarathy,B. V. (Ed.), Multisensor, Multisource Information Fusion: Architectures, Algorithms, andApplications 2003. Vol. 5434. SPIE.
Boyd, J., 1987. A discourse on winning and loosing. Tech. Rep. Air University Library doc-ument No. M-U 43947, Maxwell Air Force Base, Al.
47
Brehmer, B., 2005. The dynamic OODA loop: amalgamating Boyd’s OODA loop and theCybernetic approach to command and control. In: Proc. of 10th international commandand control research and technology symposium.
Card, S., Mackinlay, J., Shneiderman, B., 1999. Readings in Information Visualization. Mor-gan Kaufmann, San Francisco, CA.
Card, S., Robertson, G., York, W., April 13-18 1996. The WebBook and the WebForager: AnInformation workspace for the World-Wide Web. In: ACM Conference on Human Factorsin Computing Systems (CHI’96). Vancouver, British Columbia, Canada, pp. 111–117.
Cartwright, W., Crampton, J., Gartner, G., Miller, S., Mitchell, K., Siekierska, E., Wood, J.,January 2001. Geospatial information visualization user interface issues. Cartography andGeographic Information Science 28 (1), 45–60.
Cedilnik, A., Rheingans, P., 2000. Procedural annotation of uncertain information. In: 11thIEEE Visualization 2000 (VIS’00). IEEE Computer Society Press, Los Alamitos, CA, USA,pp. 77–83.
Chen, C., 2004. Information Visualization - Beyond the Horizon. Springer, London, Berlin,Heidelberg.
Chen, C., Czerwinski, M., 2000. Empirical evaluation of information visualizations: an intro-duction. International Journal of Human-Computer Studies 53 (5), 631–635.
Dasarathy, B., 2000. Industrial applications of multi-sensor multi-source information fusion.In: Proc. of the IEEE int. Conference on Industrial Technology. Vol. 2. pp. 5–11.
Dasarathy, B., 2001. Information fusion – what, where, why, when, and how? InformationFusion 2, 75–76.
Eick, S., Steffen, J., Sumner, E., 1992. Seesoft– a tool for visualizing software. IEEE Trans-actions on Software Engineering 18, 957–968.
Endsley, M., 1995. Toward a theory of situation awareness in dynamic systems. HumanFactors Journal 37(1), 32–64.
Endsley, M. R., Garland, D. J. (Eds.), 2000. Situation Awareness Analysis and Measurement.Lawrence Erlbaum Associates, Publishers, Mahwah, New Jersey.
Esteban, J., Starr, A., Willetts, R., Hannah, P., Bryanston-Cross, P., 2005. A review of datafusion models and architectures: towards engineering guidelines. Neural Comput. Appl.14 (4), 273–281.
Evans, B., 1997. Dynamic display of spatial data-reliability: does it benefit the map user?Computenrs & Geosciences 23 (4), 409–422.
Finger, R., Bisantz, A., 1997. Utilizing graphical formats to convey uncertainty in a decisionmaking task. Theoretical Issues in Ergonomics Science 3 (1), 1–25.
48
Fisher, P., 1994. On animation and sound for the visualization of uncertain spatial informa-tion. Visualization in Geographical Information Systems, 181–185.
Foody, G. M., Atkinson, P. (Eds.), 2002. Uncertainty in Remote Sensing and GIS. Wiley,Sussex, England.
Freeman, J. T., Cohen, M. S., 9 June 1998. A critical decision analysis of aspects of navalanti-air warfare. Tech. Rep. 98-2, Cognitive Technologies, Inc.
Furnas, G., 1986. Generalized fisheye views. In: Proceedings CHI’86. pp. 16–23.
Gershon, N., 1998. Visualization of an imperfect world. IEEE Comput. Graph. Appl. 18 (4),43–45.
Gershon, N., Eick, S., Card, S., 1998. Information visualization. Interactions 5 (2), 9–15.
Griethe, H., Schumann, H., 2005. Visualizing uncertainty for improved decision making. In:Proceedings of the 4th International Conference on Perspectives in Business InformaticsResearch. University of Skovde, Sweden.
Griethe, H., Schumann, H., 2006. The visualization of uncertain data: Methods and problems.In: Proceedings SimVis’06, Magdeburg, Germany.
Hall, D., Llinas, J. (Eds.), 2000. Handbook of multisensor data fusion. CRC Press, BocaRaton.
Hall, D., McMullen, S. A. H. (Eds.), 2004. Mathematical Techniques in Multisensor DataFusion, 2nd Edition. Artech House, Inc., Norwood, MA.
Hall, M. J., Hall, S. A., Tate, T., 2000. Removing the HCI Bottleneck: How the HumanComputer Interaction (HCI) affects the performance of Data Fusion System. In: 2000 MSSNational Sysmposium on Sensor and Data Fusion. pp. 89–104.
Hansen, C., Johnson, C. (Eds.), 2005. The Visualization Handbook. Elsevier Inc., USA.
Harris, C. J., Bailey, A., Dodd, 1998. Multi-sensor data fusion in defence and aerospace. TheAeronautical Journal 102 (1015), 229–244.
Harrower, M., 2003. Representing uncertainty: Does it help people make better decisions? In:UCGIS Workshop: Geospatial Visualization and Knowledge Discovery Workshop, NationalConference Center. Landsdowne, VA., (invited white paper).
Hauser, H., 2005. Toward new grounds in visualization. SIGGRAPH Comput. Graph. 39 (2),5–8.
Hibbard, B., 1999. Top ten visualization problems. SIGGRAPH Comput. Graph. 33 (2),21–22.
Hunter, G. J., Goodchild, M. F., 1993. Managing uncertainty in spatial databases: Puttingtheory into practice. URISA J. 5 (2), 55–62.
49
Hutchins, E., 1995. Cognition in the wild. MIT Press, Cambridge, MA.
J. M. Chambers and W. S. Cleveland and B. Kleiner, and P. A. Tukey, 1983. GraphicalMethods for Data Analysis. Chapman and Hall, New York.
Jaa-Aro, K., 2006. Lecture notes: Visualization of uncertainty, KTH. Stockholm.
Jiang B., B. A., Ormeling, F. J., 1996. Some perceptual aspects of colouring uncertainty.Advances in GIS Research II, 477–490.
Johnson, B., Shneiderman, B., October 1991. Tree-maps: A space filling approach to the visu-alization of hierarchical information structures. In: Proceedings of the IEEE Visualization91. pp. 284–291.
Johnson, C., 2004. Top scientific visualization research problems. IEEE Computer Graphicsand Applications 24 (3), 13–17.
Johnson, C. R., Sanderson, A. R., 2003. A next step: Visualizing errors and uncertainty.IEEE Computer Graphics and Applications 23 (5), 6–10.
Kao, D., Luo, A., Dungan, J. L., Pang, A., 2002. Visualizing spatially varying distribu-tion data. Proceedings of the Sixth International Conference on Information Visualisation(IV’02).
Keller, T., Gerjets, P., Scheiter, K., Garsoffky, B., 2006. Information visualizations for knowl-edge acquisition: The impact of dimensionality and color coding. Computers in HumanBehavior 22 (1), 43–65.
Kokar, M. M., Tomasik, J. A., Weyman, J., 2004. Formalizing classes of information fusionsystems. Information Fusion 5 (3), 189–202.
Kolojejchick, J., Roth, S., Lucas, P., 1997. Information appliances and tools: Simplicity andpower trade-offs in the visage exploration environment. Computer Graphics and Applica-tions, 32–41.
Larkin, J., Simon, H., 1987. Why a diagram is (sometimes) worth ten thousand words.Cognitive Science 11, 65–99.
Leitner, M., Buttenfield, B. P., 2000. Guidelines for the display of attribute certainty. Car-tography and Geographic Information Science 27 (1), 3–14.
Llinas, J., Bowman, C., Rogova, G., Steinberg, A., Waltz, E., White, F., 2004. Revisiting theJDL Data Fusion Model II. In: Proc. 7th Int. Information Fusion Conference. Stockholm,Sweden.
MacEachren, A., 1995. How Maps Work, Representation, Visualization and Design. TheGuildford Press, New York.
50
MacEachren, A. M., Robinson, A., Hopper, S., Gardner, S., Murray, R., Gahegan, M.,Hetzler, E., 2005. Visualizing geospatial information uncertainty: What we know and whatwe need to know. Cartography and Geographic Information Science 32 (3), 139–160.
Mackinlay, J. D., 2000. Opportunities for information visualization. IEEE Computer Graphicsand Applications 20 (1), 22–23.
Masalonis, A., Mulgund, S., Song, L., Wanke, C., Zobell, S., 2004. Using probabilistic demandpredictions for traffic flow management decision support. In: AIAA Guidance, Navigation,and Control Conference and Exhibit. Providence, Rhode Island.
Matheus, C. J., Kokar, M. M., Baclawski, K., 2003. A core ontology for situation awareness.In Proceedings of Sixth International Conference on Information Fusion, Caims Australia,545–552General article about situation awareness. Ontology.
McDaniel, D., 2001. An information fusion framework for data integration. In: ThirteenthAnnual Software Technology Conference ”2001 Software Odyssey: Controlling Cost, Sched-ule, and Quality” 29 April - 3 May 2001 - Salt Lake City, Utah.
Meadow, C. T., Yuan, W., 1997. Measuring the impact of information: Defining the concepts.Information Processing & Management 33 (6), 697–714.
Morse, E., Lewis, M., Olsen, K., 2000. Evaluating visualizations: using a taxonomic guide.International Journal of Human-Computer Studies 53 (5), 637–662.
Munzner, T., 2002. Guest’s editors introduction: Information visualization. Computer Graph-ics and Applications, IEEE 22 (1), 20–21.
Pang, A., Wittenbrink, C., Lodha, S., 1997. Approaches to uncertainty visualization. TheVisual Computer 13 (8), 370–390.
Rasmussen, J., 1987. Skills, rules, and knowledge; signals, signs, and symbols, and otherdistinctions in human performance models. System design for human interaction, 291–300.
Rhyne, T.-M., 2003. Does the difference between information and scientific visualization reallymatter? Computer Graphics and Applications, IEEE 23 (3), 6–8.
Rhyne, T.-M., Hibbard, W. L., Johnson, C., Chen, C., Eick, S., 2004. Panel 1: Can wedetermine the top unresolved problems of visualization? In: IEEE Visualization. pp. 563–566.
Riveiro, M., (to appear) July 2007. Evaluation of uncertainty visualization techniques forinformation fusion. In: 10th international Conference on Information Fusion. Quebec,Canada.
Robertson, G., J., M., Card, S., April 28–May 2 1991. Cone trees: Animated 3D visualizationsof hierarchical information. In: Proceedings of CHI’91. New Orleans, LA, pp. 189–194.
51
Roy, J., Breton, R., Paradis, S., 2001. Human-computer interface for the study of informationfusion concepts in situation analysis and command decision support systems. In: Kadar, I.(Ed.), Signal Processing, Sensor Fusion, and Target Recognition X. Vol. 4380. SPIE, pp.361–372.
Schweizer, D., Goodchild, M., 1992. Data quality and choropleth maps: An experiment withthe use of color. In: Proceedings, GIS/LIS. Vol. 2. pp. 686–699.
Shedroff, N., 2001. An overview of understainding. In: Wurman, R. S., Sume, D., Leifer, L.(Eds.), Information Anxiety 2. Que, Indianapolis, pp. 27–29.
Spence, R., 2001. Information Visualization. Addison-Wesley, Harlow.
Spence, R., Apperley, M., 1982. Data base navigation: An office environment for the profes-sional. Behaviour and Information Technology 1 (1), 43–54.
Steinberg, A. N., Bowman, C. L., 2004. Rethinking the JDL data fusion levels. In: Proc.2004 MSS National Symposium on Sensor and Data Fusion.
Steinberg, A. N., Bowman, C. L., White, F. E., 1998-1999. Revisions to the JDL data fusionmodel. In: Joint NATO/IRIS Conference Proceedings, Quebec, October, 1998 and SensorFusion: Architectures, Algorithms, and Applications III. Vol. 3719. pp. 430–441.
Thomson, J., Hetzler, E., MacEachren, A., Gahegan, M., Pavel, M., 2005. A typology forvisualizing uncertainty. In: Erbacher, R. F., Roberts, J. C., Grohn, M. T., Borner, K.(Eds.), Visualization and Data Analysis 2005. Vol. 5669. SPIE, pp. 146–157.
Tory, M., Moller, T., 2002. A model-based visualization taxonomy. Tech. Rep. CMPT-TR2002-06, Computing Science Dept., Simon Fraser Univ.
Tory, M., Moller, T., 2004. Human factors in visualization research. IEEE Transactions onVisualization and Computer Graphics 10 (1), 72 – 84.
Tufte, E. R., 1994. Envisioning Information. Graphics Press, Cheshire, Connecticut.
Tufte, E. R., 1997. Visual explanations. Images and Quantities, Evidence and Narrative.Graphics Press, Cheshire, Connecticut.
Tufte, E. R., 2001. The Visual Display of Quantitative Information, 2nd Edition. GraphicsPress, Cheshire, CT.
van Dam, A., Forsberg, A. S., Laidlaw, D. H., LaViola, J. J., Simpson, R. M., 2000. ImmersiveVR for scientific visualization: A progress report. IEEE Comput. Graph. Appl. 20 (6), 26–52.
van der Wel, F. J. M., van der Gaag, L. C., Gorte, B. G. H., 1998. Visual exploration ofuncertainty in remote-sensing classification. Computers & Geosciences 24 (4), 335–343.
Vessey, I., 1991. Cognitive fit: a theory-based analysis of the graphs versus tables literature.Decision Sciences 22, 219–240.
52
Wallenius, K., 2004. Support for situation awareness in command and control. In: The 7thInternational Conference on Information Fusion. Stockholm, Sweden.
Waltz, E., Llinas, J. (Eds.), 1990. Multisensor Data Fusion. Artech House, Inc., Norwood,MA.
Ware, C., 2000. Information Visualization: Perception for Design. Morgan Kaufmann, SanFrancisco, CA.
Wickens, C. D., Hollands, J. G., 1999. Engineering Psychology and Human Performance (3rdEdition). Prentice Hall.
Wittenbrink, C. M., Pang, A. T., Lodha, S. K., 1996. Glyphs for visualizing uncertainty invector fields. IEEE Transactions on Visualization and Computer Graphics 2 (3), 266–279.
Zuk, T., Carpendale, S., 2006. Theoretical analysis of uncertainty visualizations. In: Er-bacher, R. F., Roberts, J. C., Grohn, M. T., Borne, K. (Eds.), Visualization and DataAnalysis, Proc. of SPIE-IS&T Electronic Imaging. Vol. 6060.
53