Evaluating Climate Visualization: An Information Visualization Approach

Jimmy Johansson∗, Tina-Simone Schmid Neset† and Bjorn-Ola Linner†∗C-Research, Linkoping University, Sweden

†Centre for Climate Science and Policy Research, Linkoping University, Swedenjimmy.johansson@liu.se, tina.schmid.neset@liu.se, bjorn-ola.linner@liu.se

AbstractTo meet the growing demand of communicating climate

science and policy research, the interdisciplinary field ofclimate visualization has increasingly extended its tradi-tional use of 2D representations and techniques from thefield of scientific visualization to include information visu-alization for the creation of highly interactive tools for bothspatial and abstract data.

This paper provides an initial discussion on the needand design of evaluations for climate visualization. Wereport on previous experiences and identify how evalua-tion methods commonly used in information visualizationcan be used in climate visualization to increase our under-standing of visualization techniques and tools.

Keywords—Evaluation, climate visualization, informationvisualization

1 IntroductionClimate change has been broadly discussed in a large

variety of media over the last years. There is thus an ev-ident need for communicating scientific data to all ages,social, cultural and professional groups. The most recentconflicts of trust towards data referred to in the reports ofthe International Panel for Climate Change (IPCC) [9] hasfurther increased the need for clear and uncompromisedcommunication of uncertainties, global implications andeffects on a regional to global scale.

Visualization is a powerful tool, as it is persuasive andcan provide intuitive understanding of complex data. Cli-mate visualization refers to tools for analysis and commu-nication of climate change issues and research results, andprovides decision making support for planners and policymakers [15], see figure 1. The field of climate visualiza-tion has a tradition of representing data using both static2D representations and visualization techniques originat-ing from scientific visualization and engineering [17]. Thecomplexity of climate research and its inherent need forcommunication within scientific fields, to decision makersas well as to a broader audience, however, benefits frominteractive information visualization techniques [11].

Analysis Communication

Decision Making

Figure 1: The three interlinked research areas of climatevisualization.

Due to the power of visualization, all choices relatedto data selection, transformation and representation needto be carefully reflected upon and their effectiveness andefficiency [8] evaluated. This calls for a close collabora-tion between climate, visualization and human-computerinteraction researchers to define key communication chal-lenges and to design appropriate evaluation tools for dif-ferent groups of audiences.

The aim of this paper is to initiate a discussion on how toperform scientific evaluations of climate visualization toolsoriginating from the field of information visualization. Weargue the need for advanced interactive information visu-alization tools for climate visualization and the need forappropriate evaluation techniques. A number of surveyevaluations have so far provided a general understandingof the requirements and aims of our audiences and usersbut more detailed evaluations are necessary to get a morein-depth understanding of the effectiveness and efficiencyof climate visualization. This paper will discuss challengesthat arise with the use of visual representations of climatedata sets and how evaluation methods applied within the

Figure 2: An inflatable dome, with space for an audienceof 25 visitors. In this immersive environment, the World-View project [19] has presented a programme on climatevisualization at several venues during 2009.

field of information visualization can support the develop-ment and interaction between climate and visualization re-searchers. We will exemplify evaluations of visual presen-tations created for immersive environments (figure 2) for abroad audience where the focus is on the cause, effect andfuture alternatives related to climate changes, as well as foranalysis tools (figure 3) for a specific user group.

2 BackgroundClimate visualization as a general concept of interdis-

ciplinary research over the fields of visualization and cli-mate research ‘refers to interactive research platforms,which use computer graphics to create visual images ofcauses and effects of climate change as well as mitiga-tion and adaptation options’ [15]. Key challenges are thecomplexity of societal and environmental systems and thevast amount of heterogeneous data on spatial and temporalscales.

Within the climate system and impact research com-munities standard visualization techniques have been usedfor communicating results between scientists themselvesas well as to a broader public. The rapid development ofcomputational power and computer graphics has, however,created a broad spectrum of opportunities to visually rep-resent vast amounts of data on climate processes and itseffects as well as the associated complexities and uncer-tainties interactively, using consumer computer hardware.

Further, information visualization techniques enable theinteractive analysis of interrelations of abstract data to con-vey synergies within social and environmental systems. Anumber of attempts have been made to use information vi-sualization techniques for climate visualization, see for ex-

Figure 3: Example of a web-based application—the Inter-active Climate Visualization Environment (ICE [7]).

ample [17, 16, 11]. Using information visualization tech-niques within climate visualization is today receiving in-creasing attention but discussions on how to perform sci-entific evaluations of these tools are not as prominent.

The information visualization community has todayrecognized the importance of usability and evaluation isone of the key research challenges [1, 2]. Evaluation canbe qualitative or quantitative and be performed at differ-ent levels, ranging from evaluating small, individual fea-tures to entire systems. For a review of currently availablemethods and further discussions, see [4, 20, 18]. In [18]Plaisant derived four different categories of evaluation ob-tained from a survey of literature of about fifty differentuser studies conducted within the area of information vi-sualization: (1) controlled experiments comparing designelements, (2) (qualitative) usability evaluation of a tool,(3) controlled experiments comparing several tools, and(4) case studies of tools in realistic settings. Among thesecategories, case studies of tools in realistic settings (4) isthe one least occurring in literature. The most frequentlyreported evaluations concern controlled experiments andcomparisons between different tools (1 and 3). Controlledexperiments are typically used to test the effectiveness orefficiency [8] of an isolated feature. For example, in [10]common patterns in parallel coordinates were examinedwhen distorted by different levels of noise in order to de-termine thresholds for visual identification. Comparisonsof several tools can be found in, for example, [12, 3].

To meet the specific requirements of climate visualiza-tion tools originating from information visualization, eval-uation techniques already used within this community canbe applied and adopted.

3 Climate VisualizationOne of the main challenges within climate visualiza-

tion is the creation of nuanced representations of climatechange related issues. Within climate communication sev-

eral challenges have been identified [14], and a responsibleapproach has to be taken by climate scientists regardingthe selection of data, their visual representations and thecreation of narratives.

Information on climate change is of a particularly com-plex nature. Studies of causes, effects and means to takeaction include several interlinked areas of studies. Cli-mate science includes areas such as atmospheric chem-istry, oceanography, biology, palaeoclimatology, physics,biology, glaciology as well as natural and human geogra-phy, political science, economics and sociology. One ofthe great challenges is how to capture the numerous intri-cate linkages between these areas in order to understandthe consequences of inaction as well as of the various ac-tion alternatives to mitigate greenhouse gas emissions oradapt to a changing climate. In addition, information onclimate change involves large data sets and complex mod-elling. The character of climate change information posesa particularly wicked problem to convey to public and de-cision makers.

The intricate interaction between humans and natureis often conveyed in narratives. The storylines of cli-mate change narratives typically follow a temporal struc-ture (what has happened, what is happening and how willthis develop in the future). Narratives in climate visual-ization often imply the specific storyline of a sequence ofvisuals and the link between areas of content. It is how-ever equally important to relate the narrative to the selec-tion of presented scenarios, alternative futures and how afuture society will adapt to climate change. Visions ofalternative futures and future societies are frequently em-ployed in climate science and multilateral negotiations, forinstance, the storylines developed by the IPCC. Indica-tors that are represented by visualization range from eco-nomic development, greenhouse gas emission and energyuse to regional scenario representations including effectsof water scarcity and sea level rise. The selection of thesenarratives are however strongly related to the conceptionof (un)sustainable futures. Whichever future scenario wechoose to represent, this implies a choice of a brighter ordarker future. As climate debate gains a massive medialand political breakthrough, the risk of exaggerating worst-case scenarios has been substantial, creating a general anx-iety over the unavoidable doomsday [5, 6].

The climate debate has been characterized by rhetori-cal apocalypse, i.e. warnings of catastrophic consequencesif we do not change our course of action. However, suchrhetorical narratives run the risk of being contra produc-tive, making the public, in particular young people, dis-heartened and in the end passive. Further, if the worst-case scenarios do not come true, the history of environmen-tal debate reveals that opponents of stricter environmental

policies use the failed forecasts or exaggerated scenarios todiscredit the environmental organizations and policy mak-ing [13]. For each selected data set and narrative, we thushave to thoroughly evaluate their impact, both in terms ofcontent and representation, on the audience.

3.1 The WorldView ProjectSince 2008, the WorldView project [19] is the ba-

sis of a cooperation between climate and visualization re-searchers at Linkoping University, the Norrkoping Visu-alization Centre and the Swedish Meteorological and Hy-drological Institute. Within the project we have developeda number of narratives and data sets focused on the causeand effect of climate change as well as on a number of ac-tion alternatives aimed for approximately 30 minutes pre-sentations in an immersive dome environment (illustratedin figure 2). During 2009 several presentations were heldfor mostly heterogeneous audiences where survey evalua-tions were conducted [14], see section 3.2.

The narrative of the WorldView presentation includedboth effects of climate change to regional change in tem-perature, changes in arctic sea ice as well as regional andglobal sea level rise. Principles for responsibility and ef-fort sharing related to green house gas emissions were pre-sented in form of volume bar charts for each country (asillustrated in figure 4) and discussed with the audience.Both categorical and numerical data sets were producedby the Swedish Meteorological and Hydrological Instituteand the Centre for Climate Science and Policy Research.The general aim of the project is to provide a nuanced pic-ture based on scientifically validated data to communicatethe complexity of climate change as well as to open up tojoint discussions with the audience. Presentations were al-ways held by climate experts to facilitate interaction anddialogue with the audience.

3.2 Previously Performed EvaluationsOur experience of evaluating climate visualization in

the WorldView project has so far been limited to surveys.In six different studies, some ongoing, we have focused onfive specific areas: data, representation, narrative, narratorand presentation environments.

All survey evaluations have been performed as paper orweb-based surveys and included grading scales as well asfree-text answers. Evaluations were conducted with het-erogeneous groups and most respondents indicated that thecontent and form of the presentations were relevant. Mostsurvey questions were related to the understanding of datasets and on how these are related to increased engagement,own lifestyle choices, general knowledge and understand-ing for climate related issues and the audience’s profes-sional interest. When the participants were asked to iden-

tify which audience would have the largest gaining of thiskind of climate visualization presentation, most pointed to-wards education [14].

Questions that were related specifically to the visualrepresentation were posed in three areas: novelty, aesthet-ics and understanding of the visual representations. Ques-tions related to these areas were posed as positive claimse.g. ‘it was clear that emission levels were symbolized bythe height of the bars in the bar chart, not the volume’ andthe respondents answered using a rating scale ranging fromdisagreement to agreement.

We received positive response to the aesthetics and nov-elty of the climate visualization presentation as well as inrespect to the immersive environment as ‘a new way of pre-senting this data’. The aesthetics of the visuals was alsoby most respondents considered as supporting their gen-eral understanding of the data. Further, the response tothe visual representation as volume bar chart (see figure 4)was positive and the colouring was by most respondentsconsidered as helpful when interpreting the data.

These surveys are, however, limited in detail and in-depth communication of the responses due to the fact thatthe questions were directly posed after a brief presentationand the format of using rating scale answers. Further, gen-eralized surveys for heterogeneous audiences pose a chal-lenge and may cause mis-interpretation of both questionsand responses. While even brief comments on open-endedquestions generate an insight into the argumentation andmotivation of the respondent, these were not sufficient inorder to reflect upon the general understanding and expe-rience of the visual representation. The performed eval-uations provide us thus with a general knowledge on theaudience’s background and general rating of the presen-tation, but lack a deeper insight both in terms of climatecommunication as well as of the visual representations.

4 Future DirectionsThe understanding and perception of visual representa-

tions needs to be evaluated beyond the realms of a gen-eral audience survey. In particular since climate visual-ization covers three distinct but overlapping areas: sciencecommunication, data analysis and decision making support(figure 1), evaluation of visualization tools and visual rep-resentations need to focus on features relevant to one ormore of these three dimensions. To capture in depth knowl-edge on the effectiveness and efficiency of climate visual-ization, we have identified the below listed areas, followingthe categorization by Plaisant [18].

Controlled experiments comparing elements. This cat-egory refers to testing the usability of small, individ-ual features. Specific areas for climate visualizationinclude:

(a)

(b)

(c)

Figure 4: Illustrations of climate visualization, using ex-truded regions with different colour maps to convey infor-mation on a regional and global scale using example datasets [7].

i) the evaluation of colour maps both directed towardsclimate researchers from different fields and their un-derstanding and connotation of colour choice as wellas for a laymen audience. For example, what doesa colour choice deviating from a white-red scale fortemperature increase imply, or how does the colour-ing of a high-emission country in bright red influenceyour perception of the underlying data?

ii) evaluation of how effective and efficient differentgraphical primitives (points, lines, polygons) conveysignificant structures in data. How are, for example,multivariate parameters best represented on a local orglobal scale? Are multiple simple 2D glyphs a betterchoice than fewer complex 3D ones?

iii) assessment of interaction techniques and theirefficiency in revealing information. For example,how are parameters selected and how do pre-definedchoices of adaptation and mitigation parameters influ-ence the users’ perception of the inter-linkages in thedata set.

iv) evaluation of how uncertainties and variations inresearch data are conveyed by the visual representa-tions. How do, for instance, parallel Earth represen-tations opposed to a single representation effect theunderstanding of future climate scenarios for green-house gas emissions and temperature changes?

v) mapping of data values to opacity using linear andnon-linear transfer functions. For data sets containinglarge ranges of values, a linear mapping might not al-ways be sufficient in order to represent all differenceswithin the data. Non-linear mappings allow the userto put emphasis on selected value ranges but at thecost of distorting the underlying structures in the cli-mate data.

vi) effectiveness and efficiency of advanced illumina-tion models; e.g. how does the gaining of attractiveadvanced illumination models reflect upon engage-ment and interest for the presentation itself opposedto the accuracy and reliability of the presented climatedata.

(Qualitative) usability evaluation of a tool. Specific ap-plication areas for climate visualization tools mightrange from:

i) web-based platforms and their applicability for up-loading, displaying and sharing various data sets fora number of research areas (e.g. climate temperaturemodels, land use data, water scarcity data)

ii) tools for dome presentations, their interactivity andcompatibility for different data formats. In these set-tings the visualization tool is ideally accompanied by

a climate expert, guiding the audience through thenarrative and providing additional information and thepossibility for increased interaction and response tospontaneous questions.

Controlled experiments comparing two or more tools.Presentations using climate visualization in immer-sive environments are becoming more frequent asthe number of visualization domes (both portable aswell as large dome theatres) increases throughout theworld. Hence, it is vital that the visualization toolis flexible (supporting numerous data formats, inter-action techniques and graphical representations) andintuitive to use for the presenter. Controlled experi-ments, using the same environment with identical set-tings, should thus be used to compare visualizationtools currently under development with what is state-of-the-art to advance the insights on user perspectives.

Case studies of tools in realistic settings. While pre-sentations to a larger audience have certain require-ments on the visualization tool, climate visualizationfor analysis and decision making support is an equallyimportant area with a specific setup of demands. Aparticular need of flexibility, interactivity and ease ofuse is posed by climate researchers handling both cat-egorical and numerical data and the inherent need oftheir studies to provide decision support to plannersand policy makers on a local to global scale. Theevaluation process should involve new applicationsfunctioning as visualization platforms, that includethe development, storage and possibility to displayand share relevant data sets in an interactive sessionwith for instance regional decision makers. As thedirect use of the visualization tool is of significant im-portance for the development and improvement of theapplication, climate researchers and decision makersneed to be observed in their interaction with the toolin a realistic decision environment.

5 ConclusionsCommunicating climate change is a global challenge,

both in terms of the large amount and complexity of data,scenarios and the mixture of categorical and numerical datasets. Communication and analysis tools therefore need tobe adapted to meet the specific goals and tasks at hand—be it to provide a scientific understanding of complex phe-nomena or to tell a nuanced story to a school class. In-formation visualization provides climate researchers withsuch interactive tools to meet the demands of science com-munication, analysis, planning and decision making.

In this paper we have reported on previous experiencesand identified specific areas of climate visualization where

evaluation methods commonly used in information visu-alization can be applied to develop and improve existingtools as well as gain a better understanding of the effectsand efficiency of developed visual representations of cli-mate change related issues. These evaluation methods needto be further adapted and designed towards a large varietyof audiences and user groups.

AcknowledgementsThis work was partly supported by the Swedish Re-

search Council in the Linnaeus Centre CADICS.

References[1] C. Chen. Top 10 unsolved information visualiza-

tion problems. Computer Graphics and Applications,25(4):12–16, 2005.

[2] C. Chen and M. P. Czerwinski. Empirical evalua-tion of information visualizations: an introduction.International Journal of Human-Computer Studies,53(5):631–635, 2000.

[3] C. Forsell and J. Johansson. Task-based evaluationof multi-relational 3D and standard 2D parallel co-ordinates. In Proceedings of SPIE-IS&T ElectronicImaging, volume 6495, pages 64950C–1–12, 2007.

[4] C. Forsell and J. Johansson. An heuristic set for eval-uation in information visualization. To appear in Ad-vanced Visual Interfaces, 2010.

[5] J. Hedren and B.-O. Linner. Utopian thought andthe politics of sustainable development. Futures,41(4):210–219, 2009.

[6] M. Hjerpe and B.-O. Linner. Utopian and dystopianthought in climate change science and policy. Fu-tures, 41(4):234–245, 2009.

[7] ICE: www.visualiseringscenter.se/1/1.0.1.0/255/2.

[8] ISO 9241-11. Ergonomic requirements for officework with visual display terminals, part 11: Guid-ance on usability, 1998. http://www.iso.ch/iso/en.

[9] IPCC. Climate change 2007: Synthesis report. con-tribution of working groups i, ii and iii to the fourthassessment report of the intergovernmental panel onclimate change. IPCC, 2007.

[10] J. Johansson, C. Forsell, M. Lind, and M. Cooper.Perceiving patterns in parallel coordinates: Deter-mining thresholds for identification of relationships.Information Visualization, 7(2):152–162, 2008.

[11] F. Ladstadter, A. K. Steiner, B. C. Lackner,B. Pirscher, G. Kirchengast, J. Kehrer, H. Hauser,P. Muigg, and H. Doleisch. Exploration of climatedata using interactive visualization. Journal of Atmo-spheric and Oceanic Technology, 2010. Advance on-line Publication: doi:10.1175/2009JTECHA1374.1.

[12] M. Lanzenberger, S. Miksch, and M. Pohl. Exploringhighly structured data: A comparative study of stardi-nates and parallel coordinates. In International Con-ference on Information Visualisation, IV05, pages312–320, 2005.

[13] B.-O. Linner. The Return of Malthus: Environmental-ism and Postwar Population-Resource Crises. WhiteHorse Press, Cambridge, UK, 2003.

[14] T. S. Neset, V. Wibeck, O. Uhrqvist, and J. Johans-son. Visualizing climate change: the potential ofdome presentations as a tool for climate communi-cation. In 31st Annual Conference of the EuropeanAssociation for Computer Graphics.

[15] T-S. S. Neset, J. Johansson, and B.-O. Linner, ed-itors. State of climate visualization. CSPR ReportN:o 09:04, Centre for Climate Science and Policy Re-search, Sweden, 2009.

[16] T. Nocke, M. Flechsig, and U. Bohm. Visual ex-ploration and evaluation of climate-related simulationdata. In WSC ’07: Proceedings of the 39th conferenceon Winter simulation, pages 703–711, 2007.

[17] T. Nocke, T. Sterzel, M. Bottinger, and M Wrobel.Visualization of climate and climate change data: anoverview. In Digital Earth Summit on Geoinformatics2008: Tools for Global Change Research, pages 226–232, 2008.

[18] C. Plaisant. The challenge of information visualiza-tion evaluation. In Advanced Visual Interfaces, pages109–116, 2004.

[19] The WorldView project: http://www.cspr.se/vcc.

[20] T. Zuk, L. Schleiser, and P. Neuman. Heuristics forinformation visualization evaluation. In BEyond timeand errors: novel evaLuation methods for Informa-tion Visualization (BELIV’06), pages 1–6, 2006.