ARTICLE IN PRESS

0098-3004/$ - se

doi:10.1016/j.ca

�Tel.: +1 86

E-mail addr1Notice: This

LLC, under

Department of

and the publi

acknowledges t

exclusive, paid-

reproduce the p

to do so, for U

Computers & Geosciences 33 (2007) 685–695

www.elsevier.com/locate/cageo

Implementation of a parallel high-performancevisualization technique in GRASS GIS

Alexandre Sorokine�,1

Geographic Information Science & Technology Group, Oak Ridge National Laboratory, P.O. Box 2008 MS– 6017,

Oak Ridge, TN 37831-6017, USA

Received 29 March 2006; received in revised form 24 July 2006; accepted 18 September 2006

Abstract

This paper describes an extension for GRASS geographic information systems (GIS) that enables users to perform

geographic visualization tasks on tiled high-resolution displays powered by the clusters of commodity personal computers.

Parallel visualization systems are becoming more common in scientific computing due to the decreasing hardware costs

and availability of the open source software to support such architecture. High-resolution displays allow scientists to

visualize very large data sets with minimal loss of details. Such systems have a big promise especially in the field of GIS

because users can naturally combine several geographic scales on a single display.

This paper discusses architecture, implementation, and operation of PD-GRASS—a GRASS GIS extension for high-

performance parallel visualization on tiled displays. PD-GRASS is specifically well suited for very large geographic data

sets, such as light detecting and ranging data or high-resolution nation-wide geographic databases. This paper also briefly

touches on computational efficiency, performance, and potential applications for such systems.

r 2006 Elsevier Ltd. All rights reserved.

Keywords: Geographic information systems; High-performance computing; Visualization; Open source software

1. Introduction

The importance of visualization and graphicaltechniques for the geographic information systems

e front matter r 2006 Elsevier Ltd. All rights reserved

geo.2006.09.008

5 576 7597; fax: +1 865 241 6261.

ess: sorokina@ornl.gov.

manuscript has been authored by UT-Battelle,

contract DE-AC05-00OR22725 with the US

Energy. The United States Government retains

sher, by accepting the article for publication,

hat the United States Government retains a non-

up, irrevocable, world-wide license to publish or

ublished form of this manuscript, or allow others

nited States Government purposes.

(GIS) is difficult to overestimate. The main challengesof GIS visualization stem from the size and complex-ity of the geographical data. The amount of GIS datasuitable for visualization is ever increasing and haslong exceeded capacities of computer graphicalsystems and human cognitive abilities. Hence, theimprovements of geographic visualization and gra-phical techniques can significantly benefit all areas ofgeographic information science.

The growth of the amount of available geo-graphic information can be attributed to severalfactors. Among these factors are the introduction ofa variety of new data acquisition techniques, such ashigh-resolution satellite imagery and light detecting

.

ARTICLE IN PRESS

Display

HeadNode

Display

Display

RenderingNode

DisplayRendering

Node

RenderingNode

Fig. 1. Visualization cluster architecture.

A. Sorokine / Computers & Geosciences 33 (2007) 685–695686

and ranging (LIDAR) data, or the recently emergedability to integrate various sources of georeferenceddata from the Internet.

The lack of computational resources is not amajor limiting factor in accommodating the in-creased data sizes. The growth of the processingspeed, amount of random access memory (RAM),and disk space available on an average personalcomputer (PC) is well able to compensate for thelarger data sets. However, there has been nocomparable improvement of the graphical systemcapabilities apart from the enhancements to thethree-dimensional (3D) rendering processing speed.Display resolution is seldom viewed as a resourcethat has a potential for significant growth.

In the high-end graphics systems, bigger displaysize and higher level of detail were traditionallyachieved by combining a large number of smallerdisplay devices into a single display. Until recently,such systems have had limited availability due to thevery high cost of specialized hardware and software.Recent advancements in display and cluster tech-nology have resulted in a significant decrease in thecost of high-resolution, tiled displays by employingcommodity PC hardware.

This paper discusses a software package that wasdeveloped at the Oak Ridge National Laboratory(ORNL) to enable effective visualization of largegeographic data sets on tiled high-resolution dis-plays powered by the clusters of Linux-basedcommodity PCs. The package was implemented asan add-on for GRASS GIS (Neteler and Mitasova,2002) and is called PD-GRASS (abbreviated from‘‘Parallel Display for GRASS GIS’’). GRASS GISis a community-supported, free GIS distributedunder the GNU Public License (GNU, 1991).

The remainder of the paper is structured asfollows. Section 2 introduces the reader to parallelvisualization and tiled high-resolution displaystechnology. Section 3 discusses the current stateand challenges of using parallel visualizationtechniques for GIS. Section 4 details the designprinciples, configuration and operation of PD-GRASS. This paper is concluded with a discussionabout the advantages of parallel visualizationtechniques in the GIS context and potential futuredevelopments in this area.

2. Parallel visualization and tiled display technology

For its practical operation PD-GRASS requiresspecialized hardware—a cluster of computers and

display devices that is set up to produce a single fieldof view combined from several display tiles. Thearchitecture of a visualization cluster is shown inFig. 1. Such clusters typically consist of severalcommodity PCs connected through a high-speedInternet Protocol (IP) network. One of the PCs isdesignated as the head node and is used to controlother PCs that are used as rendering nodes to drivethe displays. Each rendering node is equipped with agraphics card and has one or more display devicesattached to it. Liquid crystal displays (LCD) anddigital light projectors (DLP) are two displaydevices that are commonly used in such systems.

The physical arrangement of the display devicesshould allow users to perceive them as a singlecontinuous display field. Synchronization and otheroperations needed for the display devices andrendering nodes to perform in a coherent mannerare sustained by the cluster software.

The EVEREST visualization cluster at the ORNLNational Center for Computational Sciences(NCCS) is one such system. EVEREST combines27, 3500 lumen DLPs to produce a single field ofview on a back-lit 300 � 80 ð9:14m� 2:43mÞ glassscreen. Each projector displays a native resolutionof 1280� 1024 pixels. The total size of thecontiguous pixel array that is generated by theEVEREST visualization cluster is 11520� 3072pixels. The cluster is powered by 14 networkedcommodity dual processor Opterons PCs runningthe Linux operating system (OS) (Daniel, 2004).

ARTICLE IN PRESS

2http://dmx.sf.net, accessed on February 14, 2006.

A. Sorokine / Computers & Geosciences 33 (2007) 685–695 687

3. GIS in the context of parallel visualization

GIS visualization has emerged at the intersectionof several disciplines including traditional paper-based cartography and map-making, spatial data-bases, cognitive science, and scientific visualization(MacEachren and Kraak, 1997, 2001). Even thoughGIS visualization builds strongly upon the advance-ments in cartography, these two disciplines cannotbe equated to each other (Taylor, 1994). The mainpurpose of visualization is ideation, i.e., a process offorming and relating ideas and thoughts (Gahegan,2005). Unlike paper maps, GIS visualization arti-facts are highly dynamic and interactive and tend tohave a short life span. GIS visualization methodsare targeted at manipulation of available data setswith the purpose of discerning previously unknownmodels and relationships.

GIS visualization extends the whole notion of amap itself and overcomes many limitations of thepaper-based cartography like, for example, staticnature of paper as a medium (Goodchild, 2000). Yetat the same time GIS visualization strongly dependsupon the underlying computer technology anddevelopment of new concepts and theories ingeographic information science. Dykes et al.(2005) outlines four major challenges of GISvisualization for mid-2000s: (1) development ofexperimental and multi-modal maps, (2) effectiveaccess and representation of large data sets,(3) support for the group work, (4) human-centeredapproach.

Tiled displays as described in Section 2 caneffectively address such challenges as access to largedata sets and support for the group work. Theproblem of the visualization of large data sets istwo-fold. On one hand, developers of such systemmust address the issues of effective access andprocessing of large volumes of geographic data. Inmodern and near-future applications the datavolume can rich terabytes and in some casepetabytes of storage. On the other hand, newrepresentation, generalization and abstraction tech-niques have to be developed with the main goal notto overwhelm the user with the vast amount ofinformation.

One of the problems with desktop monitors is thedifficulty to combine a good overview of anyreasonably large data set with a detailed viewwithout using zoom and pan functions. In thissense, desktop GIS visualization falls short evencompared to traditional paper maps that have much

higher resolution capacity than desktop monitors.Simple zoom-in and zoom-out functions availablein most existing GIS only partially address thisproblem. Typically users have to zoom in and outseveral times to be able to observe objects atdifferent scales on the map. This is a result of alimited size of the working visual memory inhumans that causes users to loose the generalcontext easily (Vogel et al., 2001).

Two typical approaches to improve navigationbetween scales are hyperlinks and multiple coupledviews. With hyperlinked maps a user can click on apart of the map to get a larger view. This makeszooming easier and faster and helps users to keeptheir attention on the subject thus preserving moreobjects in the working visual memory. Visualizationsystems with coupled views present a user withseveral windows each containing a portion of themap rendered at different scale. Operations in onewindow are reflected in all other coupled windows.In such environment a user can switch betweendifferent scales only with the eye movement withoutperforming any additional operations like mouseclicks and window drags (Ware and Plumlee, 2005;Ware et al., 2001).

Tiled displays can dynamically generate a sig-nificantly larger number of pixels as compared todesktop computers thus allowing users to see localdetails and a much wider context at the same time.GIS visualization can strongly benefit from high-resolution large displays because of the virtuallyunlimited size and multi-scale nature of geographicdata. Compared to paper maps, tiled displays canshow dynamically updated or streamed content.

Visualization clusters significantly improve ren-dering performance due to the inherent parallelismof their architecture. Typically each cluster nodegets only a portion of the whole picture to renderand has to process only a small subset of a biggerdata set. However, running GIS software on aparallel architecture represents significant chal-lenges. GIS software applications must be eithermodified for the parallel execution or used inconjunction with an OS-level software that is ableto distribute its output among rendering nodes.

There are several OS-level software packagesthat support visualization on tiled displays. In ourstudy we evaluated two packages (Xdmx2 and

ARTICLE IN PRESSA. Sorokine / Computers & Geosciences 33 (2007) 685–695688

Chromium3) for their suitability to enable parallelGIS visualization. Xdmx is a distributed X/Win-dows server that unifies multiple displays and usesthem to render graphical output produced by anystandard X/Windows application. Chromium is adistributed OpenGLt renderer that performs asimilar function for the OpenGLt-based applica-tions. Neither package requires modifications of theuser application.

This approach is effective only with relativelysmall GIS data sets and static visualization. BothXdmx and Chromium require the user applicationto be run on the head node (see Fig. 1) while itsgraphical output is intercepted and distributedbetween the rendering nodes. In case of large datasets this leads to a significantly impaired graphicalperformance. The decreased performance is due tomost of the computations being carried out by thehead node rather than the load being shared evenlyamongst the nodes of the cluster.

Application-level software for tiled displays isrepresented by several packages that were developedto support scientific and engineering visualization orhigh-resolution image and video rendering. Suchsystems have support for the visualization ofgeographic data. NCSA TerraServer Blaster,4 de-veloped by the National Center for SupercomputerApplications (NCSA) is capable of rendering aerialimagery transparently downloaded from the Micro-softs Terraserver.5 One of the most popularparallel visualization systems is VisIt,6 developedat Lawrence Livermore National Laboratory. Itfeatures a very wide range of visualization tools andhas an option to import GIS file formats.

Visualization software for display walls lacksmany salient GIS functions. Obvious shortcomingsinclude the lack of support for cartographic projec-tions and special cartographic symbology. None ofthe systems has a function to apply drawingattributes to the display objects based on thecontent of the associated relational database table.Another important feature lacking in the existingsoftware is the ability to combine many geographic

3http://chromium.sf.net, accessed on February 14,

2006.4http://brighton.ncsa.uiuc.edu/�prajlich/

wall/tsb.html, accessed on February 14, 2006.5http://terraserver.microsoft.com, accessed on

February 14, 2006.6http://www.llnl.gov/visit/, accessed on February

14, 2006.

data sources on a single display with on-the-flyprojection and file format conversion.

4. Design and implementation

The choice of the strategy to develop a parallelvisualization system for geographic data as anextension for an existing GIS was based on twomain considerations: (1) the system must have areasonably wide range of GIS visualization func-tions, and (2) development costs should be mini-mized. One of the goals of the development was tocreate a system that would be able to employEVEREST visualization capabilities to displaygeographic data in an environment that is familiarfor a GIS user. Other requirements were tominimize the necessary knowledge of parallelcomputing by a user to set up and operate thesystem.

We have chosen the freely available GRASS GISas a base system. GRASS GIS was originallydeveloped at the US Army Construction Engineer-ing Research Laboratories (USA-CERL) and is afull-featured GIS with a wide range of analytical,data management, and visualization capabilities.Currently GRASS GIS is supported by a growingcommunity of volunteer developers. The use ofGRASS GIS in a parallel visualization environmentpresents many advantages.

GRASS GIS was originally designed for the Unixenvironment, and its current development is pri-marily Linux-based. Linux support is critical forthis study because the field of parallel computationis dominated by Linux and UNIX systems. Thisfactor significantly limits the use of commercial GISbecause most commercial vendors do not provideLinux support.

Another important consideration is the cost ofsoftware licensing. GRASS GIS is licensed underthe GNU Public License (GPL; GNU, 1991) anddoes not impose fees on each running copy of thesystem. This has a major impact on the clusterenvironment where a copy of the software must berun on several nodes at the same time. The cost oflicensing commercial GIS software for use in amulti-node cluster environment was prohibitivelyexpensive for this project. Also the availability ofthe source code, as demanded by the GPL allowsdevelopers to make necessary modifications to thesystem to adapt it for the parallel computationenvironment.

ARTICLE IN PRESSA. Sorokine / Computers & Geosciences 33 (2007) 685–695 689

GRASS GIS features wide functionality and iscapable of operating with raster and vector topolo-gical data that can be stored in an internal format ora relational database. However, compared tocommercial GIS, GRASS has limited cartographiccapabilities and an outdated user interface. In thecontext of this study, GRASS GIS advantages faroutweighed its limitations.

4.1. Design principles

The two-dimensional (2D) GIS rendering processcan be effectively parallelized by subdividing thegeographic region to be displayed into rectangularsubregions corresponding to each tile of the display.This is the main principle of PD-GRASS opera-tion—subdivide the regions of interest among therendering nodes proportionally to the size of thedisplay tiles as controlled by the node.

PD-GRASS is implemented as a series of shellscripts that wrap existing GRASS functionality tomake it available in cluster environment. Also, thereare several accessory scripts to maintain the GRASSGIS installation and the GRASS GIS database onthe cluster.

All PD-GRASS commands follow GRASS GISnaming and command-line argument conventions.PD-GRASS scripts are prepended with a two-lettercommand group name followed by a period (pd.),i.e., pd.region, pd.mon, etc. Most PD-GRASScommands mimic the behavior of their standardGRASS counterparts and use the standard GRASSGIS parser to process command-line arguments.The meaning of the command line argumentsconforms to GRASS GIS conventions.

Another design requirement was to reduce thedependence of PD-GRASS upon external librariesand utilities due to the difficulty of installing andmaintaining them in the cluster environment. Theonly external programs that are required for PD-GRASS operation are a remote shell client and acommand-line calculator GNU bc.7 PD-GRASS wastested with an OpenSSH client as a remote shell, butit can be easily substituted with other similarprograms.8 The rsync remote copy utility may beneeded if the PD-GRASS replication functionis used.

7http://www.gnu.org/software/bc/, accessed on Feb-

ruary 14, 2006.8http://www.openssh.com/, accessed on February 14,

2006.

PD-GRASS is being developed against the mostrecent developers’ version of GRASS GIS that isbeing regularly checked-out from the GRASS GISproject version control system repository. At themoment of the preparation of this manuscriptGRASS GIS version 6.1.x was used for thedevelopment. PD-GRASS relies on a relatively smalland stable subset of the GRASS GIS functionalityand is fully operational on the stable GRASS GISversion 6.0.x.

4.2. Installation and configuration

PD-GRASS was developed as an add-on toGRASS GIS and requires a functional GRASSGIS installation for its operation. GRASS GIS, inturn, depends on several external libraries (proj4,gdal, and others) that are not commonly found inLinux distributions. Typically this is not a signifi-cant problem on a personal workstation where theselibraries can be downloaded and installed by theuser. However, it is much harder to solve thisproblem in a cluster environment when the librariesneed to be synchronously installed and updated on alarge number of hosts.

To address this problem PD-GRASS distributioncontains a script called gengrass.sh that auto-mates download, configuration, and compilation ofGRASS GIS and its supporting libraries. Aftercompilation, GRASS GIS and its libraries areinstalled in a self-contained catalog under the user’shome directory. This installation directory containsa fully operational GRASS GIS installation and canbe easily copied across different computers withidentical OS set-up and architecture. No adminis-trative privileges are needed to access all functionsof gengrass.sh, and gengrass.sh can beconfigured with the site-specific compilationoptions.

PD-GRASS configuration is stored in a file(pd.init) under the PD-GRASS directory withinthe GRASS GIS installation tree. The PD-GRASSconfiguration file specifies a list of the cluster nodehostnames and the geometry of the tiled display. PD-GRASS implies that the cluster nodes are accessiblethrough a remote shell (OpenSSH client by default)with no password prompt. Each node must run anX/Windows server without a display manager.Configurations with either shared or local filesystems are supported.

An example of the PD-GRASS configuration file isshown in Fig. 2(a) with the corresponding display

ARTICLE IN PRESS

2560 12801024

2048

node01

node03 node04

node02

node05

0,0

a single nodedisplay tile

2560,0

0,1024 1280,1024 2560,1024

tile 4

tile 1

tile 7

tile 2

tile 5

tile 8 tile 9

tile 6

tile 3

tiles served by

38403072

b

Fig. 2. Display geometry and configuration file. (a) Configuration file and (b) tiled display geometry.

A. Sorokine / Computers & Geosciences 33 (2007) 685–695690

geometry in Fig. 2(b). In this example, the displayconsists of nine display tiles (tiles 1–9 are outlinedwith dashed lines on Fig. 2(b)) driven by fiverendering nodes (nodes numbered 01–05 are out-lined with solid lines on Fig. 2(b)). The totaldimensions of the display is 3840� 3072 pixels.Nodes 01, 03, 04, and 05 drive two tiles each and arealigned either horizontally (node 01) or vertically(nodes 03, 04, 05). Node 02 drives a single tile in theupper right corner of the screen.

Each line of the configuration file (Fig. 2(b))describes geometry of the display rendered by asingle node. It must contain the host name of thenode, the pixel coordinates of the origin of the nodedisplay relative to the upper left corner of theunified screen, and the pixel dimensions of the nodedisplay. Lines starting with ‘‘#’’ are comments. Sucha display specification is flexible enough to accom-modate various configurations of tiled displays

including configurations with unequally sized tiles.It also allows for gaps in the combined pixel arrays.Gaps are typically needed to support tiled displaysthat are assembled from LCD panels with a bezel.

The final step to make PD-GRASS operational isby replicating the GRASS GIS database andexecutables from the head node to the renderingnodes of the cluster. This step is not necessary if thecluster is set up to use a shared file system. However,file systems like Network File System (NFS) cansignificantly degrade PD-GRASS performance be-cause the NFS server becomes overwhelmed withthe data retrieval requests during the redrawoperation. Typically it makes sense to keep a copyof GRASS GIS executables on a shared file systemand replicate the GRASS GIS database to eachnode. Replication of both the GRASS GIS codeand database can be performed using the pd.syncscript from PD-GRASS distribution. Internally,

ARTICLE IN PRESS

Fig. 3. SRTM data set on the tiled display at ORNL.

9http://www2.jpl.nasa.gov/srtm/, accessed on February

14, 2006.

A. Sorokine / Computers & Geosciences 33 (2007) 685–695 691

pd.sync uses rsync to reduce the time needed forupdates on non-shared file systems.

4.3. Operation

The operation of PD-GRASS is similar to theoperation of GRASS GIS itself. GRASS GIS, withthe PD-GRASS module, must be installed on thecluster head node and replicated or shared throughthe rendering nodes. PD-GRASS needs to beinitialized from the GRASS GIS shell using thepd.init command. This command will startGRASS GIS sessions on each node of the cluster.The sessions will stay open until the pd.termi-nate command is issued.

Next, start the GRASS GIS monitors with thepd.mon command. This command will start theGRASS GIS monitors on each rendering node. Ithas the same arguments as the standard GRASSGIS d.mon command that can be used to set themonitor background color and name. On the headnode, pd.mon will open a GRASS GIS monitor ina window the size of which is proportional to thedimensions of the tiled display. This monitor will beused by consequent PD-GRASS commands to echothe content of the tiled display and to performoperations such as zoom and pan.

The final step necessary to prepare PD-GRASS todraw maps on the tiled display is to set the global

region. To do so, specify a region for the activeGRASS GIS mapset with a standard g.regioncommand and then run the PD-GRASS pd.re-gion command. This command will set the regionof each node to the portion of the global region thatwas set on the head node. Not running pd.regionwill cause each node to use the global regions asthe default.

After invoking pd.region, the user can rundistributed display commands such as pd.rastand pd.vect. These commands have exactly thesame command line arguments as their GRASS GIScounterparts d.rast and d.vect. The outputshown in Fig. 3 was produced using Shuttle RadarTopography Mission (SRTM)9 elevation data andsome additional data layers with the following PD-GRASS commands:

pd.rast srtm3secpd.vect streams color ¼ bluepd.vect lakes fcolor ¼ bluepd.vect states color ¼ white

Most of the distributed commands like pd.vectand pd.rast are implemented as symbolic links topd.run that is a universal wrapper for any GRASS

ARTICLE IN PRESSA. Sorokine / Computers & Geosciences 33 (2007) 685–695692

GIS command and accepts a GRASS GIS com-mand as its argument. It executes this commandfirst on the rendering nodes through a remote shelland then on the head node. On each node thecommand will use the region that is currently set onthat node. If pd.region was run before pd.run,then each node’s region is proportional to thedisplay size of that node and within the boundariesof the region set on the node. pd.run can be usedto execute in parallel almost any GRASS GIScommand. For example, the following commandwill render in parallel a hue–intensity–saturationcomposite image with a white grid on top of it onboth local and tiled displays:

pd.run �b command ¼ ’d.his n

h _map ¼ hue i_map ¼ intensity ns_map ¼ saturation’

pd.run �b command ¼ ’d.grid –b size ¼ 6 ncolor ¼ white’

Some other commonly used PD-GRASS com-mands are pd.redraw and pd.zoom. The formercauses both the head node GRASS GIS monitorand tiled display to be regenerated and pd.zoomallows users to interactively zoom and pan the mapon the tiled display. Zoom and pan controloperations are conducted by the user on the headnode GRASS GIS monitor. After the user hasfinished zooming and panning, pd.zoom willrecalculate the boundaries of the region andredistribute it across the rendering nodes. It willthen redraw both the head node monitor and thetiled display. In the non-interactive mode pd.zoomcan be invoked by specifying a predefined zoom

Fig. 4. PD-GRASS test sess

level or the coordinates of the corners of the regionof interest.

PD-GRASS version 1.1 has additional function-ality coupling it with the GRASS GIS displaymanager d.m. Newly introduced commandpd.replay retrieves a list of layers from thecurrently active GRASS GIS monitor on the headnode and renders them on the tiled display. Userscan create a visual representation of a map usingeither advanced display manager controls, GRASSGIS menu system or command-line interface. Afterinvoking pd.replay this map will be recreated onthe tiled display at the highest resolution of thedisplay system.

Closing a PD-GRASS session involves two steps:pd.mon stop ¼ xN will stop the GRASS GISmonitors and pd.terminate (the counterpart ofpd.init) will terminate the GRASS GIS sessionson the rendering nodes. It is strongly recommendedthat users close PD-GRASS sessions using thepd.terminate command; otherwise, temporaryfiles are left on the nodes.

4.4. Maintenance and development

PD-GRASS also provides two additional main-tenance commands:

�

ion

pd.check is used to verify the consistency ofthe configuration files across the cluster, to checkthe physical alignment of the tiles, and toperform other similar tasks.

� pd.clean is used to remove temporary files that

are left from improperly terminated PD-GRASSsessions.

on a local system.

ARTICLE IN PRESSA. Sorokine / Computers & Geosciences 33 (2007) 685–695 693

For testing purposes, all PD-GRASS commands canbe run on a local system. There is a special sampleconfiguration file in the PD-GRASS distribution thatdemonstrates how six independent GRASS GISsessions can be run by a single user (Fig. 4). The sixwindows on the right side of Fig. 4 imitate displaytiles. They can be controlled from a head-nodewindow as on the left side of the figure.

4.5. Code availability

The code for PD-GRASS has been releasedunder GNU Public License (GNU, 1991) and isavailable for download from the ORNL web site(http://www.ornl.gov/sci/gist/software/grass/). Currently the PD-GRASS code is maintainedby Alexandre Sorokine (e-mail: SorokinA@ornl.gov).

5. Conclusions and future development

This paper presents a parallel visualization soft-ware package for GIS applications called PD-GRASS. The package is implemented as an add-on module for the freely available GRASS GIS(Neteler and Mitasova, 2002). It is intended to beused on specialized hardware—high-resolution tileddisplays powered by the clusters of Linux-basedPCs. Such displays are not yet commonly availablefor the geocomputing community. However, theirnumber is expected to grow. PD-GRASS has beentested with the EVEREST visualization cluster(see Section 2) and a smaller 9-tile LCD-basedparallel visualization system at ORNL. This type ofhardware significantly decreases the costs associatedwith high-end visualization systems and at the sametime opens new possibilities for scientific visualiza-tion and presentation graphics.

GIS is a good candidate application for parallelvisualization and very high-resolution displays. GISdata sets are often large or very large, meaning thatthey can productively utilize all pixels of a high-resolution display. In many cases, GIS computa-tional and visualization algorithms can be effec-tively parallelized by partitioning a data set intonaturally defined geographic regions. With tileddisplays, these regions can correspond to thegeometry of the tiles.

The biggest benefit of using high-resolution, tileddisplays to visualize geographic data is the ability ofthe user to observe local details while preserving afirm grasp on the larger and more general context.

In other words, on a high-resolution display, itis possible to perceive several geographical scalesat once. Such systems allow users to employthe ability of human cognition to easily compre-hend a general picture while simultaneously main-taining the intricate details that are typicallyimpaired by small, and low-resolution desktopdisplays.

The use of high-resolution displays is especiallyproductive with very large geographic data sets suchas LIDAR data or high-resolution, global geo-graphic databases such as the LandScan globalpopulation data set or SRTM elevation data(Bhaduri et al., 2002). With LIDAR data, it ispossible to observe large elevation features such asriver valleys and gorges while identifying minordetails such as houses, trees, or ditches. On nation-wide data layers representing electric grids or roadnetworks, a user is able to see the large regions ofvarying network density and at the same time trace-specific electric lines or routes.

The use of parallel hardware also improvesrendering speed because the processing load canbe shared by the rendering nodes. For example,rendering of the 3-second SRTM data set for thewhole US (data set size is approximately 90; 000�30; 000 cells, see Fig. 3) takes about 40 s on the headnode at 1280� 1024 window size. Rendering of thesame data set using PD-GRASS on the tiled display(11; 520� 3072 pixels) with the database replicatedon each node takes only about 20 s. The approachpresented in this paper relies on the full paralleliza-tion of data and code and is applicable to 2D GISvisualization.

Design principles that predicated PD-GRASSdevelopment were geared toward the ease ofimplementation, maintenance, and use of thepackage by professionals familiar with GRASSGIS. The package was designed to minimize restric-tions on other GRASS GIS functionality.

The current version of PD-GRASS exhibits certainlimitations that its potential users have to consider.One of the biggest issues is the inefficiency ofupdates in cases where a replicated GRASS GISdatabase is used. The easiest way to address thisissue while preserving high performance is to use aparallel file system (e.g., Lustres). Other limita-tions of PD-GRASS stem from GRASS GIS itself.GRASS GIS has rather outdated cartographiccapabilities as compared to commercial offeringsand lacks rendering optimization for large rasterdata sets.

ARTICLE IN PRESSA. Sorokine / Computers & Geosciences 33 (2007) 685–695694

PD-GRASS has room for improvement. One ofthe obvious extensions is the support for dynamicGIS visualization. In the user-interface area, inte-gration with the GRASS GIS interactive menusystem and a new GRASS GIS display manager(gis.m) would be a significant usability enhance-ment. Another interesting possibility would be toextend PD-GRASS to support both visualizationand computational parallelism of a cluster byutilizing idling central processing units to performsome computational tasks. For example, the clustercan be running a simulation model or performinganalytical GIS operation and at the same timedisplaying intermediate or final results of suchoperations (Sorokine et al., 2005).

It would also be possible to build a systemfor dynamic, real-time, 3D GIS visualization ontiled displays, including the support of a full setof cartographic portrayal features. However,such systems would require a significantly differentsoftware architecture and much higher invest-ments in programming power. Rendering of a3D image in a parallel computing environment isnot as simple as rendering of a 2D image be-cause there is no direct correspondence betweenthe display tiles and subregions of the data set. Insuch case more sophisticated mechanisms of redis-tribution of graphical elements between renderingnodes would have to be implemented (Molnar et al.,1994).

Native GRASS GIS 3D visualization tool nvizcan be run on a tiled display with the help ofChromium and Xdmx in which case nviz has tobe executed on the head node and its 3D outputwould be distributed among rendering nodes.However, this solution is not practical due to theperformance penalty as the processing cannot befairly distributed between the rendering nodes andmost of the computations has to be performed onthe head node. To solve the problem other strategieshave to be sought such as integration of GIS toolswith existing parallel visualization software like, forexample, VisIt or ParaView10 (Rhyne, 1997; Rhyneet al., 1994).

Acknowledgments

Research sponsored by the Laboratory DirectedResearch and Development Program of Oak Ridge

10ParaView: Parallel visualization application by KitWare Inc.,

http://www.paraview.org/

National Laboratory (ORNL), managed by UT-Battelle, LLC for the US Department of Energyunder Contract No. DE-AC05-00OR22725.

This research used resources of the NationalCenter for Computational Sciences at Oak RidgeNational Laboratory, which is supported by theOffice of Science of the US Department of Energyunder Contract No. DE-AC05-00OR22725.

References

Bhaduri, B., Bright, E., Coleman, P., Dobson, J., 2002. Land-

scan: locating people is what matters. Geoinformatics 5 (2),

34–37.

Daniel, J.R., 2004. A middleware graphical toolkit for multi-

framed display environments. Master’s Thesis, The University

of Tennessee, Knoxville, 82 pp.

Dykes, J., MacEachren, A.M., Kraak, M.-J., 2005. Exploring

geovisualization. In: Dykes, J., MacEachren, A.M., Kraak,

M.-J. (Eds.), Exploring Geovisualization, first ed. Elsevier,

Oxford, UK, pp. 3–22 (Chapter 1).

Gahegan, M., 2005. Beyond tools: visual support for the entire

process in GIScience. In: Dykes, J., MacEachren, A.M.,

Kraak, M.-J. (Eds.), Exploring Geovisualization, first ed.

Elsevier, Oxford, UK, pp. 83–99 (Chapter 4).

GNU, 1991. GNU GENERAL PUBLIC LICENSE, second ed.

Free Software Foundation, Inc., Boston, MA, USA,

hhttp://www.gnu.org/copyleft/gpl.htmli.

Goodchild, M.F., 2000. Communicating geographic information

in digital age. Annals of the Association of American

Geographers 90 (2), 244–355.

MacEachren, A.M., Kraak, M.-J., 1997. Exploratory carto-

graphic visualization: advancing the agenda. Computers &

Geosciences 23 (4), 335–343.

MacEachren, A.M., Kraak, M.-J., 2001. Research challenges in

geovisualization. Cartography and Geographic Information

Science 28 (1), 3–12.

Molnar, S., Cox, M., Ellsworth, D., Fuchs, H., 1994. A sorting

classification of parallel rendering. IEEE (Institute of

Electrical and Electronics Engineers) Computer Graphics

and Applications 14 (4), 23–32.

Neteler, M., Mitasova, H., 2002. Open Source GIS: A GRASS

GIS Approach. Kluwer International Series in Engineering

and Computer Science, vol. 689. Kluwer Academic Publish-

ers, Springer, Boston, MA, USA, 424pp.

Rhyne, T.M., 1997. Going virtual with geographic information

and scientific visualization. Computers & Geosciences 23 (4),

489–491.

Rhyne, T.M., Ivey, W., Knapp, L., Kochevar, P., Mace, T., 1994.

Visualization and geographic information system integration:

what are the needs and the requirements if any?

In: Proceedings IEEE (Institute of Electrical and Elec-

tronics Engineers) Conference on Visualization’ 94. IEEE,

pp. 400–403.

Sorokine, A., Daniel, J., Liu, C., 2005. Parallel visualization for

GIS applications. In: Proceedings GeoComputation 2005.

hhttp://igre.emich.edu/geocomputation2005/i.

Taylor, D.R.F., 1994. Perspectives on visualization and modern

cartography. In: MacEachren, A.M., Taylor, D.R.F. (Eds.),

ARTICLE IN PRESSA. Sorokine / Computers & Geosciences 33 (2007) 685–695 695

Visualization in Modern Cartography. Modern Cartography,

vol. 2. Pergamon, Oxford, UK, pp. 333–341 (Chapter 17).

Vogel, E.K., Woodman, G.F., Luck, S.J., 2001. Storage of

features conjunctions, and objects in visual working memory.

Journal of Experimental Psychology: Human Perception and

Performance 27 (1), 92–114.

Ware, C., Plumlee, M., 2005. 3d geovisualization and the

structure of visual space. In: Dykes, J., MacEachren, A.M.,

Kraak, M.-J. (Eds.), Exploring Geovisualization, first edition.

Elsevier, Oxford, UK, pp. 567–576 (Chapter 29).

Ware, C., Plumlee, M., Arsenault, R., Mayer, L.A., Smith, S.,

2001. GeoZui3D: data fusion for interpreting oceano-

graphic data. In: Proceedings OCEANS, 2001. MTS/IEEE

(Marine Technology Society/Institute of Electrical and

Electronics Engineers) Conference and Exhibition, vol. 3.

pp. 1960–1964.