0 1 / 1 2 / 2 0 1 0

This paper provides a high-level description of the various

types of visualization clusters and briefly covers the use of

commodity hardware to build cost effective solutions. It lists

a selection of commercial and open source applications by

field or specialty, with in-depth descriptions of some of

those applications with an emphasis on the unique value

provided by the cluster solution. While describing different

visualization clusters it focuses on hybrid computational

multi-display clusters. (VizWalls)

Visualization Clusters using Commodity Hardware

2 Visualization Clusters using Commodity Hardware

EXECUTIVE SUMMARY ........................................................................................................................................... 3

VISUALIZATION CLUSTERS ..................................................................................................................................... 4

RENDERING OPTIONS ...................................................................................................................................................... 4

Single System Rendering ........................................................................................................................................ 4

Parallel Rendering .................................................................................................................................................. 5

IMAGE DISPLAY OPTIONS ................................................................................................................................................. 5

Single Workstations ............................................................................................................................................... 5

Large Single Display ............................................................................................................................................... 5

Seamless Multi-Display Systems ............................................................................................................................ 6

Tiled Liquid Crystal Display Panel Display Wall ...................................................................................................... 6

COMBINED DISPLAY WALL AND RENDERING CLUSTER ............................................................................................................ 6

Typical Hardware ................................................................................................................................................... 7 Software .............................................................................................................................................................................. 9 Cluster Management System ............................................................................................................................................... 9 Visualization Middleware – Open Source ............................................................................................................................ 9

Commercial Visualization Applications ................................................................................................................ 11

Open Source Modeling ......................................................................................................................................... 11

COLLABORATION .......................................................................................................................................................... 12

Access Grid ........................................................................................................................................................... 12

OptIPortal ............................................................................................................................................................ 12

ABOUT X-ISS ........................................................................................................................................................ 13

3 Visualization Clusters using Commodity Hardware

Executive Summary

With the exponential growth of affordable compute power has come a similar growth in the ability to solve

more and more complex problems. These complex problems require us to visualize and interact with

increasingly large data set sizes. Dell is a leader in the high performance compute arena and is now

bringing leading edge solutions for users to be able to visualize these massive datasets. The paper

describes the various types of visualization solutions in general and provides specific details on high

performance visualization clusters and walls. It describes the various components of these ‘VizWalls’ and

how customers can leverage these new solutions.

4 Visualization Clusters using Commodity Hardware

Visualization Clusters

Visualization is an important process for making large data sets accessible to humans. It allows us to

explore, interact and understand complex relationships and attributes from the large volume of data

created by scientific instruments and high performance computer simulations and other applications.

Commodity High Performance Computing (HPC) environments have allowed many scientists to use

scientific visualization techniques that were once reserved for the few. Natural scientists may render

molecules; ecologists may model large-scale climate change; and investigators in the life sciences have

access to increasingly detailed datasets for deepening our knowledge and understanding. Creating and

manipulating highly dense and complex images is essential so humans can identify patterns, singularities,

and detail not apparent from the raw data. Visualization clusters have become an indispensable tool for

viewing this complex data.

Just a few years ago, setting up a HPC visualization cluster and display wall required either a large

budget or a lot of do-it-yourself effort. Today, vendors such as Dell provide turnkey visualization clusters

built with low-cost commodity hardware that can be up and running in a matter of days.

Rendering Options

The first aspect of visualization is how the image for display is actually generated (also called rendering).

This section describes the two common methods and technologies used for this process.

Single System Rendering

A common use case in High Performance Computing (HPC) Clusters is the visualization of

computationally intensive problems as static images or animations. This sort of application is

characterized by large input datasets and comparatively small output sizes, such as the visualization of

seismic data. The cluster nodes frequently have no specialized graphics hardware. Long running tasks on

many cluster nodes generate image files that are then displayed on separate display devices or systems.

See Figure 1 for an illustration. In this scenario, the display devices are workstations with good

performance, and they may use high performance graphics cards to display on one or two high-resolution

monitors. The compute and display power of this workstation is typically sufficient for the user to render

and interpret the images from the datasets interactively.

Hundreds or Thousands of

Nodes with Petabytes of Data Graphics Workstations

Local

Storage of

generated

images

Figure 1 - Diagram of Remote Visualization System

5 Visualization Clusters using Commodity Hardware

Parallel Rendering

Another common use case is parallel rendering using graphics hardware without attached displays on

HPCC compute nodes. This need usually arises if the complexity, size, and density of the image make it

difficult for one system to allow effective interaction with the user. In the case of a complex/dense image,

even though the final image may still be on the same display size as the previous example, the compute

and graphics power needed to create and/or interactively manipulate the image (rotate, shade, live

simulation etc.) is not available in a single graphics workstation.

For example, a large 3D model is divided up between available resources on the cluster and a single

workstation is used to display the rendered images. Each of the compute nodes is responsible for

rendering a portion of the image, with the resulting pieces assembled on a single workstation. This type of

parallel rendering known as data parallelism, often used in CAD and medical visualization applications,

provides the user with the required ability to manipulate large models in real time. See Figure 2 below.

A slightly different approach, known as temporal parallelism, divides the rendering into many sequential

frames with each node rendering a complete image. This technique is ideal for animation where real-time

rendering is not as important as total rendering time.

An excellent description of both of these approaches is available in the Dell Power Solutions article

Parallel rendering Technologies for HPC Clusters, available at

http://www.dell.com/downloads/global/power/ps4q07-20070550-Ou.pdf

HPCC cluster with specialized

graphics hardware renders

images

Sends Request (ie: rotate image)Distributes work to

compute nodes

Workload manager

Returns rendered tiles

Graphics workstation

Assembles Complete

Image

High Bandwidth Network

Figure 2 - Parallel Rendering using Data Parallelism

Image Display Options

Single Workstations

The most basic option for display of generated images is the single workstation with one or two high-

resolution screens. Lowest in cost and complexity of the solutions discussed here, this is most useful for

individual users and users collaborating with shared screens with VNC or NoMachine software.

Limited to 4-8 megapixels, these solutions force the user to choose between detailed views of small

datasets and overviews of large datasets. They are not well suited for in-person collaboration with more

than a couple of people.

Large Single Display

Moving up in scale, cost, and complexity, large single displays are available. These displays use LCD

projectors to display large images. Single projector displays are currently limited to about 8 megapixels,

6 Visualization Clusters using Commodity Hardware

nearly four times the resolution of HD video. These displays are useful for larger groups of people viewing

less detailed images from several feet distant, but sacrifice detailed viewing for size.

Seamless Multi-Display Systems

More costly and complex are multi-projection displays. Usually rear projected, these displays can achieve

both high resolution and large, seamless displays. These are the most complex and expensive of the

display types, require precise alignment and consume 300-400 watts per projector, making them the most

power hungry of the group and requiring the most cooling. At 7-9 feet deep, they also require the most

space.

Because of the multiple displays, these solutions require dedicated compute and display servers and

specialized software identical to the tiled LCD panel display wall below.

These displays are very well suited for in-person collaborative visualization and viewing from both

moderate distances and close-up.

Tiled Liquid Crystal Display Panel Display Wall

With cost and complexity between large single displays and seamless multi-display systems, tiled LCD

panel display walls are becoming a very popular solution.

As with multi-projector systems, tiled LCD panels can be built to arbitrary viewing sizes and resolutions. A

modest wall built with nine 2460x1600 30" monitors can contain over 38 megapixels with a 4.5 x 7 foot

viewing area. A 40-monitor wall can have over 168 megapixels in an 8’ x 18’ viewing area. A large wall

like Stallion at the Texas Advanced Computing Center at the University of Texas at Austin has over 307

megapixels.

At less than a foot deep, these walls are slim compared to the multi-projection displays. Power and

cooling is lower also, with each display consuming less than 200 watts.

Large, high-resolution displays like these enable the observer to visualize very detailed information with

clarity while allowing it to be visualized in the context of a large body of data.

Combined Display Wall and Rendering Cluster

A visualization cluster consists of a standard HPC cluster with the addition of high-resolution graphics

cards and monitors arranged in a grid. Typically, an additional graphics workstation is used to control the

wall. The cluster uses standard network interconnects (GbE, IB) and storage.

The addition of specialized visualization software makes the cluster useful for a large range of

commercial and academic applications including medical imaging, cartography, engineering simulations,

geologic exploration, and biologic modeling.

7 Visualization Clusters using Commodity Hardware

Typical Hardware

A visualization wall is typically driven by a standard HPC cluster using graphics workstations for compute

nodes installed with one or more high performance graphics cards. Storage for images and models can

be direct attached to the head node or remotely accessed from enterprise storage pools. For smaller

walls, as show in figure 3, gigabit Ethernet is sufficient for both the management network and the cluster

data fabric. The wall control workstation may also be used as the cluster head node. Ethernet

connection to the control workstation should be at least dual bonded GbE or 10GbE.

Figure 3- Small VizWall Example

For larger walls, with 8 more display nodes such as the one shown in Figure 4, a separate high-speed

network should be added for the cluster fabric.

Figure 4- Large VizWall Example

8 Visualization Clusters using Commodity Hardware

At the extremely large end of the

scale is Stallion at the Texas

Advanced Computing Center.

Stallion has a resolution of over

307 megapixels, provided by 75

30” Dell LCD displays, each with

2650 x 1600 pixels.

The wall is driven by 23 Dell XPS

workstations connected with an

Inifiniband network.

Stallion, build with all commodity

components processes datasets

of a massive scale with interactive

visualization.

At the more modest end of the scale is the 3x3 wall at the left. This wall, also build with Dell 30” high resolution monitors is driven by 5 Dell R5400 rack mounted graphics workstations. Because of the smaller size of the wall, GbE is used for a cluster interconnect, although two ports are bonded on the display controller node. Walls of this size and larger are now

available from Dell as compete turnkey

systems including monitor mounts,

software and installation.

Photo courtesy of: Texas Advanced Computing Center (TACC)

Photo courtesy of: X-ISS, Houston, Texas

9 Visualization Clusters using Commodity Hardware

Software

There are four major software components necessary for a visualization cluster: an operating system, a

cluster management system, visualization middleware and application software.

Cluster Management System

A Cluster Management System (CMS) is used as in a traditional HPC cluster to reduce the complexity of

system provisioning and management. Windows HPC Server 2008 is the CMS of choice for Windows

clusters, but still enjoys less commercial support for viz walls.

For a Linux viz wall, the clear choice for a CMS is Rocks, open source software developed at the

University of California, San Diego. More tiled display walls are constructed using Rocks and its support

for building tiled display walls through its viz roll than any other solution.

Dell provides Linux viz walls using Rocks+GPU from Clustercorp which combines the viz roll and the

CUDA roll with support for GPGPU clusters. The full commercial support from Clustercorp and

professional installation from Dell contribute

to a successful deployment.

Visualization Middleware – Open Source

Displaying across multiple monitors driven by

multiple nodes requires specialized software.

SAGE

SAGE refers to the Scalable Adaptive

Graphics Environment from the Electronic

Visualization Laboratory (EVL) at the

University of Illinois at Chicago.

SAGE is a graphics streaming architecture for

supporting collaborative scientific visualization

environments with potentially hundreds of

megapixels of contiguous display resolution. In

collaborative scientific visualization, it is crucial

to share high-resolution imagery as well as

high-definition video among groups of collaborators at

local or remote sites. (Scalable Adaptive Graphics

Environment -EVL website).

Unlike many display drivers that display the output of a single application on an entire wall, SAGE can

allow many different applications to display on different areas of a high-resolution display wall

simultaneously. It allows simultaneous display of pixel streams from various sources to be tiled across a

wall making it an excellent choice for collaboration.

Linux Software Stack

Application Software

Visualization Tools SAGE, Chromium, DMX

Cluster Management System

Cluster Corp Rocks+

Operating System Red Hat or CentOS

Windows Software Stack

Application Software

Visualization Tools SAGE, Chromium

Cluster Management System

HPC08

Operating System Windows Server 2008

Table 1 - Typical Software Stacks

10 Visualization Clusters using Commodity Hardware

Many applications support SAGE, including images and movie viewers, scientific visualization, and

collaboration. For applications without built in support, SAGE comes bundled with OpenGL capture, a

package that enables software that uses OpenGL to display through SAGE without source code

modifications.

SAGE includes a graphical user interface that models the tiled display, allowing users to position

application windows. Collaborators may each run a user interface and interact with the display at the

same time.

SAGE is under active development and is used by dozens of visualization applications. It is available for

Linux, Windows, and Mac OS X. It was recently used by a team from LSU as part of an application that

won first prize at the IEEE Scalable Computing Challenge in Shanghai.

Chromium

Chromium is a flexible framework for scalable real-time rendering on clusters of workstations, derived

from the Stanford WireGL project code base. It intercepts OpenGL calls and automatically divides the

work up between cluster nodes, each with its own display(s).

It is a completely extensible architecture, so that parallel rendering algorithms can be implemented on

clusters with ease. (Chromium Documentation) It runs on Windows, Linux, and several Unix distributions.

First released in 2001, the current version, Chromium 1.9, was released in 2006.

DMX

Ximerama is an X windows extension that presents multiple displays as one contiguous screen. The size

of the aggregated screen is limited by the number of physical devices that are supported in a single

machine.

DMX is a proxy X server that extends multi-head support for displays attached to different machines.

When used with Ximerama the displays are presented to the user as a single screen. DMX has been

integrated into the X.org server software.

ParaView

ParaView is an open-source, multi-platform data analysis and visualization application. ParaView users

can quickly build visualizations to analyze their data using qualitative and quantitative techniques. The

data exploration can be done interactively in 3D or programmatically using ParaView's batch processing

capabilities.

ParaView was developed to analyze extremely large datasets using distributed memory computing

resources. It can be run on supercomputers to analyze datasets of terascale as well as on laptops for

smaller data. (Paraview - Open-source Scientific Visualization)

ParaView is currently maintained and is available for Linux and Windows.

VR Juggler

VR Juggler can be used on complex multi-screen systems running on clusters, high-end workstations,

and supercomputers. VR Juggler allows applications to execute in many VR system configurations,

including desktop VR, HMD, CAVE™-like devices, and Powerwall™-like devices. VR Juggler 2.0

supports IRIX, Linux, Windows, FreeBSD, Solaris, and Mac OS X. (The VR Juggler Suite website)

11 Visualization Clusters using Commodity Hardware

Commercial Visualization Applications

Wolfram Research’s gridMathematica

gridMathematica adds extra computation kernels and automated network-distribution tools to Wolfram

Research’s Mathematica. gridMathematica runs tasks in parallel, over multiple CPUs with fully

automated process coordination and management and no need for code changes. Mathematica supports

multiple Windows, Macintosh, Linux, and Solaris platforms.

gridMathematica Server provides 64-bit optimizations, including integration with many cluster

management systems such as Altair PBS Professional, Microsoft Windows Compute Cluster Server,

Microsoft HPC Server, Platform LSF, and Sun Grid Engine. gridMathematica Server also provides

MathLink's support for both Gigabit and high-speed data networks. (Wolfram gridMathematica website)

CEI’s EnSight

EnSight provides an icon-based interface that allows the user to learn the program quickly and move

easily into layers of increased functionality. EnSight works on all major computing platforms and supports

interfaces to most major CAE programs and data formats.

EnSight can be run standalone or distributed using client-server operation on Windows, Apple, Linux,

SGI, SUN, IBM, and HP platforms. It runs in parallel on up to two processors on shared-memory

computer systems.

EnSight reads multiple data sets and enables the user to run comparisons between them. The data can

originate from different solvers and/or different disciplines. EnSight post-processes data remotely with

client-server operation. (CEI EnSight website)

VCOVISE

COVISE is a modular visualization software program that supports Virtual Reality and collaborative

networking capabilities. It is a system platform for constant use of Virtual and Augmented Reality

technologies.

The modular structure for add-ons to the basic COVISE module ensures successful customization for

suite-specific needs. Modules are available for fluid dynamics and structural mechanics readers, virtual

reality rendering, collaborative engineering, volume rendering, web browser view rendering, batch

processing, viewing VRML files, development environment, casting and molding simulation, multi-body

systems, QA and tolerance analysis, and CAD file visualization.

Open Source Modeling

VMD

VMD is designed for modeling, visualization, and analysis of biological systems such as proteins, nucleic

acids, lipid bilayer assemblies, etc. It may be used to view more general molecules, as VMD can read

standard Protein Data Bank (PDB) files and display the contained structure. VMD provides a wide variety

of methods for rendering and coloring a molecule: simple points and lines, CPK spheres and cylinders,

licorice bonds, backbone tubes and ribbons, cartoon drawings, and others. VMD can be used to animate

and analyze the trajectory of a molecular dynamics (MD) simulation. In particular, VMD can act as a

graphical front end for an external MD program by displaying and animating a molecule undergoing

simulation on a remote computer. http://www.ks.uiuc.edu/Research/vmd/

12 Visualization Clusters using Commodity Hardware

Collaboration

One of the most exciting uses of a viz wall is for collaboration between groups of users. Each can display

graphics on the screen simultaneously to illustrate differing views of the data, compare the results of

different calculations, or create synergistic displays of data from different disciplines. This is even more

exciting when the participants are geographically dispersed, using part of the wall for teleconferencing

while using other areas on the wall for graphics and demonstrations.

Two of the most popular collaboration environments are Access Grid and OptlPortal, both open source.

Access Grid

The Access Grid® includes multimedia large-format displays, presentation and interactive environments,

and interfaces to Grid middleware and visualization environments. These resources are used to support

group-to-group interactions across the Grid.

AG3 is the newest version of the Access Grid software. It has been updated to conform to standard

Internet technologies and protocols and to maximize robustness, performance, and interoperability. All

network connections are encrypted by default using an X509 certificate at the venue server. Users do not

require a certificate. Access Grid is supported on Windows, Apple OSX, and a number of Linux

platforms. (Access Grid website)

OptIPortal

An OptIPortal is a visualization cluster that can be deployed on a variety of hardware platforms.

Functionally, one or more OptIPortals can be used to view high definition static images, video, or in

streaming mode in both 2D and 3D environments. 3D OptIPortals in use currently use COVISE and

CGLX as middleware. 2D OptIPortals in use currently run CGLX or SAGE.

Actions on the OptIPortals can be concurrent or persistent. Clusters of OptIPortal computers (nodes) can

run general computations as well as display and collaboration management. OptIPortal supports SAGE,

DMX, Chromium, CGLX, and OpenCOVER (Covise) as middleware. (OptIPortal website)

13 Visualization Clusters using Commodity Hardware

About X-ISS

eXcellence in IS Solutions, or X-ISS, is a professional services firm located in Houston, Texas, that

specializes in delivering high-performance computing solutions. X-ISS expertise includes designing,

deploying, managing, and benchmarking clusters of all types and sizes, including various interconnects,

cluster management suites, distributed resource managers, and popular HPC applications.

X-ISS manages and designs HPC Linux and Windows Compute Cluster Server based-clusters for oil and

gas, government, education, life sciences, and numerous other vertical markets. X-ISS is vendor-agnostic

and works with any hardware vendor or independent software vendor (ISV) to deliver cluster solutions.

X-ISS can help lower total cost of ownership, reduce administrative costs, optimize performance of both

hardware and applications, provide independent benchmarks of ISV applications, and manage HPC

clusters from small work group clusters through 12,000 core high throughput compute facilities.

If you have any questions regarding this white paper, please contact hpc@x-iss.com.