In the November/December 2000 issue of IEEEComputer Graphics and Applications,1 Vicki Interrante

posed a visualization problem she and I have been inter-ested in for several years. The problem is that of visuallyrepresenting a 2D field of data that has multiple datavalues at each point. For example, 2D fluid flow has avector value at each location and derived values areoften available at each location. Interrante suggestsusing natural textures to attack this problem, becausethe textures can potentially encode lots of information.She provides some intriguing examples and proposesa psychology-based approach for developing an under-standing of how we perceive natural textures, like thoseBrodach photographed.2 Understanding this helps usbuild better visualizations.

Based on Interrante’s suggestions, I would like to positand explore what is, perhaps, a less well-definedapproach. Through evolution, the human visual systemhas developed the ability to process natural textures.However, in addition to natural textures, humans alsovisually process man-made textures—some of the rich-est and most compelling of which are in works of art.Art goes beyond what perceptual psychologists under-stand about visual perception and there remain funda-mental lessons that we can learn from art and art historyand apply to our visualization problems.

The rest of this article describes and illustrates someof the visualization lessons we have learned studyingart. I believe that these examples also illustrate some ofthe potential benefits of further study. While thisapproach is more open-ended than a perceptual psy-chology approach, both approaches are worthy of pur-suit, and the potential benefits of using the lessstructured approach outweigh any risk of failure.

How humans see and understandScientific visualization, a term coined only a little over

10 years ago, is the process of using the human visualsystem to increase our understanding of phenomenastudied in various scientific disciplines. While the termis young, the process (modulo the computer) has beenused since the beginning of science. Many scientists havecreated drawings or built 3D models to understand andcommunicate their science. The history of science andart can provide us with lessons for using computerseffectively. Over time, artists have developed techniquesto create visual representations in particular communi-cation goals. Art history provides a language for under-

standing and communicating that knowledge.Historically, two disciplines approach the human visu-

al system from different perspectives. Art history pro-vides a phenomenological view of art—painting Xevokes response Y. Art history, however, doesn’t decon-struct the perceptual and cognitive processes underlyingresponses. Perceptual psychology, on the other hand,strives to explain how humans understand those visualrepresentations. There’s a gap between art and percep-tual psychology—we don’t know how humans combinevisual inputs to arrive at the responses art evokes. Shape,shading, edges, color, texture, motion, and interactionare all components of an interactive visualization. Buthow do these components interact and how can theymost effectively be deployed for a particular scientifictask? Answers to these questions are likely to fill some ofthe gap between art and perceptual psychology. As anexample, the human-computer interaction (HCI) com-munity is using and extending knowledge about per-ception to test and develop better user interfaces. Wecan find analogous inspiration for improved methodsfor scientific visualization in the gap between art andperceptual psychology. Many of these lessons willimpact the visual representation of multivalued data.

Looking up from our monitors A number of times over the last few years I’ve shep-

herded my students to art museums for guided tours bymy artist collaborator davidkremers, the CaltechDistinguished Conceptual Artist in Biology. After ini-tially searching for scientific visualization inspiration inart, these visits let us formulate a plan for finding andapplying the concepts. Our initial focus was on oil paint-ing, particularly from the Impressionist period, becausethese paintings are so visually rich. The multiple layersof brush strokes in these paintings provide a naturalmetaphor for constructing visualizations from layers ofsynthetic “brush strokes.” Some of my colleagues look atme askance when I describe our research field trips, asif to say, “This is research?” But stepping out of the labhelps students build a new picture of what they canaccomplish when they come back to the computer. Ittrains their eyes and minds to see differently.

During these field trips, we studied, in particular, theworks of three painters:

■ Van Gogh, whose large, expressive, discrete strokescarry meaning both individually and collectively.

David H.Laidlaw

BrownUniversity

0272-1716/01/$10.00 © 2001 IEEE

Loose, Artistic “Textures” for Visualization ____________

Visualization ViewpointsEditors: Theresa-Marie Rhyne andLloyd Treinish

6 March/April 2001

■ Monet, whose smaller strokes are often meaninglessin isolation—the relationships among the strokes givethem meaning, far more than in van Gogh.

■ Cezanne, who combined strokes into cubist facets,playing with 3D perspective and time within his paint-ings more then either van Gogh or Monat. His layeringalso incorporates more atmospheric effects. In a sense,his work shifts from surface rendering toward volumerendering.

The three artists’ work in this sequence builds in com-plexity and subtlety. In our field trips, we studied allthree, but most of our experiments thus far are limitedto ideas we learned from van Gogh’s work.

Van Gogh introduced us to the concept of under-painting, or laying down a rough value sketch of theentire painting. The underpainting shows through theoverlying detailed brush strokes to define the anatomyof the painting. Figure 1b shows underpainting forFigure 1a. It divides the canvas into two parts—a primedlower region of hillside, rocks, and ground cover and adarker upper region of tree, sky, and distant hills.Underpainting helped us present some overall parts ofour data. We found that an analogous underlying formin our visualizations anchors and literally gives shapeto disparate data components. Outlines around regionsprovide separation and emphasis, lending definition toour sea of data.

In van Gogh’s The Mulberry Tree (1889, oil on canvas),brush strokes represent the solid trunk of the tree, bend-ing branches, leaves blowing in the wind, and tufts of grass(Figure 1a). We learned many shorthand ways of depict-ing complexity using icons, geometric shapes, or texturesthat evoke a characteristic of the subject, or the data—andwith that comes the responsibility of choosing brushstrokes that don’t create opposing or unwanted secondaryimpressions. Beyond this direct representation, they alsoinvite the viewer to experience the scene, not just view itpassively. Similarly, brush-stroke size and proximity depict

density, weight, and velocity. In our visualizations, wewant to capture this marriage between direct representa-tion of independent data and the overall intuitive feelingof the data as a whole.

Back in the lab Returning to our computer lab, we tried to use some

of the ideas we had gleaned, once again drawing most-ly from van Gogh’s work. We experimented with brushstroke shapes and ways of layering them. Our initialattempts were free–form and produced interestingresults. Our next attempts were more directly appliedto scientific problems. We show two of the images wegenerated in Figures 2 and 3 (next page). The problem-directed approach led us to iconic-looking strokes.

In Figure 2, we show one 2D slice of a 3D second-

IEEE Computer Graphics and Applications 7

1 (a) Van Gogh’s The Mulberry Tree (1889, oil on canvas) illustrates the visual shorthand that van Gogh used withhis expressive strokes. Multiple layers of strokes combine to define regions of different ground cover, aspects ofthe hillside, and features of the tree. (b) An underpainting shows the “anatomy” or composition of the scene inbroad strokes. (Image of The Mulberry Tree granted by the Norton Simon Foundation, Pasadena, California. Gift ofNorton Simon, 1976.)

(a) (b)

2 Visualization of half of a section through a mouse spinal cord. The data isa symmetric 3D second-order tensor field, with the equivalent of six inde-pendent scalar values at each point. The detail on the right shows thelower right part of the section.

order tensor field, which has about six different data val-ues at each point in the image. The image shows theright half of a section through a mouse spinal cord. Tocreate the visualization, we used a layer resemblingunderpainting with ellipse-shaped strokes on top of it.On each of the strokes, a texture represents more of thedata. For more details on the scientific interpretationand the visualization, see Laidlaw et al.3

In Figure 3, we show 2D fluid velocity together witha number of derived quantities. About nine values arerepresented at each spatial location in this visualization.We again used a layer resembling underpainting withlayers of ellipse, wedge, and box strokes on top. Theellipse strokes have a subtle texture superimposed. Moredetails on the visualization appear in Kirby et al.4

SpaceWe learned that paintings (and, in some cases, visu-

alizations) are multiscale. They can be viewed from dif-ferent distances and seen and understood differently.This raises interesting issues about the definition of tex-ture. Let’s consider van Gogh’s Mulberry Tree (Figure 1a).

From a few inches away (look closely at Figure 4b), youcan see shapes from the bristles of the brush as well ascolors mixed within a stroke. At this distance, the shapeand color features might be considered texture, but theycould also be interpreted individually. At a distance of18 inches (Figure 4b at a normal reading distance of 18inches), these features appear smaller and resemble atexture on each stroke. The strokes themselves are stillindividual. At a distance of five feet (Figure 4a), thestrokes merge together to appear more like a texture.Finally, at 15 feet (Figure 1a), the strokes blend togeth-er and become almost invisible individually.

We can use this lesson by encoding different informa-tion at different scales. Iconic information at one scalecan turn into texture information at another scale. Withcare, we can design features at different scales into the

same images. In the scientific visual-izations of Figures 2 and 3, we designvisual features at different scales. Thetexture on strokes is at a much finerscale than the strokes themselves,and the dark box strokes of Figure 3are at a different scale than the otherstrokes.

To take full advantage of the mul-tiscale nature of paintings and visu-alizations we have to have ways ofinteracting with them—that is, waysof changing our viewing distance.We rarely change the distance fromwhich we view our monitors—onlya bit more frequently do we do sowith paper publications. That’s whythe same image is shown at differ-ent scales in Figure 4 and in some ofthe other detailed figures. However,we do view images hung on a wallfrom different distances. And someimages on paper—often artistic—inspire that sort of study. Projection

systems like PowerWalls and CAVEs may be goodoptions for encouraging this sort of exploration, as mayother hangable large-format output media.

TimeWe also learned that paintings (and, in some cases,

visualizations) have a temporal component. Forinstance, we see different aspects of an image at differ-ent viewing times. Some parts stand out quickly, like theoverall composition or palette of a painting, and sometake more time to become apparent, like the texture orshape of individual strokes. The scale and speed ofrecognition correlate, as do contrast and speed of recog-nition—but these are not the only factors.

To use this lesson, we can design our visualizations sothat important data features are mapped to quickly seenvisual features. For example, features we want to mea-sure directly from an image are present for detailed studybut don’t intrude on the visualization’s initial impres-sion. The multiscale examples from the “Space” sectionillustrate this temporal concept. Figure 3 gives anotherexample: we can read the wedges more quickly than the

Visualization Viewpoints

8 March/April 2001

3 Visualizationof 2D fluidvelocity togeth-er with severalderived datavalues.Approximatelynine values arerepresentedvisually at eachpoint in theimage.

4 Variances in viewing van Gogh’s Mulberry Tree. Viewed in this article fromabout 18 inches, Figure 1a shows what you would see 15 feet from thepainting. Comparatively, Figure 4 shows the following: (a) a detail of whatyou would see 5 feet from the painting, and (b) a detail at actual size (whatyou would see from 18 inches). Look at (b) more closely for viewing dis-tances less than 18 inches.

(a) (b)

ellipses because of a difference in contrast.Studies of preattentive vision and knowledge about

low-level vision are useful for designing quickly seen visu-alization parts. It’s more difficult to test the more slowlyseen parts, which makes it more difficult to design them.Task-oriented experimental tests seem logical, but thetasks are often so complex that the performance varianceis relatively high, making methods difficult to compare.

Our initial experimentsOur initial experiments were much looser than the

examples shown in Figures 2 and 3. Some examples inFigure 5 show 2D or 3D fluid flow. Since I want toemphasize the overall texture and visual qualities, Iwon’t go into detail about the mappings for each. Tomany, the images are visually compelling, yet it has beendifficult to extract concrete visualization lessons fromthem beyond those I previously described. What peoplesee in these images includes not only the mappings thatwere used for the data value, but also other visual char-acteristics. Despite being 2D, some images give an over-all sense of depth. Some of the strokes appear to layer,like feathers or scales. One of our challenges with theselooser images is in understanding what works, whatdoesn’t, and (we hope) why.

Closing thoughtsI’ve tried to illustrate some examples of looking toward

art for inspiration in creating visualizations. Here we fea-ture van Gogh and mention Monet and Cezanne for con-text. In your artistic searches, choose the artists in whomyou have a passionate interest. I believe that any artisthas lessons to offer to visualization.

Working on scientific visualization problems, wealready interact with scientists and adopt their prob-lems. As toolsmiths, we do better computer sciencethrough addressing scientists’ problems on scientists’terms.5 Similarly, we benefit from critical feedback fromartists, despite the difficulty of creating and maintainingthese relationships. I try to look at and understand art—early and often—and emulate it in scientific visualiza-tions and get critical feedback from artists. I explainwhat I’m trying to do visually and have artists critique it.Then I iterate, iterate, and iterate.

Of course, scientists must be involved in this iterativeprocess. Artists can help with inspiration and feedbackon the visual and communicative aspects of visualiza-tion, but scientists define the tasks performed and there-fore must ultimately evaluate the success of the methods.For instance, the fluid flow example in Figure 3 may beaesthetically pleasing, but without explanation—per-haps via a legend or key—it’s not scientifically useful.

Figure 3 displays as many as nine values at each pointof the image. With some research indicating that tex-ture has roughly three independent dimensions, theability to represent nine values is somewhat surpris-ing—perhaps it’s due to combining color with textureor layering textures at different scales.

Texture is hard to define. Understanding black andwhite natural textures like the photographs in Brodatz2

is a good start, but we also need to look broadly. Task-ori-ented user testing may help, and perhaps we can use thecritiques that are part of artists’ training. This mightcombine perceptual psychology and art to fill in part ofthe gap in our understanding of how humans see. Byhaving artists cognitively analyze what is shown by morecomplex textures, we might come to a consensus onwhat works, what doesn’t work, and why it does or does-n’t work in the context of art and art history. ■

AcknowledgmentsI’d like to thank my collaborators—in particular, artist

davidkremers—as well as Clifford Elgin, Mike Kirby, andMatthew Avalos. I also want to thank Victoria Interranteand Barbara Meier for their encouragement and ideas.The NSF (CCR-9619649, CCR-0086065) and the HumanBrain Project (NIMH and NIDA) supported some of thework described.

References1. V. Interrante, “Harnessing Natural Textures for Multivari-

ate Visualization,” IEEE Computer Graphics and Applica-tions, vol. 20, no. 6, Nov./Dec. 2000, pp. 6-10.

2. P. Brodatz, Textures: A Photographic Album for Artists andDesigners, Dover, New York, 1966.

3. D.H. Laidlaw et al., “Visualizing Diffusion Tensor Images ofthe Mouse Spinal Cord,” Proc. Visualization 98, IEEE Com-puter Soc. Press, Los Alamitos, Calif., 1998, pp. 127-134.

4. R.M. Kirby, H. Marmanis, and D.H. Laidlaw, “VisualizingMultivalued Data from 2D Incompressible Flows UsingConcepts from Painting,” Proc. Visualization 99, IEEE Com-puter Soc. Press, Los Alamitos, Calif., 1999, pp. 333-340.

5. F.P. Brooks, “The Computer Scientist as Toolsmith II,”Comm. ACM, vol. 39, no. 3, Mar. 1996, pp. 61-68.

Readers may contact Laidlaw at the Department ofComputer Science, P.O. Box 1910, Brown University, Prov-idence, RI 02912, email dhl@cs.brown.edu.

Readers may contact department editors Rhyne andTreinish by email at rhyne@siggraph.org and lloydt@us.ibm.com.

IEEE Computer Graphics and Applications 9

5 Loose texture examples.