BY

DA

VID

KO

LL

ER

AN

DM

AR

CL

EV

OY

74 June 2005/Vol. 48, No. 6 COMMUNICATIONS OF THE ACM

The digital rights management problemof protecting data from theft and misuse has been addressed for manyinformation types, including softwarecode, digital images, and audio files.Few technological solutions are designedspecifically to protect interactive 3Dgraphics content.

Demand for ways to protect 3D graphical models is significant and growing. Contemporary 3D digitizationtechnologies allow the efficient creationof accurate 3D models of many physicalobjects. For example, our Stanford Digital Michelangelo Project [3] hasdeveloped a high-resolution digital

To prevent the theft of 3D models, these methods defend the high-resolution geometric detail of their physical shape, while still allowing interactive display and manipulation.

Protecting 3DGraphics Content

COMMUNICATIONS OF THE ACM June 2005/Vol. 48, No. 6 75

t

CO M P U T E R R E N D E R I N G O FMI C H E L A N G E L O’S DAV I D M A D E F R O M A

L A S E R-S C A N N E D 3D M O D E L O F T H ES TAT U E C O N TA I N I N G E I G H T M I L L I O N

P O LY G O N S, E A C H 2.0 M M I N S I Z E. TH E M A R B L E V E I N I N G A N D

R E F L E C TA N C E A R E A RT I F I C I A L. (STA N F O R D DI G I TA L MI C H E L A N G E L O

PR O J E C T, R E N D E R I N G B YHE N R I K WA N N JE N S E N)

76 June 2005/Vol. 48, No. 6 COMMUNICATIONS OF THE ACM

archive of 10 of Michelangelo’s large statues, includ-ing the David (see the sidebar “Generating a Replicaof Michelangelo’s David”). These statues representthe artistic patrimony of Italy’s cultural institutions,and our contract with the Italian authorities permitsdistribution of the 3D models only to establishedscholars for noncommercial use. Though everyoneinvolved would like the models to be available for anyconstructive purpose, the digital 3D model of theDavid would quickly be pirated if it were distributedwithout protection; simulated marble replicas wouldbe manufactured outside the provisions of the partiesauthorizing creation of the model.

Digital archives of archaeo-logical artifacts are anotherexample of cultural her-itage 3D models that couldrequire piracy protection.Curators of such artifactcollections increasingly

turn to 3D digitization as a way to preserve and widenscholarly use of their holdings, but they often wantstrict control over the manner of that use of the 3Ddata and to guard against theft. An example of such acollection is our Stanford Digital Forma Urbis Project(formaurbis.stanford.edu) we’ve undertaken withItalian archaeological officials to digitize more than athousand marble fragments of an ancient Romanmap and make them publically available through aWeb-based database—provided the 3D models haveadequate protection.

Other application areas (such as entertainment andonline commerce) could also require protection for3D graphics content. Valuable 3D animated charac-ter models developed for use in motion pictures and3D body scans of high-profile actors may be repur-posed for widespread use in video games and promo-tional materials. Content developers might bereluctant to distribute the 3D models in interactiveapplications without control over piracy and reuse. Anumber of Internet application developers havereported that their clients are unwilling to pursueonline 3D graphics projects due to the inability toprevent theft of 3D content.

Prior technical research in intellectual propertyprotection for 3D data has concentrated on 3D digi-tal watermarking techniques. These steganographicapproaches have sought to embed hidden informa-tion into 3D graphical models, with varying degreesof robustness to attacks aimed at disabling the water-marks by altering 3D shape or data representation.Many of the most successful 3D watermarkingschemes are based on spread-spectrum frequency

domain transformations, embedding watermarks atmultiple scales by introducing controlled perturba-tions into the coordinates of the 3D model vertices[4]. Complementary technologies search collectionsof 3D models, examining them for the presence ofdigital watermarks in an effort to detect piracy.

For the digital representations of valuable 3Dobjects (such as cultural heritage artifacts), it is notsufficient to detect piracy after the fact; piracy mustbe prevented. The computing industry has experi-mented with a number of techniques for preventingunauthorized use of digital data, including physicaldongles, software access keys, node-locked licensingschemes, copy-prevention software, obfuscation, andencryption with embedded keys. Most are either bro-ken or bypassed by determined attackers, causingundue inconvenience and expense for nonmalicioususers. High-profile data and software are particularlysusceptible to attackers.

Fortunately, 3D graphics data differs from mostother forms of digital media in that the presentationformat—2D images—is fundamentally differentfrom the underlying representation—3D geometry.3D graphics data is usually displayed as a projectiononto a 2D display device, resulting in a large infor-mation loss for single views. This property supportsan optimistic view that protected 3D graphics systemscan still be useful to users, without making the 3Ddata as vulnerable to piracy as other types of digitalcontent.

Here, we address the problem of preventing thetheft of 3D models, while still allowing for their inter-active display and manipulation. Our goal is to pro-vide a solution for maintainers of large collections ofhigh-resolution static 3D models (such as the culturalheritage artifacts we are digitizing). The methods weare developing aim to protect both the physical shapeof the 3D models and their particular geometric rep-resentation (such as 3D mesh vertex coordinates, sur-face normals, and connectivity information). Weaccept that the coarse shape of visible objects is easilyreproduced regardless of protection efforts, so we con-centrate on defending the high-resolution geometricdetail of 3D models. This detailed geometry is usuallythe most expensive to model or measure (perhapsrequiring special access and advanced 3D digitizingtechnology) and is often the most valuable in exhibit-ing fidelity to the original object.

PROTECTION TECHNIQUESFigure 1 outlines an abstraction of the 3D graphicspipeline, identifying some of the methods anattacker might use to attempt to recover 3D geome-try data in a computer graphics system. To counter

COMMUNICATIONS OF THE ACM June 2005/Vol. 48, No. 6 77

Demonstrating recent improvements in digitizing theshape of physical objects, a team of 30 faculty, staff, and stu-dents from Stanford University and the University of Washing-ton spent the 1998–99 academic year digitizing the sculpturesand architecture of Michelangelo, including the well-knownsculpture of David. Our main scanning device was a laser tri-angulation rangefinder mounted on a motorized gantry (imagea in the figure), using it to digitize the marble statues with aspatial resolution of 0.25mm, dense enough to captureMichelangelo’s chisel marks. After scanning, our range data-processing pipeline aligned the scans taken from differentgantry positions, combined them into a unified surface mesh,and automatically filled any small holes that could not be seenby the scanner [2]. Scanning the David took three weeks, with400 individually aimed scans resulting in two billion polygonsof data. Reduced-resolution versions of the resulting 3D modelhave been used to make computer renderings (b) and for avariety of scientific and scholarly studies.

A simplified 1.25-million polygon model of the David wasthe basis for replicas we have manufactured in collaborationwith Gentle Giant Studios (www.gentlegiantstudios. com/).We used a thermojet wax printer to make a master (c), follow-ing with a latex molding and casting procedure. The final repli-

cas (d), standing 15 inches tall, are made of plastic resin,though other materials (such as marble dust in a binder) canbe used.

Though the economic value of digital representations of 2Dartwork is uncertain due to the proliferation of photographicreplicas (many unlicensed), the situation for 3D artwork maybe different. Until recently, few famous sculptures had beendigitized, and of those that had been scanned, even fewer havebeen used to manufacture replicas for retail sale. Althoughreplicas of famous statues like the David abound, they are usu-ally based on handmade models and are of relatively poorfidelity (e). Thus, the potential economic value of digital repre-sentations of 3D artifacts is great, as evidenced by the vigor-ous market for replicas of statuary conducted throughmail-order catalogs.

Our replica is not for sale to the public, though eventually itwill be. Meanwhile, our 3D models can be studied using theScanView protected rendering system described here. The par-ticular model of David embedded in ScanView is the same oneused to generate the replica shown in the figure, with theexception of some small, inconspicuous modifications wemade to each model, effectively watermarking them to identifyillegal copies. c

Generating a Replica of Michelangelo’s David

Producing a replica of Michelangelo’s David: (a) laser scanner positioned in front of the statue in the Galleria dell’Accademia inFlorence; (b) computer rendering of 3D model; (c) replica master under construction (Gentle Giant Studios); (d) replica producedfrom scanned data; and (e) inexpensive replica ($100) purchased from a street vendor in Florence.

these attacks, we haveconsidered several possi-ble approaches for shar-ing and renderingprotected 3D graphics.One involves bypassingthe graphics processingunit (GPU) driver andhardware, and using soft-ware-only graphics ren-dering for at least aportion of the data. Soft-ware rendering keeps con-trol of the renderingprocess in the hands of theviewing application pro-grammer, allowing for spe-cialized data encryption orobfuscation techniques tobe used to protect the datain the early stages of the pipeline, trading off display performance.Another involves theintroduction by the view-ing application of subtledeformations in the geom-etry of the model beforepassing the 3D vertex datato the graphics driver;attackers would have diffi-culty reconstructing thefull 3D model due to thedistortions.

The drawback of suchprotection techniques isthat they all eventuallyrely on “security throughobfuscation,” which isunsound from a com-puter-security point ofview. Any 3D graphics-protection technique thatmakes the actual 3D data available to potentialattackers in software can be broken [5], as anyattacker with enough time and resources will be ableto reverse engineer the protections on the data. It ispossible that future “trusted computing” platformsfor general-purpose computers will make softwaretampering difficult or impossible, but few such sys-tems are deployed today.

Other approaches to protecting 3D graphicsinclude hardware GPU decryption and image-basedrendering. If the 3D models were encrypted using

public-key encryptionwhen they are created,then custom GPUs couldaccept encrypted dataand perform on-chipdecryption and render-ing. This technologywould provide robustprotection for the 3Ddata, but such GPUs donot exist today, and itwill be years before it is

widely available in personal computers. Image-basedgraphics data representations (such as light fields [2])are densely sampled data structures that do not explic-itly include a geometric description for 3D shape yetare still amenable to interactive and accurate display.However, distributing 3D models as image-basedlight fields at the high sampling resolutions requiredwould involve huge, unwieldy file sizes and not allowfor geometric operations (such as surface measure-ments performed by archaeologists) on the data.

A final scheme for securing 3D graphics is to retainthe 3D model data on a secure server—controlled bythe content owner—and pass only 2D-rendered imagesof the models back to user-client requests. The 3Dgeometry is thus safe from all types of graphics pipelineattacks (except reconstruction from images), though theserver itself is still vulnerable to direct attack.

REMOTE RENDERING TO PROTECT 3D MODELSWe have implemented such a remote rendering sys-tem with a client-server architecture to provide con-trolled, protected access to collections of 3Dgraphics models (see Figure 2). Users employ a spe-cial 3D client viewer program to interactively viewthe protected 3D content. The program includeslow-resolution, decimated versions of the 3D mod-els that can be interactively rotated, zoomed, andilluminated by the user in real time. When the userstops manipulating a low-resolution model, detectedby a “mouse up” event, the client program queriesthe remote rendering server via the network for amatching image rendered from the high-resolutionmodel data, replacing the low-resolution renderingseen by the user (see Figure 3). On computer net-works with reasonably low latencies, the user has theimpression of manipulating a high-resolution ver-sion of the model. In typical use involving culturalheritage artifacts, we use models with approximately10,000 polygons for the low-resolution version,whereas the server-side models often contain tens ofmillions of polygons. Low-resolution model com-plexities are of little value to potential thieves yet still

78 June 2005/Vol. 48, No. 6 COMMUNICATIONS OF THE ACM

Koller new fig (6/05)

(a) 3D Data File

(e) DisplayDevice Video

Signal

GPU

(b) 3D ViewerApplicationProgram

(c) GraphicsDriver

(d) FrameBuffer

Rasterization

VertexTransform

and Lighting

Figure 1. Abstracted graphicspipeline showing possibleattack locations (a–e) for

recovering 3D graphics datafrom an interactive viewingprogram: (a) 3D model filereverse engineering. Users

with full access to 3D modeldata files can reverse engineer

obfuscated or encrypted formats; (b) 3D viewer

application tampering. Hackersuse program tracing, memory

dumping, and other techniques to obtain access to

data being used by application programs;

(c) Graphics driver tampering.3D data passes through

graphics driver software on itsway to the graphics hardware

where the drivers are vulnerable to tampering or

replacement by attackers trying to capture streams of3D data; (d) Reconstruction

from the frame buffer. Sophisticated attackers can

access rendered images fromthe graphics memory, using

3D computer vision techniquesto reconstruct the original

model; and (e) Reconstructionfrom the final image display.

Irregardless of any systemprotections in the pipeline, thefinal video images output from

a graphics system are vulnerable to capture and

reconstruction.

provide enough clues for the user to navigate andmanipulate.

The remote rendering server receives renderingrequests from users’ client programs, renders corre-sponding images, and passes them back to the clients.It is implemented as a module running under theApache 2.0 HTTP server, communicating with clientprograms using the standard HTTP protocol and tak-ing advantage of the access-protection and monitor-ing tools built into the Web server software. As renderrequests are received from clients, the server checkstheir validity and dispatches valid requests to a GPUfor OpenGL hardware-accelerated rendering. Therendered images are read back from the frame buffer,compressed using JPEG compression, and returned to

the client. The server uses level-of-detail techniques tospeed the rendering of highly complex models andmaintain high throughput rates. In practice, an indi-vidual server node with, say, Pentium 4 CPU andNVIDIA GeForce 4 video card can handle a maxi-mum of eight typical client requests per second; thebottlenecks are in the rendering and readback stage(about 100 milliseconds) and in the JPEG compres-sion step (about 25 milliseconds). Incoming requestsizes are about 700B each; the images returned fromour servers average 30kB per request.

The primary benefit of using a remote image-ren-dering system to share 3D models is that the high-res-olution model geometry data is never made availableto potential attackers. Only 3D reconstruction from2D images remains as a possible attack from thoseoutlined in Figure 1, but general 3D reconstructionfrom images is a challenging computer vision researchproblem. However, synthetic graphics renderings canbe particularly susceptible to reconstruction, as thehuman cost of harvesting large numbers of images islow, and the attacker may be able to specify the para-

meters used to create the images. Moreover, syntheticimages are potentially perfect, with no sensor noise ormiscalibration errors.

To combat such reconstruction attacks, we’veimplemented a number of defenses in our renderingserver system. To deter image-harvesting attacks, weperform automatic analysis of the server logs, detect-ing suspicious sequences or frequencies of imagerequests. We employ obfuscation to create hurdles forattackers by encrypting the rendering request mes-sages sent from the client programs, as well as byencrypting the low-resolution client-side 3D models.The server imposes constraints on rendering requests,disallowing extremely close-up views of models andrequiring a fixed field of view.

Finally, we employa number of pertur-bations and distor-tions to the imagesreturned from theserver. They are gen-erally applied in apseudorandomlygenerated fashion, sotheir effects are noteasily modeled andreversed, and theirmagnitude is limitedso as not to distractnonmalicious usersviewing the models.

Examples of such distortionsinclude nonlinear image warps,adding high-frequency noise toimages, and perturbing the lighting

parameters slightly from those being requested. We have experimentally validated the effectiveness

of these defenses against a variety of traditional com-puter vision reconstruction techniques [1]. However,we know of no formalism for rigorously analyzing thesecurity provided by our systems-based approach.One inevitably falls into an “arms race” betweenattacks and countermeasures (such as the ones we’veimplemented).

RESULTS AND FUTURE WORKThe protected-graphics software we developed—ScanView, available at graphics.stanford.edu/soft-ware/scanview/—has been used to share 3D modelsfrom the Digital Michelangelo Project and other col-lections of cultural heritage artifacts. More than10,000 users have installed the client software ontheir computers and accessed the remote servers toview the 3D models. Art students, sculptors, and

COMMUNICATIONS OF THE ACM June 2005/Vol. 48, No. 6 79

Figure 2. Remote rendering system.

enthusiasts have examined thehigh-resolution artwork, whilearchaeologists have studied indi-vidual artifacts. Few of themwould have qualified under the

strict guidelines required to obtain unrestricted accessto the models, so the protected remote rendering sys-tem has made it possible to grant whole new cate-gories of users access to 3D graphical models for bothprofessional scholarship and personal enjoyment.

User comment is uniformly positive. Fetchinghigh-resolution renderings over intercontinentalbroadband Internet connections involves less thantwo seconds of latency, while fast continental connec-tions generally experience latencies dominated by theprocessing time of the rendering server. Moreover, therender server architecture scales up to support an arbi-

trary number of requests per second, andservers can be installed at distributed loca-tions around the world to reduce long-dis-tance latencies.

One direction for further research isanalysis of computer vision techniques thatspecifically address 3D reconstruction ofsynthetic data under antagonistic condi-tions—to increase our understanding ofthe efficacy of such attacks and the corre-sponding render server defenses. Anotherissue is how to grant users a greater degreeof geometric analysis of protected 3D mod-els without further exposing the data totheft. Scholarly users have expressed inter-est in measuring distances and plotting pro-files of 3D objects for analytical purposesbeyond the simple 3D viewing supportedin the current system. Finally, there is gen-eral interest in alternative approaches toprotecting 3D graphics, including special-ized systems that make data security a pri-ority while sacrificing general-purposecomputing platform capabilities. A GPU-decryption scheme might, for example, beappropriate for console devices and othercustom graphics systems.

References1. Koller, D., Turitzin, M., Levoy, M., Tarini, M., Crocia,

G., Cignoni, P., and Scopigno, R. Protected interactive3D graphics via remote rendering. ACM Transact. Graph-ics 23, 3 (Aug. 2004), 695–703.

2. Levoy, M. and Hanrahan, P. Light field rendering. In Pro-ceedings of ACM SIGGRAPH 1996 (New Orleans, Aug.4–9). ACM Press, New York, 1996, 31–42.

3. Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller,D., Pereira, L., Ginzton, M., Anderson, S., Davis, J.,Ginsberg, J., Shade, J., and Fulk, D. The DigitalMichelangelo Project. In Proceedings of ACM SIGGRAPH2000 (New Orleans, July 23–28). ACM Press, New York,2000, 131–144; graphics.stanford.edu/projects/mich/.

4. Praun, E., Hoppe, H., and Finkelstein, A. Robust mesh watermarking.In Proceedings of ACM SIGGRAPH 1999 (Los Angeles, Aug. 8–13).ACM Press, New York, 1999, 49–56.

5. Schneier, B. The fallacy of trusted client software. Information Security(Aug. 2000).

David Koller (dk@cs.stanford.edu) is a Ph.D. student in theComputer Science Department at Stanford University, Stanford, CA.Marc Levoy (levoy@cs.stanford.edu) is an associate professor ofcomputer science and (jointly) electrical engineering at Stanford University, Stanford, CA.

This work has been supported in part by National Science Foundation contract IIS-0113427 and the Max Planck Center for Visual Computing and Communication.

© 2005 ACM 0001-0782/05/0600 $5.00

c

80 June 2005/Vol. 48, No. 6 COMMUNICATIONS OF THE ACM

Figure 3. Client-sidelow-resolution (top)and server-side high-resolution (bottom) model renderings.