52 August 2001/Vol. 44, No. 8 COMMUNICATIONS OF THE ACM

Populatingthe HermitageMuseum’s NewWeb Site Experts from around the world tackle one of the most ambitious Web-based museum projects and the result captures the true artistry of teamwork, technology, and yes, art.

COMMUNICATIONS OF THE ACM August 2001/Vol. 44, No. 8 53

Russia’s palatial State HermitageMuseum in St. Petersburg possessesmany great collections [10] includ-ing prehistoric art, antiquities,Western European art, arms andarmor, oriental art, and numismat-

ics. Indeed, the Hermitage’s collections are vast,numbering over three million holdings. Arguably,it is the largest art gallery in the world. Althoughmore than 500 rooms are open to the public, theydisplay less than 10% if its holdings.

In 1997, IBM and the Hermitage began explor-ing the nature of museum information services thatwould best serve the museum. Underlying thesediscussions was the vision of Mikhail Piotrovski,director of the Hermitage, that information ser-vices could help fulfill the museum’s mission in the21st century. Three end-user applications wereidentified for development: multimedia-based arteducation housed in an education and technologycenter, visitor information kiosks, and a new Website that would permit the Hermitage’s collectionsto be searched and better experienced from afar.

In addition, an image creation studio would bedeveloped to produce high-quality digital imagesfor the three applications. The project was brieflydescribed in [7] and www.hermitagemuseum.org/html_En/00/hm0_8.html. Its initial deploymentwas unveiled on June 15, 1999 (seewww.ibm.com/news/1999/06/15.phtml)

The multimedia-based education programs werecreated to serve Hermitage visitors—especially chil-

dren—by providing interactive art education. Theyrun on seven workstations in an Education andTechnology Center (ETC) within the museum.Initially, each workstation offered two coursesdeveloped cooperatively by Hermitage and IBMeducational personnel; one featuring art on the lifeof Jesus, the other featuring antiquities. Much ofthe material in each course is presented by display-ing high-resolution images of art with a narrativedescribing the artist, the art’s theme, and otherinteresting aspects. A variety of quiz-type games areincluded.

The visitor information kiosks were created tohelp Hermitage visitors choose a set of holdings tovisit and then navigate the museum in order to seethem. Visitors may select ageneral interest suggestedtour of specified durationor a museum collection, orprepare a personalized tour;six suggested tours and 31tours of museum collec-tions are offered.Alternatively, the visi-tor may compose apersonalized tour byleafing through a set ofHermitage highlightsand selecting up to five.To facilitate navigation,each kiosk plots the tourroute on maps of the

Fred Mintzer, Gordon W. Braudaway, Francis P. Giordano, Jack C. Lee, Karen A. Magerlein, Silvana D’Auria, Amnon Ribak, Gil Shapir, Fabio Schiattarella, John Tolva, and Andrey Zelenkov

Figure 1. Exterior view ofthe Hermitage.

Used by permission of

the State Hermitage

Museum, St. Petersburg,

Russia.

Hermitage’s interior. Thumbnail images of promi-nent Hermitage features, serving as navigationalmilestones, are superimposed on the maps.

Here, we use the term “museum collections” toindicate an organization of holdings by region andera of origin, as is done in the museum community.We use the term “tour” to describe the route to theholdings of interest, and not a person-hosted tour.

The Web site was created to serve remote patronsby familiarizing them with the Hermitage’s hold-ings, its history, news, temporary exhibitions, andthe visitor services it provides. Remote patrons are

served in two languages: Englishand Russian. The quality of itsimages was intended to permitvisitors to experience the muse-um’s art. A digital library ofimages and descriptive text addsthe ability to search the Her-mitage’s holdings.

The Web SiteThe Web site (www.hermitage-museum.org) represents the col-lective expertise of a largetechnical team that developedthe system, a large team ofmuseum professionals thatdeveloped procedures for datadevelopment and validation,and a large design team respon-sible for the aesthetics andusability of the site. It also repre-sents the dedicated labor ofmany professionals who popu-lated the site with text andimages. More than 100 technicaland Hermitage museum profes-sionals contributed to the site.

The Web site consists of twonearly independent parts thatserve different purposes. Thestatic part consists of over 1,500pages of detailed descriptions ofmuseum collections (“Collec-tion Highlights”). Like an artcatalog, it reflects museum col-lection structure and is illus-trated by well-known highlights.It rarely changes. The dynamicpart consists of a digital libraryof holdings (“Digital Collec-tion”) accessible in many ways.It is updated often.

Prior to the Hermitage/IBM project, the museumprovided an attractive Web site presenting many ofthe museum’s collections. The static part of the cur-rent Web site was based on the design and contentof this prior site. An international team led by IBMe-business services in Atlanta structured, designed,and implemented the static part. Its creation beganwith development of a comprehensive set of stan-dards for the graphical design of the pages and forthe ways information could be presented. Next, thegraphical elements were completed. Images anddescriptive text were created. Pages were then assem-

54 August 2001/Vol. 44, No. 8 COMMUNICATIONS OF THE ACM

Figure 2. The Small Throne Room, with painting Peter I with Minerva by Jacopo Amigoni. Used by permission

of the State Hermitage Museum, St. Petersburg, Russia.

bled and tested in English. Finally, companion Russ-ian-language pages were produced.

Many Hermitage specialists contributed to the datadevelopment process for the dynamic element of thesite. This begins with selection of holdings, followedby their classification by a team of database and infor-mation administrators. Then, with a curator, the cat-aloging attributes (for example, technique, genre,epoch) are added. Concurrently, Hermitage writersdevelop annotations validated by a curator. When theholding’s information is completed, it is translatedinto English. Photography of the holdings, scanning,and image preparation takes place at the same time.Lastly, a complete set of information is entered—in

English and Russian—into the digital library.It was anticipated that many site visitors might

not be familiar with the organization of museumcollections. To help them better browse the Her-mitage’s holdings, the dynamic part of the site orga-nized the holdings in a different manner—as 12digital collections whose contents depend on theform of the art’s representation. The digital collec-tions are:

•Paintings, prints, and drawings;•Sculpture;•Machinery and mechanisms;•Arms and armor;

COMMUNICATIONS OF THE ACM August 2001/Vol. 44, No. 8 55

Figure 3. Woman in the Garden.Sainte-Adresse by Claude Monet.Used by permission of the State Hermitage Museum,St. Petersburg, Russia. (a) original image. (b) water-marked image. (c) visualizationof extracted watermark.

(a)

(b) (c)

•Furniture and carriages;•Ceramics and porcelain;•Applied arts;•Jewelry;•Textiles;•Numismatics and glyphs;•Costumes; and•Archaeological artifacts.

To browse, one first selects the desired digital col-lection; and collection-specific information is pre-sented to help refine the search. Each collection hasits own categories of cataloging information. “Artist’sname” and “painting’s title,” for example, facilitate

browsing through paint-ings, but would be of lit-tle use in browsingthrough archaeologicalartifacts.

Parametric searches ondatabase fields within thedigital library thatdescribe the digital col-lections are also pro-vided. Here, the visitorfirst selects a digital col-lection, then enters adescription of the desiredholdings. Although per-haps cumbersome for thecasual visitor, thesesearches better supportthe Hermitage staff andthe expert visitor.

The Web site alsooffers a free text-basedquick search across digitalcollections. One mightuse this function, forexample, to locate allworks of Michelangelo(Buonarroti), includingboth sculpture and paint-ing. We note that search-ing on “Michelangelo”locates the statue Crouch-ing Boy by MichelangeloBuonarroti, a virtual tourof the Michelangelogallery, and a painting byMichelangelo Merisi(commonly known asCaravaggio).

The Web site also sup-ports searches on the paintings, prints, and drawingsdigital collection based on the visual layout (form andcolor) or color palette of holdings using IBM’s QueryBy Image Content (QBIC) technology [3].

Having located a holding, the visitor can launcha similarity search for other holdings that share somecharacteristic. The supported similarity searchesdepend on the digital collection to which the locatedholding belongs. For example, the similarity searchfor paintings enables searching for other paintingsby the same artist, of the same style, from the samecountry/region, made using the same technique, ofthe same genre, or with similar visual layout.

Creating images that permit the Web site visitor

56 August 2001/Vol. 44, No. 8 COMMUNICATIONS OF THE ACM

Figure 4. Portrait of Lady-in-Waiting to the Infanta Isabella by Peter Paul Rubens. Used by permission of the State Hermitage Museum, St. Petersburg, Russia.

to experience the Hermitage’s holdings is very chal-lenging. The Web site provides visitors with high-resolution images carefully retouched by Hermitagestaff. For each holding, two full-screen images areoffered to the visitor. One is intended for Web sitevisitors using a display with 800 X 600 pixel resolu-tion, the other for visitors using a display with 1,024 X 768 pixel resolution. They provide thelargest images that will fit on the respective displays,given the normal amount of screen space occupiedby the Web browser.

In addition, an interactive zoom function allowsthe visitor to select a region of an image and viewthat region with an increased level of detail. A coarseenlargement is first provided, then refined as addi-tional information is received from the site.

Still images provide excellent representations ofmany of the Hermitage’s holdings. However, theHermitage’s galleries challenge the limitations of stillimages. To enable Web site visitors to better experi-ence Hermitage galleries, the site contains 14panoramic tours of Hermitage rooms, galleries, andexterior views, presented with IBM’s HotMediatechnology [6]. To better represent Hermitage stat-ues, multiple views are sometimes provided.

The Digital LibraryThe digital library enables the browsing and search-ing provided by the site. It generates dynamic pagesfor specifying browsing and searches, executes thesearches, and prepares dynamic pages that presentthe search results.

Earlier we noted the Hermitage’s holdings on thedynamic side were organized as 12 digital collec-tions, each of which could be searched (or browsed)on descriptive information specific to that digitalcollection. The digital library stores each holdingwith cataloging information specific to its digitalcollection, a text object that combines the structuredinformation, and visual features that enable theQBIC searches.

In addition, seven derivative images are stored foreach holding, pregenerated to improve response time(versus generation on the fly). Created from theholding’s scan, the derivative images satisfy differentapplication needs. They are:

•A small thumbnail used to present search results;•A larger thumbnail used in pages that provide

“more information about an artwork”;•A quarter-screen image describes the holding and

permit zoom regions to be specified;•A full-screen image for display at 1024 X 768

pixel resolution;

•A full-screen image for display at 800 X 600 pixelresolution;

•A high-resolution image from which the zoom’sdetail is composed; and

•A scan-resolution image not used by the Web sitethat may be manually extracted for use by kiosks,ETC programs, or other applications.

IBM’s Digital Library product (since renamedContent Manager) provides the digital library’s coreinformation management functions. For this proj-ect, it was enabled to support the Russian characterset. Application code was developed to generate thedynamic pages that permit queries to be specifiedand results presented. This was a significant taskbecause of the variety of queries offered and the needto support them in both Russian and English.

Application code was also written to enter Her-mitage content into the digital library. Each digitalcollection has its own data model that specifies howthat collection’s information is structured in the dig-ital library (structured fields, objects and their rela-tionships). Each digital collection requires a loaderto collect the information associated with the hold-ing, perform consistency checking on it, and enter itinto the digital library (in batch mode). The devel-opment of a composite loader that supports the 12data models in English and Russian was a significantproject task.

Several data-entry applications were written tofacilitate secured data entry. Dictionaries were usedby the data-entry applications to better manageentries for some information categories. The use ofdictionaries eased the translation burden, since arepeated entry would not require a new translationwhen encountered again. It also increased catalogingconsistency. For example, as holdings painted byRembrandt (Harmensz Van Rijn) were loaded intothe digital library, his name would be extracted fromthe same dictionary entry, ensuring all instanceswould use the same representation.

The Image Creation StudioThe Image Creation Studio (ICS) permits Her-mitage staff to scan its holdings, retouch the images,and prepare derivative images for the Web site. Fur-thermore, the ICS integrates these functions into aunified workflow.

Image capture is provided by an IBM Pro/3000scanner [4]. Other Pro/3000 scanners have beenused in earlier cultural heritage projects, often tocapture images of homogeneous collections of mate-rials. In the IBM/Vatican Library project [8], forexample, they captured predominately images of

COMMUNICATIONS OF THE ACM August 2001/Vol. 44, No. 8 57

manuscript pages. The Hermitage’s holdings, bycontrast, are very diverse. Some are photographic,some are not; some are transmissive, some reflective;some are 2D, some 3D; and they are of varying size.Developing illumination, fixtures, and workflowneeded to create images from the diversity of Her-mitage holdings was another significant task.

Significantly less retouching is required when aholding can be moved to the ICS and captureddirectly (versus scanned from a photograph of theholding). Several lighting and mechanical fixtureswere developed to enable direct capture, includingreflective halogen lighting for photographic printsand drawings, reflective fluorescent lighting forcoins and medallions, a lighting booth for small 3Dholdings, and a book easel for holding bound books.

However, it is often notpractical to move holdings tothe studio because of theirsize, fragility, or sheer value;they must first be pho-tographed, then scanned fromthe photographic materials.This procedure was the domi-nant form of image capturedespite the advantages ofdirect capture.

To achieve the highest qual-ity, the scanned images mustbe retouched. Many holdings,such as oil paintings, haveshiny surfaces that producereflections that should beremoved. Photography dis-torts the color of the holdings.Scanned holdings oftenexhibit dust. Both Adobe Pho-toShop and Micrographx Pic-ture Publisher were used toperform the retouching.

The ICS prepares the deriv-ative images for the Web sitefrom the retouched imagesusing a batch process. Foreach source image, derivativesare created by: reducing thesource image; sharpening thereduced-size image; colortransforming the reducedimage to the color space of astandard monitor [8]; embed-ding an invisible watermark(optional); and compressingthe resulting image.

Larger derivatives are generally created first andintermediate reductions are retained to serve assource images for the creation of smaller derivatives(to reduce computations). A color management sys-tem helps retain the color content throughout theprocessing.

To better protect the Hermitage’s intellectualproperty, invisible watermarking was applied tolarger derivative images made available through theWeb site. The watermark cannot be seen by ahuman observer, but can be extracted by a computerprogram. If an image, perhaps used commercially, issuspected of being Hermitage property, a watermarkextraction is attempted; an extracted watermark pro-vides evidence of Hermitage ownership.

The watermarking technique [1] used embeds

58 August 2001/Vol. 44, No. 8 COMMUNICATIONS OF THE ACM

Figure 5. Ptolemy II and Arsinoe II (The Gonzaga Cameo), Ancient Egypt, 3d century BC. Used by permission of the

State Hermitage Museum, St. Petersburg, Russia.

watermarks that are resistant to many types of imageprocessing [2]. We note the watermark may beextracted from printed material by first scanning thematerial and then running the extraction. Further-more, each watermark is embedded with a compan-ion watermark key that functions much like apassword; the watermark cannot be extracted with-out it. Hence, a malicious party who tried to removethe mark without the watermark key would notknow if the removal attempt was successful. Detec-tion of the mark, following a removal attempt,would always be a possibility.

The watermarking process inserts a small textureinto the source image. Extraction is performed bycomputing a number of statistics that compare thetexture of a candidate image with the inserted tex-ture. A visualization of the watermark extraction iscomposed by exclusive oring of the statistics with aninstitutional logo. An example source image, water-marked image, and visualization of an extractedwatermark are shown in Figure 3.

The diversity of the visual content of Hermitage’sholdings proved challenging to the watermarking.Some images, especially large bright images with lit-tle texture, were successfully marked at low water-marking strength. Other smaller, darker, or moretextured images required higher watermarkingstrength. To accommodate this diversity, thestrength was determined adaptively. For each hold-ing, one of the derivative images was marked andextraction attempted. If unsuccessful, the strengthwas increased and the process repeated; when suc-cessful, the other derivative images were marked atthis strength.

The three ICS functions—scanning, retouching,and derivative-image creation—are often performedby different personnel; the ICS’s workflow providesthat the output of each ICS function may be storedon CD-ROM to decouple the functions.

The ICS workflow also ensures that vital informa-tion about the images entered during scanning isretained through later stages. During the scanningprocess, the operator enters an annotation thatuniquely identifies the holding; this is preserved in theimage’s header. This unique identifier is also enteredin the cataloging information describing the holding.The loader later uses this information to match theholding’s images with its cataloging information. Thescan application also enters color (calibration) infor-mation into each scanned image’s header.

The unique identifier and color informationwould normally be removed by the retouching soft-ware. Prior to retouching, the ICS workflow readsthe image headers and saves this information; it also

rotates the images to the proper orientation and cor-rects their appearance for the monitor on which theyare viewed. After retouching, annotations and colorcalibration information are replaced in the imageheaders. The derivative image creation retains theidentifiers and color information throughout its pro-cessing so they will be available to the loader.

Emerging IssuesSeveral of the authors have collectively participatedin the development of other information systems forcultural institutions [5, 7, 8]. In this project, somenew issues were addressed; some old issues wereaddressed in new ways.

Protecting online intellectual property from mis-appropriation is a continuing issue for online digitallibraries. In this project, it was a matter of greaterconcern because the Web site made images availableat high resolution (and consequently high value).However, the visible watermarking technology usedin earlier IBM projects [5, 8] was judged inappro-priate for art. When darkly applied, it interferedwith the visitor’s experience of the art; when lightlyapplied, it left the visitor wondering what was paint-ing and what was watermark. This project was thefirst in which we relied on invisible watermarking toprotect images made available online.

One of the project’s great challenges was diver-sity—of applications, of holdings, and of user com-munities. The diversity of applications required asignificant software development effort. The diver-sity of holdings inspired the development of 12 datamodels and a composite loader to support them. Itinspired the development of scanning fixtures and aworkflow to support capture of many types ofobjects. It inspired the kiosks and Web site to pro-vide panoramas and multiple views of statues. Itinspired a watermarking application process thatadapted watermark strength to image content.

The diversity of user communities motivated theproject to develop multiple applications in multiplelanguages. It motivated the Web site to support twosizes of full-screen image for user communities withdifferent display resolutions. It motivated the projectto develop multiple search techniques—browsing oncollection-specific parameters and free text-basedquick searches for the general user, parametricsearches for the expert user, and visual searches forthose most interested in visual appearance. Often,the result of dealing with the diversity was to permitthe user communities to view the same source con-tent in different ways through different applications,translations, search interfaces, and image resolutions.Providing systems that more easily create multiple

COMMUNICATIONS OF THE ACM August 2001/Vol. 44, No. 8 59

views of the same content is a worthy goal for futuresystems.

Dealing with diversity can have its benefits.Preparing content for one application, for examplean ETC course, entails performing much of thework (research, image creation, data entry) neededto prepare it for others. This speeds both the cre-ation of content and gives it greater value.

Another of the project’s great challenges wassize—the size of the museum’s collections and thesize of the user communities. Although images offewer than 3,000 holdings were provided when theWeb site was unveiled, this is a relatively large num-ber of high-quality images for a museum site.

It is difficult to create large volumes of high-qual-ity images. Scanning is one of the less laborious oper-ations. Retouching, inspection, and researching theholding’s cataloging are more time-consuming chores.There is a great need for technology that streamlinesdigital content creation through faster capture,inspection, and retouching; much progress is neededto make truly large collections available online.

RetrospectiveIn the days following the unveiling of the new Website, it received nearly a million hits a day. Subse-quently, it’s averaged over 200,000 hits a day. In theyear following the unveiling, it received AdvancedImaging magazine’s “Solution of the Year” award fordigital photography and still-image databasing, wasvoted “Best Site Overall,” by the Museums and theWeb 2000 conference, and received the RussianInternet Academy’s Grand Prix award for best over-all Web site in Russia. It has been covered in theNew York Times, the Financial Times, and NationalGeographic Traveler.

However, the e-commerce community has shownlittle interest to date in replicating its function. Inthe museum community, there is a steady desire toprovide a better experience of their collections; amore immersive experience is desired. Some of theauthors are now exploring more realistic representa-tions of 3D objects.

In the commercial realm, creating a fuller experi-ence of merchandise has often been a lesser concernthan image data size and scanning cost. This sitemay just be ahead of its time.

Interest in invisible watermarking continues togrow. However, we have come to appreciate thatwatermarking algorithms alone do not provide agood watermarking system. Proper support ofwatermarking also requires secure key generationand management; such a system has been a focus ofsome recent work.

References1. Braudaway, G.W. Protecting publicly available images with an invisi-

ble image watermark. In Proceedings of the IEEE ICIP’97. (Oct. 1997),524–527.

2. Braudaway, G.W. Results of attacks on a claimed robust digital water-mark. In Proceedings of the 1998 IS&T/SPIE International Symposiumon Electronic Imaging Symposium on Optical Security and CounterfeitDeterrence Techniques. (Feb 1998), 122–131.

3. Flickner, M. et al. Query by image and video content: The QBIC sys-tem. IEEE Computer. (Sept. 1995), 23–32.

4. Giordano, F.P. et al. Evolution of a high-quality digital imaging sys-tem. In Proceedings of the Conference on Sensors, Cameras, and ImagingApplications of Digital Photography, IS&T/SPIE International Sympo-sium on Electronic Imaging Science and Technology. (Jan. 1999),110–118.

5. Gladney, H.M. et al. Digital access to antiquities. Commun. ACM 41,4 (Apr. 1998), 49–57.

6. Kumar, K.G. et al. The HotMedia architecture: Progressive & interac-tive rich media for the Internet. Seewww.developer.ibm.com/library/articles/hotmedia.html.

7. Mintzer, F. Developing digital libraries of cultural content for Internetaccess. IEEE Communications. (Jan. 1998), 72–78.

8. Mintzer, F.C. et al. Towards on-line worldwide access to Vaticanlibrary materials. IBM J. of Research and Dev. 40, 2. (Mar. 1996),139–162.

9. The Hermitage, Buildings and Interiors. Alfa-Colour, St. Petersburg,Russia, 1997.

10. The Hermitage, Selected Treasures from a Great Museum. Booth-Clib-born Editions, London, England, 1990.

Fred Mintzer (mintzer@us.ibm.com) is senior manager of theVisual Technologies department at IBM Thomas J. Watson ResearchCenter, Yorktown Heights, NY.Gordon W. Braudaway (braud@us.ibm.com) is a researchstaff member at IBM Thomas J. Watson Research Center, YorktownHeights, NY.Francis P. Giordano (giordanof@us.ibm.com) is a senior engineer and project leader of Image Capture Systems at IBM Thomas J. Watson Research Center, Yorktown Heights, NY.Jack C. Lee (jacklee@us.ibm.com) is an advisory engineer at IBMThomas J. Watson Research Center, Yorktown Heights, NY.Karen A. Magerlein (kmager@us.ibm.com) is a software developer at IBM Thomas J. Watson Research Center, YorktownHeights, NY. Silvana D’Auria (silvana_dauria@it.ibm.com) is leader of theContent Manager area at IBM’s e-business Solutions Center, Naples,Italy.Amnon Ribak (RIBAK@il.ibm.com) is the founder and formermanager of the Customer Access Applications group at IBM’s HaifaResearch Laboratory, Haifa, Israel. Gil Shapir (SHAPIR@il.ibm.com) is a project leader and staffmember at IBM’s Haifa Research Laboratory, Haifa, Israel.Fabio Schiattarella (schiatta@us.ibm.com) is a senior IT architect in the National Digital Media Practice of IBM Global Services, Atlanta, GA.John Tolva (jtolva@us.ibm.com) is a creative director at theChicago Center for IBM e-business Innovation, Chicago, IL.Andrey Zelenkov (andreyz@us.ibm.com) is an e-business project manager for IBM Global Services, New York, NY.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. To copy otherwise, to republish, to post on servers or to redis-tribute to lists, requires prior specific permission and/or a fee.

© 2001 ACM 0002-0782/01/0800 $5.00c

60 August 2001/Vol. 44, No. 8 COMMUNICATIONS OF THE ACM