semantic classification of byzantine iconsreformat/ece720w2012/papers/ieeexplore... · byzantine...

March/april 2009 1541-1672/09/$25.00 © 2009 IEEE 35Published by the IEEE Computer Society

In the past, people gathered cultural in-formation from physical objects (such as books, sculptures, statues, and paintings). Now, digital collections have become com-mon practice for most cultural-content providers.

However, as the amount of the Web’s cul-tural content grows, search and retrieval procedures for that content become increas-ingly difficult. Moreover, Web users need more efficient ways to access huge amounts of content. So, researchers have proposed so-phisticated browsing and viewing technolo-gies, raising the need for detailed metadata that effectively describes the cultural con-tent. Several annotation standards have been developed and implemented, and Semantic Web technologies provide a solution for the semantic description of collections on the Web.1 Unfortunately, the semantic annota-tion of cultural content is time consuming and expensive, making it one of the main difficulties of cultural-content publication. Therefore, the need for automatic or semi-automatic analysis and classification of cul-tural assets has emerged.

To meet this need, researchers have pro-posed using image analysis methods (for some examples, see the sidebar on the next page). However, these methods use domain knowledge for only low-level analysis. Also, two main difficulties have arisen in using image analysis to automate annotation and classification of cultural digital assets. The first is the failure of semantic image seg-mentation and image analysis algorithms in some real-world conditions. This is due to the high variability of content and environ-mental parameters (luminance and so on), which makes the problem complex. The sec-ond difficulty is the extensive and vague na-ture of domain knowledge (at least for some cultural content), which complicates formal knowledge representation and reasoning.

However, some cultural domains are ap-propriate for automatic analysis and classi-fication methods. Byzantine icon art is one of them. The predefined image content and the low variability of the image characteris-tics support the successful application of im-age analysis methods. Consequently, we’ve developed a system that exploits traditional

Recent advances in Web technologies that ensure quick, effective infor-

mation distribution have created the ideal terrain for spreading vari-

ous cultures through cultural-content presentation. Nowadays, we are wit-

nessing the publication of a huge amount of cultural information on the Web. This system uses

fuzzy description

logics and patterns

to automatically

determine the sacred

figure depicted

in an icon.

Paraskevi Tzouveli, Nikos Simou, Giorgios Stamou, and Stefanos Kollias, National Technical University of Athens

Semantic Classification of Byzantine Icons

A I A n d C u l t u r A l H e r I t A g e

Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on October 1, 2009 at 14:57 from IEEE Xplore. Restrictions apply.

36 www.computer.org/intelligent iEEE iNTElliGENT SYSTEMS


iconographic guidelines to automatically analyze icons. Our system detects a set of concepts and properties de-picted in the icon, according to which it recognizes a sa-cred figure’s face, thus providing semantic classification.

Byzantine IconographyByzantine art refers to the art of Eastern Orthodox states that were concurrent with the Byzantine Empire. Certain artistic traditions born in the Byzantine Empire, particu-larly regarding icon painting and church architecture, have continued in Greece, Bulgaria, Russia, and other Eastern Orthodox countries up to the present. The icons usually depict sacred figures from the Christian faith, such as Je-sus, the Virgin Mary, the apostles, and the saints.

At the beginning of the 17th century, iconography man-uals represented the most important source of inspiration for icon painters. The 16th-century monk Dionysios from Fourna is credited as the author of the prototypical man-ual, Interpretation of the Byzantine Art.2 In it, he defines the rules and iconographic patterns that painters still use to create icons.

Byzantine iconography follows a unique convention of painting. The artistic language of Byzantine painters is characterized by apparent simplicity, overemphasized flat-ness, unreal and symbolic colors, lack of perspective, and strange proportions. The sacred figures are set beyond real time and real space through the use of gold backgrounds. The most important figure in an icon is depicted frontally, focusing on the eyes, facial expression, and hands. Clarity is the rule, not only in describing shapes but also in arrang-ing figures in balanced compositions so that the narrative’s actions are clear. The figures have large eyes, enlarged ears, long, thin noses, and small mouths, each painted following specific rules.

When depicting various figures, the artist is obliged to observe the manual’s rules regarding posture, hair and beard style, apparel, and other attributes. Specific sacred

figures of the Christian faith are defined by specific facial characteristics. For example, according to Dionysios, Jesus is illustrated as a young figure with long, straight, thick, black hair and a short, straight, thick, black beard.

According to these rules, the area in which the sacred figure will be painted is separated into equal segments. The first segment contains the head, focusing on the eyes and facial expression. This segment contains the sacred fig-ure’s most distinguishable features—for example, hair and beard style. The remaining segments contain the sacred fig-ure’s body, focusing on the hands and arms. The body can be depicted in different poses—for example, standing, sit-ting, from the head to the waist, and from the head to the thorax.

The head should be elliptical. The major axis is along the vertical part of the head and is equal to four times the height of the nose (H). (According to Dionysios, the nose serves as the metric for producing harmonious facial and body proportions.) The minor axis is along the horizontal and is equal to 3H. The head segment can be separated into four equal vertical sections of height H. The hair is in the first part, the forehead is in the second, the nose is in the third, and the area between the mustache and the chin is in the fourth.

A Byzantine painter applies four base colors in layers on the face segment. The darking is a dark color for the base of the face segment and the darkest shadows. The proplas-mos is a less-dark color for lighter shadows. The sarkoma is for flesh, and the lightening is for highlights, containing a good amount of white and being light enough to show off against the base color. The absolute value of these colors differs from icon to icon.

From Art Manual to Semantic RepresentationAlthough the knowledge in Dionysios’s manual concerns vague concepts such as “long hair,” “young face,” and so on, it’s quite strict and formally described. Consequently,

The July 2008 IEEE Signal Processing Magazine pre-sented some particularly interesting applications of image analysis to the cultural-heritage domain. Ale-

jandro Ribes and his colleagues described problems con-cerning the design of multispectral cameras as well as the analysis of the Mona Lisa in the context of the digitization of paintings.1 Anna Pelagotti and her colleagues proposed a novel multispectral digital-imaging technique for clus-tering image regions in a set of images containing similar characteristics when exposed to electromagnetic radia-tion.2 Their technique provides material localization and identification over a painting’s entire surface. C. Richard Johnson and his colleagues described how three research groups developed systems that identify artists on the basis

of brushwork characteristics that art historians use for au-thentication.3 Finally, Howard Leung and his colleagues re-ported on the preservation of ancient Chinese calligraphy.4

References 1. A. Ribes et al., “Studying That Smile,” IEEE Signal Processing

Magazine, vol. 25, no. 4, 2008, pp. 14–26. 2. A. Pelagotti et al., “Multispectral Imaging of Paintings,” IEEE

Signal Processing Magazine, vol. 25, no. 4, 2008, pp. 27–37. 3. C.R. Johnson et al., “Image Processing for Artist Identifica-

tion,” IEEE Signal Processing Magazine, vol. 25, no. 4, 2008, pp. 37–48.

4. H. Leung, S.T.S. Wong, and H.H.S. Ip, “In the Name of Art,” IEEE Signal Processing Magazine, vol. 25, no. 4, 2008, pp. 49–54.

related Work on Image Analysis for Cultural Heritage


March/april 2009 www.computer.org/intelligent 37

we can create an ontological represen-tation of this knowledge using OWL. In this way, the ontology’s axiomatic skeleton will provide the terminology and restrictions for Byzantine icons.

The only problem is that even a well-tuned image analysis algorithm can’t produce perfect results for a wide set of data with different char-acteristics. Moreover, many Byzan-tine icons might be misinterpreted owing to their age, further affecting the results. Finally, the vague nature of terms such as distance, color, and length introduces fuzziness in the cat-egorization of objects. For example, sometimes a sacred figure represents a young face, but only to a certain degree. In such cases, standard de-scription logics (DLs) are insufficient because they can’t deal with vague, uncertain information. To handle such problems, researchers have pro-posed fuzzy DLs, based on fuzzy sets and fuzzy logic, which are suitable for handling imprecise information.3

To classify the important facial features of Byzantine icons and detect the sacred figures having specific charac-teristics, we combine image analysis algorithms with the expressive power of fuzzy DLs. Using the rules and pat-terns described in Dionysios’s manual, we constructed a fuzzy knowledge base that’s populated by the results of im-age analysis of the digitized paintings. Using fuzzy reason-ing and on the basis of the defined terminology, our system can classify the figures in Byzantine icons.

The Icon Classification SystemFigure 1 shows the general architecture of our Byzantine icon classification system. It consists of the Byzantine icon analysis subsystem and the knowledge representation and reasoning subsystem.

Byzantine icon analysisThis subsystem performs the image processing and analy-sis tasks necessary to detect a set of primitive concepts and properties (such as face and long hair) and form assertional knowledge that semantically describes the icon’s content. It consists of modules for semantic segmentation, feature ex-traction, and semantic interpretation.

The first step of system development was the interpre-tation, with the aid of Byzantine iconography experts, of Dionysios’s manual. This procedure provided us with

a large set of heuristics that facilitated the design of the subsystem modules and helped us select the most ap-propriate semantic segmentation and feature extraction methods.

Semantic segmentation. This module divides the icon into regions visualizing the icon’s physical objects and their parts.

The first step of semantic segmentation is face detection. For this, we adopted a fast method based on support vec-tor decision functions.4 This method first uses integral im-aging for very fast feature evaluation based on Haar basis functions. Then, it uses a simple, efficient classifier based on the AdaBoost learning algorithm to select a small num-ber of critical visual features from a very large set of po-tential features. Finally, this method combines classifiers in a “cascade,” quickly discarding the image’s background regions while concentrating computation on promising face-like regions. An example of this method’s results is the green rectangle in the first image of the face detection box in Figure 2 (see next page).

The next step is localization of the eyes. Be-cause the nose’s height should be equal to the dis-tance of one retina from the other, that distance is a critical parameter for segmentation. The semantic- segmentation module performs eye localization with the

Featureextraction

Byzantine iconanalysis

Semanticinterpretation

Semanticsegmentation

Sacred-figurerecognition

Knowledge representationand reasoning

Assertionalknowledge

Terminologicalknowledge

Reasoningengine

Byzantineicon

database

Figure 1. The Byzantine icon classification system’s architecture. The analysis module extracts information about the icon, and the knowledge representation and reasoning module uses this information to infer implicit knowledge that categorizes the icon.




aid of geometric information from the eyes and the sur-rounding area.5 We use a training set of sacred-figure eyes to create vector maps, and we use principal com-ponents analysis to derive eigenvectors for these maps. For a candidate face segment, the algorithm5 projects the length and its angle maps on the spaces spanned by the eigenvectors that were evaluated during train-ing. In that way, we use the similarity of the projection

weights and model eyes to deter-mine an eye’s presence and position (indicated by the red and green dots in the second image in the face de-tection box of Figure 2).

After eye localization, the mod-ule detects the nose. Let H denote the distance between the eyes and con-sequently the nose’s length. Accord-ing to Dionysios, the nose starts on a small horizontal line whose center is a distance of H/5 below the midpoint between the eyes. We use a Hough transform to find this line. Therefore, we determine the nose’s ending line by applying a Hough transform at a distance H down vertically from the starting line.

The computation of H leads to the detection of the face’s other main com-ponents—the hair, forehead, cheek, mustache, and beard—because they are well defined in Dionysios’s manual.

The module detects the four base colors by applying an Otsu segmen-tation algorithm6 to a restricted area of the face segment (see the base color analysis box in Figure 2), de-termining a range of intensity values for the pixels.

To segment each part of the face, we use H together with Dionysios’s pro-portion guidelines to define the seeds for graph-cut segmentation.7 The cost functions used as soft constraints in-corporate boundary and part infor-mation for each of the five segmented parts. To achieve segmentation of each part, we compute the minimum cut that’s the global optimum among all the segmentations satisfying the constraints (see the face-componentdetection box in Figure 2).

Feature extraction. This module automatically ex-tracts features from the segmented parts,8 providing additional information regarding each part’s length, color, shape, and texture. This information constitutes the characteristic values of the features, providing an image description set (see the image-description sec-tion of the feature extraction box in Figure 2).

Dionysios assigns four features to hair:

s1

s2

s3

s4

s5

Face detectionextracts the face

and its basiccomponents

(nose and eyes).

Face componentdetection

detects the mostimportant components

of the face (beard,moustache, cheek,forehead, and hair,

respectively).

Feature extractionautomatically extracts

features such asdistance, color,and length for

each facecomponent and

for somesegmentsof the face

components.

Semanticinterpretation

gives meaning tospecific features,properties, andrelationships of

face componentswith the aid

of fuzzypartitions.

Base coloranalysis extracts

the base colorlayers (darking,

proplasmos.sarkoma, andlightening).

Image descriptions: Faces1: Hairs2: Foreheads3: Cheeks4: Mustaches5: Beard(s, s1): hasComponent...Length(s1) = 2.8*H...darking(s1) = 88%...

Formal assertionss: Faces1: Hairs2: Foreheads3: Cheeks4: Mustaches5: Beard(s, s1): hasComponent...s1: AboveEarsHair = 0.7s1: BelowEarsHair = 0.3...

BelowEarsHair AboveEarsHair

Mem

bers

hip

10.70.3

0 2.8Hairlength

Fuzzy partitioning

5

Figure 2. Image analysis. First, the algorithm detects the sacred figure’s face region, eyes, and nose. Then, it extracts the hair, forehead, cheek, mustache, and beard parts together with the face’s base color layers. Further analysis of the extracted parts provides information about characteristic features. Finally, the algorithm produces a semantic interpretation for each of these features, together with formal assertions.



color (dark, gray, or white),• density (thin or thick),• length (above the ears or below the ears, and whether the • fi gure has a braid on the left, right, or both shoulders), andform (straight, wild, or curly).•

The feature extraction module categorizes hair color (see Figure 2, feature extraction, segment s1) using the intervals of Otsu segmentation. It estimates the density and form by combining a Canny edge detector and a morphological fi l-ter and by evaluating the mean value of the curvatures of the detected edges. It fi nds the hair’s length using H to-gether with Dionysios’s proportion guidelines.

Next, the module determines information about the sa-cred fi gure’s age by computing the number of wrinkles on the forehead (s2). Using the Hough transform, the module defi nes the number of verti-cal and horizontal lines in the fore-head, which represent the wrinkles. A young sacred fi gure has zero or few wrinkles, whereas an older sacred fi g-ure has many.

The cheek (s3) provides additional information about age. According to Dionysios, a small number of wrin-kles on the larger (left or right) part of the cheek indicates a young sacred fi gure. Conversely, an older sacred fi g-ure’s cheek will have a greater number of wrinkles (and therefore a greater proportion of the colors used for shadows).

Finally, the module defi nes the feature sets for the mus-tache (s4) and beard (s5). Using techniques similar to those for hair analysis, it analyzes the mustache’s color and shape. It also extracts information on the beard’s length (short, not very long, long, down to the waist, or down to the knees), color (fair or dark), form (straight, wild, or curly) and shape (sharp, circle-like, circle, wide, or fl uffy).

Semantic interpretation. This module represents the ex-tracted information in terms of the sacred-fi gure ontology.

Representing the face and its main components is straightforward: we simply write assertions such as s1: Hair (see the semantic-interpretation box in Figure 2). However, each component’s properties are vague. For ex-ample, you can’t always directly determine on the basis of length whether the sacred fi gure has long hair. This dif-fi culty is due to the vagueness of the defi nition of length and the imprecision introduced by feature extraction (see, for example, the result of the segmentation of the hair in the feature extraction box in Figure 2). Nevertheless, we

can express an assertion in a fuzzy manner, with the aid of fuzzy set theory. This procedure is critical for the knowl-edge representation system.

Suppose that a specifi c feature’s values lie in the inter-val X. A fuzzy set A of X associates each value x of Xto a real number in the interval [0, 1], with the value of fA(X) at x representing the grade of membership of x in A. So, we defi ne an appropriate fuzzy partition (a set of fuzzy sets) of X and connect each fuzzy set of the parti-tion with a subconcept of the terminology, defi ned by that fuzzy set’s property. For example, having extracted the value 2.8H for the length of hair (s1), we determine that s1 has a membership value of 0.7 for AboveEars Hair and 0.3 for BelowEarsHair (see the graph in the semantic-

interpretation box in Figure 2). The defi nition of the fuzzy partitions is based on the heuristics extracted from Dionysios’s manual.

Knowledge representation and reasoningThis subsystem consists of terminolog-ical and assertional knowledge and a reasoning engine.

These types of knowledge are the basic components of a knowl-edge-based system based on DLs,9 a structured knowledge-representationformalism with decidable-reasoning algorithms. DLs have become popular,

especially because of their use in the Semantic Web (as in OWL DL, for example). DLs represent a domain’s impor-tant notions as concept and role descriptions. To do this, DLs use a set of concept and role constructors on the ba-sic elements of a domain-specifi c alphabet. This alphabet consists of a set of individuals (objects) constituting the do-main, a set of atomic concepts describing the individuals, and a set of atomic roles that relate the individuals. The concept and role constructors that are used indicate the ex-pressive power and the name of the specifi c DL. Here, we use SHIN, an expressive subset of OWL DL that employs concept negation, intersection, and union; existential and universal quantifi ers; transitive and inverse roles; role hier-archy; and number restrictions.

Terminological and assertional knowledge. The termino-logical knowledge is an ontological representation of the knowledge in Dionysios’s manual. The assertional knowl-edge is a formal set of assertions describing a specifi c icon in terms of the terminological knowledge. For example, the terminology contains axiomatic knowledge such as “a Jesus Face is depicted as a young face with long, straight, thick,

the module determines

information about

the sacred fi gure’s age

by computing the

number of wrinkles

on the forehead.




black hair and a short, straight, thick, black beard,” whereas “the face depicted in the icon is a face with long hair in some degree” is a relative assertion.

The knowledge base has two main components. The terminological component (TBox) describes the relevant notions of the application domain by stating properties of concepts and roles and their interrelations. TBox is actu-ally a set of concept inclusion axioms of the form C v D, where D is a superconcept of C, and concept equivalence axioms of the form C ≡ D, where C is equivalent to D.

The assertional component (ABox) describes a concrete world by stating properties of individuals and their spe-cific interrelations. To deal with the vagueness introduced in our case, we use f-SHIN, a fuzzy extension of SHIN.3 In f-SHIN, ABox is a finite set of fuzzy assertions of the form ⟨a : C⋈n⟩, ⟨(a, b) : R⋈n⟩, where ⋈ stands for ≥, >, ≤, or <, for a, b ∈ I. Intuitively, a fuzzy assertion of the form ⟨a : C ≥ n⟩ means that the membership degree of a to concept C is at least equal to n. (A formal definition of the syntax and semantics of fuzzy DLs appears elsewhere.3,10)

In the Byzantine-icon-analysis domain, the individuals are the segments that the semantic-segmentation module extracted. The DL concepts classifying the individuals are determined by the properties measured as features during feature extraction (for example, BlackHair and AboveE-arsHair). The DL roles mainly describe partonomic (mereological) hierarchies and spatial relations (for ex-ample, the axiom isLeftOf– = isRightOf indicates that the role isLeftOf is the inverse of isRightOf, and the axioms Trans(isLeftOf) and Trans(isAboveOf) indicate that isLeftOf and isRightOF are transitive).

The terminology’s main concepts are Figure, Face, and FaceComponent. They’re connected with the partonomic role hasPart and its subroles hasSegment and hasCom-ponent, to indicate whether a segment is a subsegment of another segment and so on. Using these roles, we describe the partonomic hierarchy as follows:

Face ≡ ∀hasComponent.FaceComponent

Figure ≡ ∃hasSegment.Face

FaceComponent v Beard t Cheek t Forehead t Hair t Mustache

We describe relevant assertions as follows:

⟨s:Face ≥ 1.0⟩

⟨s1:Hair ≥ 1.0⟩

⟨(s, s1):hasSegment ≥ 1.0⟩

…

The taxonomy is constructed using the properties mea-sured as features during feature extraction. Here are some examples of terminological axioms defining the concept hierarchy:

AboveEarsHair v Hair

OldCheek v Cheek

OldForehead v Forehead

HairyBeard v Beard

Restrictions concerning the taxonomy mainly describe dis-jointness and exhaustiveness, forming axioms such as

AboveEarsHair v ¬BelowEarsHair

Hair v AboveEarsHair t BelowEarsHair

Using these concepts, we define more expressive descrip-tions that specify some specific face characteristics on the basis of Dionysios’s manual. For example, we define a young man’s face as

YoungMansFace ≡ ∃hasComponent.BlackHair u ∃hasComponent.(NotVeryLongBeard u BlackBeard) u ∃hasComponent.YoungForehead u ∃hasPart.YoungCheek u ∃hasPart.BlackMustache

We similarly define concepts specifying the characteris-tics of some popular sacred figures. For example, we define JesusFace as

JesusFace ≡ YoungMansFace u ∃hasComponent.(BraidOnTheLeftShoulder u HairyHair u StraightHair) u ∃hasComponent. (CircleLikeBeard u StraightBeard)

The reasoning engine. This module uses the terminologi-cal knowledge and the assertions to recognize the sacred figure. Our system uses FiRE, the Fuzzy Reasoning Engine (www.image.ece.ntua.gr/~nsimou/FiRE), which fully sup-ports f-SHIN.

Figure 3 illustrates how the engine performs reasoning and classification for recognizing Jesus as the sacred figure. We use FiRE’s greatest lower bound (GLB) reasoning ser-vice, which evaluates an individual’s greatest possible degree of participation in a concept. The concepts of interest in this case, YoungMansFace and JesusFace, only include conjunc-



tions, so we use the fuzzy intersection operator for the mini-mum. So, the GLB of segment s (the face) in YoungMansFace is 0.82, whereas the GLB of segment s in JesusFace is 0.78. In other words, the extracted features indicate that the face depicted is YoungMansFace to a degree of 0.82 and is Je-susFace to a degree of 0.78.

ResultsWe evaluated our system on a database, provided by the Mount Sinai Foundation in Greece, containing 2,000 digi-tized Byzantine icons dating back to the 13th century. The icons depict 50 different characters; according to Diony-sios, each character has specific facial features that makes him or her distinguishable.

Evaluation of the Byzantine-icon-analysis subsystem pro-duced promising results. The subsystem’s mean response time was approximately 15 seconds on a typical PC.

In the semantic-segmentation module, the face detection submodule reached 80 percent accuracy. In most cases, the failure occurred in icons with a destroyed face area. If the submodule detected the face, it almost always detected the

eyes and nose. The base-color-analysis submodule defined the color model used by the icon’s artist, which the fea-ture extraction module later used. Using the nose’s height, which can be easily determined, and taking into account the manual’s rules, the face-component-detection submod-ule estimated the positions of the background and fore-ground pixels for every part of the face. These positions constituted the input to the graph-cut algorithm. This sub-module achieved 96 percent accuracy.

The feature extraction module extracted features for every icon segment. Then, using the fuzzy partitions, the semantic-interpretation module interpreted each segment’s specific features and properties and the relationship of face parts. Thus, that module determined each feature’s degree of membership in a specific class, thereby creating an image description.

To evaluate overall system performance, we used precision and recall. Table 1 (on the next page) pres-ents results for 20 of the sacred-figure classes. The more distinctive facial features the sacred figure in a class contained, the better our method performed (for

YoungMansFace � �hasComponent.BlackHair � hasComponent.�.(NotVeryLongBeard � BlackBeard)� �hasComponent.YoungForehead� �hasPart.YoungCheek � �hasPart.BlackMoustache

JesusFace � YoungMansFace ��hasComponent.(BraidOnTheLeftShoulder � HairyHair � StraightHair)� �hasComponent.(CircleLikeBeard� StraightBeard)

s: Faces1: Hair …(s, s1): hasComponent …

s1: BraidOnTheLeftShoulder = 0.78s1: HairyHair = 0.96s1: StraightHair = 0.86s1: BlackHair = 0.88s2: YoungForehead = 0.82s3: YoungCheek = 0.87s4: StraightMoustache = 1.0s4: BlackMoustache = 0.83s5: CircleLikeBeard = 0.79s5: StraightBeard = 0.92s5: NotVeryLongBeard = 1.0s5: BlackBeard = 0.84

s1

s2

s3

s4

s5

JesusFace

Reasoningengine

Knowledge representationand reasoning

ABox

TBox

Figure 3. A reasoning example. The extracted information from image analysis constitutes the assertional component (ABox) of the knowledge base; the terminological component (TBox) is defined on the basis of Fourna’s specification. These components form the input to the fuzzy reasoning engine, which infers information about the icon.




example, for the Jesus class). Conversely, our method performed worse on figures with similar facial features, such as women.

We’re developing a version of our system that takes into account information about clothes associated

with specific types of figures (such as apostles, kings, an-gels, and monks) and clothing accessories (kerchiefs and so on) and nonclothing accessories (books, crosses, and so on) associated with specific sacred figures. For example, the system could incorporate the knowledge that icons of the Virgin Mary depict her wearing a kerchief with three stars. Such information might help in classifying figures with similar facial features.

Furthermore, an integrated version of our system could eventually date icons and attribute them to a particular artist or artistic school, on the basis of features specific to each artistic period. Such information can help classify an icon even if the specific sacred figure is difficult to rec-ognize. We’re also investigating applying our method to other art movements such as impressionism, trying to clas-

sify paintings on the basis of unmixed colors and dense textures.

AcknowledgmentsOur research has been partly supported by the MORFES (extraction and modeling of figure in Byzantine paintings of Mount Sinai monu-ments using measures) project, funded by the EU and the Greek Sec-retariat for Research and Development. We thank the Mount Sinai Foundation for providing access to the rich data set and especially archeologist Dimitris Kalomirakis for his participation in knowl-edge extraction and his continuing support during this research. Finally, we thank the anonymous reviewers for their constructive comments.

References 1. T. Berners-Lee, J. Hendler, and O. Lassila, “The Semantic

Web,” Scientific American, May 2001, pp. 34–43. 2. Dionysios tou ek Phourna, Hermeneia tis zografikis technis

(interpretation of the Byzantine art), B. Kirschbaum, 1909. 3. G. Stoilos et al., “Reasoning with Very Expressive Fuzzy De -

scription Logics,” J. Artificial Intelligence Research, vol. 30, no. 5, 2007, pp. 273–320.

4. P. Viola and M.J. Jones, “Robust Real-Time Face Detection,” Int’l J. Computer Vision, vol. 57, no. 2, 2004, pp. 137–152.

5. S. Asteriadis et al., “An Eye Detection Algorithm Using Pixel to Edge Information,” Proc. 2nd IEEE-EURASIP Int’l Symp.

Table 1. Evaluation of the icon classification system.

Class No. of icons in class No. of images classifiedNo. of images

correctly classified Precision Recall

Jesus 70 67 58 0.87 0.83

Virgin Mary 60 54 43 0.80 0.72

Peter 50 44 38 0.86 0.76

Paul 50 44 39 0.89 0.78

Katerina 40 35 30 0.86 0.75

Ioannis 40 37 29 0.78 0.73

Luke 40 33 28 0.85 0.70

Andrew 40 33 29 0.88 0.73

Stefanos 30 27 24 0.89 0.80

Konstantinos 40 36 30 0.83 0.75

Dimitrios 40 36 31 0.86 0.78

Georgios 40 37 32 0.86 0.80

Eleni 40 37 31 0.84 0.78

Pelegia 20 17 15 0.88 0.75

Nikolaos 40 38 33 0.87 0.83

Basileios 40 37 31 0.84 0.78

Antonios 30 27 20 0.74 0.67

Eythimios 25 23 18 0.78 0.72

Thomas 35 32 26 0.81 0.74

Minas 20 17 14 0.82 0.70



Control, Communications, and Sig-nal Processing (ISCCSP 06), 2006; www.eurasip.org/Proceedings/Ext/ISCCSP2006/defevent/papers/cr1124.pdf.

6. N.A. Otsu, “A Threshold Selection Method from Gray-Level Histograms,” IEEE Trans. Systems, Man, and Cyber-netics, vol. 1, no. 9, 1979, pp. 62–66.

7. Y. Boykov and M.-P. Jolly, “Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images,” Proc. Int’l Conf. Computer Vision (ICCV 01), vol. 1, IEEE Press, 2001, pp. 105–112.

8. R.C. Gonzalez and R.E. Woods, Digi-tal Image Processing, 3rd ed., Prentice Hall, 2008.

9. F. Baader et al., The Description Logic Handbook: Theory, Implementation and Applications, Cambridge Univ. Press, 2002.

10. G. Stoilos et al., “Uncertainty and the Semantic Web,” IEEE Intelligent Sys-tems, vol. 21, no. 5, 2006, pp. 84–87.

For more information on this or any other computing topic, please visit our Digital Li-brary at www.computer.org/csdl.

t H e A u t H o r sparaskevi Tzouveli is pursuing her PhD in image and video analysis at the National Tech-nical University of Athens. Her research interests include image and video analysis, in-formation retrieval, knowledge manipulation, e-learning, and digital cultural heritage systems. Tzouveli received her Diploma in electrical and computer engineering from the National Technical University of Athens. Contact her at [email protected].

Nikos Simou is a PhD student in the National Technical University of Athens Depart-ment of Electrical and Computer Engineering and a member of the Image, Video, and Multimedia Systems Laboratory’s Knowledge Technologies Group. His research interests include knowledge representation under certain and uncertain or imprecise knowledge based on crisp and fuzzy description logics, ontologies, and the implementation and op-timization of reasoning systems and applications for the Semantic Web. Contact him at [email protected].

Giorgios Stamou is a lecturer in the National Technical University of Athens School of Electrical and Computer Engineering and a member of the university’s Image, Video, and Multimedia Systems Laboratory, heading the lab’s Knowledge Technologies Group. His main research interests include knowledge representation and reasoning, theory of uncer-tainty, machine learning, semantic interoperability, and digital archives. Contact him at [email protected].

Stefanos Kollias is a professor in the National Technical University of Athens (NTUA) School of Electrical and Computer Engineering and the director of the Image, Video, and Multimedia Systems Laboratory. His research interests include image and video process-ing, analysis, retrieval, multimedia systems, computer graphics and virtual reality, artificial intelligence, neural networks, human-computer interaction, and digital libraries. Kollias received his PhD in signal processing from NTUA. He’s a member of the European Com-mission expert group in digital libraries and of the EC committee on semantic interoper-ability, leading NTUA participation in DL projects, EDLnet, VideoActive, Europeana Connect, Europeana Network 1.0, and EU Screen. Contact him at [email protected].

Advertising Sales Representatives

Recruitment:

Mid Atlantic Lisa RinaldoPhone: +1 732 772 0160Fax: +1 732 772 0164Email: [email protected]

New EnglandJohn RestchackPhone: +1 212 419 7578Fax: +1 212 419 7589Email: [email protected]

SoutheastThomas M. FlynnPhone: +1 770 645 2944Fax: +1 770 993 4423Email: flynntom@ mindspring.com

Midwest/SouthwestDarcy GiovingoPhone: +1 847 498 4520Fax: +1 847 498 5911Email: [email protected]

Northwest/Southern CA Tim MattesonPhone: +1 310 836 4064Fax: +1 310 836 4067Email: tm.ieeemedia@ ieee.org

JapanTim MattesonPhone: +1 310 836 4064Fax: +1 310 836 4067Email: [email protected]

EuropeHilary TurnbullPhone: +44 1875 825700Fax: +44 1875 825701Email: impress@ impressmedia.com

Product:

US East Joseph M. DonnellyPhone: +1 732 526 7119Email: [email protected] US CentralDarcy GiovingoPhone: +1 847 498 4520Fax: +1 847 498 5911Email: [email protected]

US WestLynne StickrodPhone: +1 415 931 9782Fax: +1 415 931 9782Email: [email protected]

EuropeSven AnackerPhone: +49 202 27169 11Fax: +49 202 27169 20Email: sanacker@ intermediapartners.de

Advertising PersonnelMarion DelaneyIEEE Media, Advertising Dir.Phone: +1 415 863 4717 Email: md.ieeemedia@ ieee.org

Marian AndersonSr. Advertising CoordinatorPhone: +1 714 821 8380 Fax: +1 714 821 4010 Email: manderson@ computer.org

Sandy BrownSr. Business Development Mgr.Phone: +1 714 821 8380 Fax: +1 714 821 4010 Email: sb.ieeemedia@ ieee.org

AdveRtiSeR infoRmAtionMarch/april 2009 • IEEE IntEllIgEnt SyStEmS


semantic classification of byzantine iconsreformat/ece720w2012/papers/ieeexplore... · byzantine...

Documents