Sabanci-Okan System at ImageClef 2012:Combining Features and Classifiers for Plant

Identification

Berrin Yanikoglu1, Erchan Aptoula2, and Caglar Tirkaz1

1 Sabanci University, Istanbul, Turkey 349562 Okan University, Istanbul, Turkey, 34959

{berrin,caglart}@sabanciuniv.eduerchan.aptoula@okan.edu.tr

Abstract. We describe our participation in the plant identification taskof ImageClef 2012. We submitted two runs, one fully automatic andanother one where human assistance was provided for the images inthe photo category. We have not used the meta-data in either one ofthe systems, for exploring the extent of image analysis for the plantidentification problem. Our approach in both runs employs a varietyof shape, texture and color descriptors (117 in total). We have foundshape to be very discriminative for isolated leaves (scan and pseudo-scan categories), followed by texture where mathematical morphologyhas been our main methodology. While we have used color, we did notfound color information to be very useful. Mathematical morphology isalso used in the watershed algorithm used for segmentation, in slightlydifferent forms for automatic and human assisted systems. Our systemshave obtained the best overall results in both the automatic and manualcategories, with 43% and 45% identification accuracies respectively. Wehave also obtained the best results on the scanned image category, with58% accuracy, while the 2nd best system had a 49% accuracy.

Keywords: Plant identification, mathematical morphology, classifiercombination, support vector machines.

1 Introduction

A content-based image retrieval (CBIR) system for plants would be very usefulfor plant enthusiasts or botanists who would like to learn more about a plant theyencounter. Until the first ImageCLEF Plant Identification Competition orga-nized in 2011 [6], existing research concentrated on isolated leaf identification [5,3, 9, 12, 18, 16], while few systems attempted the identification of unconstrainedwhole or partial plant images [8, 15].

As with the first competition, the plant identification task in ImageCLEF2012 consisted of identifying images of plants that were captured by differentmeans: scans, scan-like photos (called pseudo-scans) and unrestricted photos,

2 Yanikoglu et al.

as shown in Fig. 1f. In this way, the competition aimed to benchmark state-of-the-art in both isolated leaf shape and unrestricted plant image recognitionproblems. The details of this competition are described in [7].

Content-based plant identification problem faces many challenges such ascolor, illumination, size variations that are also common in other CBIR problems,as well as some specific problems such as the variations in the composition of theleaves that makes the plant shape variable. In addition, one can see that color isless identifying in the plant retrieval problem compared to many other retrievalproblems, since most plants have green tones as their main color with subtledifferences. In the rare cases that color is discriminative for a certain plant, thatis when the plant has an unusual color, then it may be the case that the leavesof that plant may also have other colors, due to individual plant or seasonalvariations.

While shape is quite discriminative in identifying isolated leafs, it is not asuseful in identifying full plant images, since the global shape of a plant is affectedfrom its leaf composition [8, 17]. In that regard, isolated leaf identification is asimpler problem compared to the unconstrained images of full or partial plants.

(a) (b) (c) (d) (e) (f)

Fig. 1: Samples from scan (a,b), pseudo-scan (c,d) and photo categories (e,f).

2 Overview

We submitted two runs, one fully automatic and another one where humanassistance was provided for the images in the photo category. The only distinctionbetween our two runs (automatic and human-assisted) is that the images in thephoto category undergoes a human-assisted segmentation process, as explainedin 3.1. Hence, in the remainder of this paper, we will talk about one system, sinceeverything besides the segmentation of photos are the same in the two submittedruns. The final system is designed as two separate sub-systems, one for scan andscan-like images and another one for photos. Since the meta-data included theacquisition type, an input image is automatically sent to the correct subsystem.The acquisition method was the only meta-data used in the overall system. Oursystem shares many common parts with last year’s system, described in [17]. Inthis paper, we give a summary of our system, with detailed descriptions of thechanges done this year.

Sabanci-Okan System at ImageClef 2012 Plant Identification Competition 3

Last year, we had concentrated our efforts almost exclusively to scanand pseudo-scan images, while photos were neglected, due to lack of time.This year, we developed automatic and human-assisted segmentation algorithmsfor the photo category. In both segmentation algorithms, we aimed to over-segment the image to leave only a single leaf, in order to use our isolated leafrecognition engine. This approach was reasonably successful, in that our systemhad obtained 5.3% accuracy in photos last year, while this year the automaticphoto identification accuracy was 16%.

For recognizing photographs, we found that global shape descriptors andmany of the descriptors considered in the present work, would not be useful ifthe photo consisted of an overlapping set of leaves, rather than a single leaf. Onthe other hand for scan and scan-like categories, all three main feature categoriesare useful: color, texture and shape. After experimenting with a large number ofdescriptors, we selected a 115-dimensional feature vector for the scan/scan-likesub-system and its 91-dimensional subset for the photo category. The featuresused in our system are explained in Section 5.

Another major problem we encountered last year was the over-fittingproblem: our results obtained in cross-validation experiments on the trainingset and those on the test set were wildly different (around 40%). This year ourfeature selection and classifier optimization approaches were run on a separatetest set which was designed not to include any images similar to those in thetraining set. For training the system, we trained a classifier combination usingSupport Vector Machines (SVMs), as explained in Section 6.2.

3 Segmentation

The ultimate goal of segmentation is the separation of the leaf from itsbackground. If no a priori knowledge is available about the image, the segmen-tation problem becomes equivalent to unconstrained color image segmentation,where the various leaves can be located within an equally immense variety ofbackgrounds, such as those shown in Fig. 2.

One might argue at this stage that the background can provide informa-tion facilitating the recognition of the leaf/plant in the foreground. For instanceif the background consists of forest ground versus the sky, we can eliminatesome alternatives as implausible. However, this would additionally require thedescription and recognition of the background, which given its limitless diversitypotential, increases the complexity of an already challenging problem.

The accurate separation of the background from the plant/leaf underconsideration is of crucial importance for the subsequent stage of featureextraction, since a poor result would affect many of the features. Given theill-posed nature of segmentation, in addition to the lack of any useful a prioriknowledge, it becomes evident that some form of human intervention or feedbackis necessary for an accurate and reliable segmentation. However, we also haveto take into account practical considerations, since no user/expert would like tospend a prolonged amount of time on this stage, especially when dealing with

4 Yanikoglu et al.

voluminous amounts of data. That is why we have explored both automatic andhuman assisted segmentation strategies.

(a) (b) (c)

Fig. 2: Examples of background variations in leaf photos.

Automatic segmentation Although the ImageClef dataset is classified intothree categories, namely scan, pseudo-scan and photos, from a segmentationviewpoint scan and pseudo-scan type images are of similar quality 1f, inother words they both possess a mostly noise-free, spectrally homogeneousbackground, occasionally containing some amount of shadow in the pseudo-scans. Consequently, as far as scan and pseudo-scan type images are concerned,automatic segmentation has been trivially resolved through Otsu’s method [10]as it is both efficient and effective (Fig. 3).

(a) (b) (c)

Fig. 3: Leaf isolation example with scan and pseudo-scan type images usingOtsu’s method. (a) Original image (b) Segmentation mask (c) Mask super-imposition on the original.

Photos on the other hand consist of unconstrained acquisitions of leaves, asshown in Figs. 2a-2c, and thus their automatic segmentation is a far greaterchallenge. In order to counter this problem we adopted a combination of spectraland spatial techniques.

More precisely the only assumption concerning the input has been thatthe object of interest, i.e. the leaf is located roughly at the center of the imageand possesses a single dominant color. The image was first simplified by meansof marginal color quasi-flat zones [11], a morphology based image partitioningmethod based on constrained connectivity, only recently extended to color data

Sabanci-Okan System at ImageClef 2012 Plant Identification Competition 5

[14], that creates flat zones based on both local and global spectral variationalcriteria (Fig. 4a). Next we computed its morphological color gradient in the LSHcolor space [1], taking into account both chromatic and achromatic variations(Fig. 4b), followed by the application of the watershed transform. Hence weobtained a first partition with spectrally homogeneous regions and spatiallyconsistent borders, albeit with a serious over-segmentation ratio which wascompensated for by merging basins below a certain area threshold (Fig. 4c).

At this point our initial assumption about the object of interest’s centrallocation was used, as we employed the central 2/3 area of the image in orderto determine its dominant color, which was obtained by means of histogramclustering in the LSH color space. Assuming that the mean color of the mostsignificant cluster (i.e. reference color) belongs to the leaf/plant, we thenswitched to spectral techniques, so as to determine its watershed basins. Sincecamera reflections can be problematic due to their low saturation, we computedboth the achromatic, i.e. grayscale distance image from the reference gray(Fig. 4d) and the angular hue distance image (Fig. 4f) from the reference huehref [2]:

∀ h, href ∈ [0, 2π], dθ(h, href ) =

{|h− href | if |h− href | < π2π − |h− href | if |h− href | ≥ π

(1)

We then applied Otsu’s method on both distance images, that provided us withtwo masks (Figs. 4e & 4g), representing spectrally interesting areas w.r.t. thereference color. The intersection of the two masks was used as the final objectmask (Figs. 4h). As shown in Fig. 4i, spectral and spatial techniques indeedcomplement each other well, while the use of both chromatic and achromaticdistances increases the method’s robustness. However the main difficulty is theaccurate determination of the reference or dominant color; this can easily corruptthe entire process if computed incorrectly or if the leaf under consideration hasmore than one dominant colors.

3.1 Human assisted segmentation

As the automatic segmentation module relies strongly on its initial assumption, itdoes not work very well, resulting in over or wrong segmentation. We investigatedhuman assisted segmentation for the semi-manual category of the competition.

Ideally, given an input image we would like the user/expert to spendat most a few tens of seconds in order to provide some amount of high levelknowledge. Assuming that the provided knowledge is valid, we chose to use themarker based watershed transform [4]. Specifically, the marker based watershedtransform is a powerful, robust and fast segmentation tool that given a numberof seed areas or markers, results in watershed lines representing the skeleton bytheir influence zones.

To accomplish this we need a suitable topographic relief as input, whereobject borders are denoted as peaks and flat zones as valleys; which is why weemploy the morphological color gradient of the input image [1] (Fig. 5b). We

6 Yanikoglu et al.

(a) Color quasi flatzones

(b) Color gradient (c) Watershed trans-form and removal ofsmall basins

(d) Grayscaledistance

(e) Grayscale mask (f) Hue distance

(g) Hue mask (h) Mask intersection (i) Mask superposi-tion on the original

Fig. 4: Stages of automatic photo segmentation of Fig. 2c.

then require at least two markers provided by the user/expert, one denotingthe background and another representing the foreground, both of which couldbe easily provided for instance through the touchscreen of a smartphone byindicating once the leaf and once its background (Fig. 5c). Next, havingsuperposed the markers on the gradient, the marker based watershed transformprovides the binary image partition (Fig. 5d). Although both efficient andeffective, this method depends utterly on the quality of the provided markers,since if they are too small they can lead to partial leaf detection and converselyif they are excessively large, the leaf will be confounded with its background.

(a) Original im-age

(b) Morphologi-cal gradient

(c) The markers (d) Result

Fig. 5: Stages of human assisted photo segmentation on image #4835.

Sabanci-Okan System at ImageClef 2012 Plant Identification Competition 7

4 Preprocessing

After the segmentation, we have a single leaf which is segmented from itsbackground, whatever is the type of the original image (scans or photos). Forthe case of photos, this isolated image is well segmented in the case of human-assisted segmentation, except possiblly for some noise at the contour; in the caseof automatic segmentation, the leaf may be over or under-segmented. Note thatin all photos, the segmented image may also be rotated in an arbitrary direction,while most of the scanned leaves are oriented with their axes aligned with thevertical, stem part down.

Our preprocessing consists of orientation normalization for photos,followed by height normalization (keeping the height/width ration unchanged).We have also experimented with stem location and orientation normalization forscans and pseudo-scans, however these steps are not yet mature, causing errorsin as many images as the ones they correct.

5 Feature Extraction and Selection

In a difficult problem such as this, it was clear that we needed to use all mainfeature categories (shape, texture and color), even though some of them wereexpected to be more useful than others. We have kept most of the features thatwere used in last years’s competition, and we added some new ones that wethough would be useful.

Considering the intra-class color variations levels, it became evident atan early stage that our approach would have to be mainly texture and shapebased, even though the discriminatory potential of color has not been ignoredcompletely. Moreover, it was chosen not to employ any further complicateddescriptors, such as scale invariant feature transforms (SIFT) or maximallystable extremal regions (MSER), for two reasons. First, our past experiencewith plant retrieval [8] has led to results indicating them as not very suitablefor this content type, and second due to time limitations, that hindered furtherefforts to test more sophisticated descriptors.

In total, we evaluated about 20 different shape, texture and color features,consisting of region-based (convexity, and contour-based (FFT,border covari-ance) shape descriptors; texture descriptors (morphological, Gabor, gradientorientation histograms); and color (color moments and different length andquantization of the color histogram in LSH space). Here we list the new featuresbriefly, while others are described in detail in [17].

1. Convex Perimeter Ratio returns the ratio of the perimeter of the convex hullto the perimeter of the original binary mask.

2. Regional Moments of Inertia is computed on grayscale data. We first dividethe input into n horizontal slices as in area width factor and then computeindependently for each slice the mean distance to the image centroid.

8 Yanikoglu et al.

3. Angle Code Histogram is computed on binary data. We first compute theobject contour, which is then subsampled. We then proceed to calculate theangle of every point triplet and return as feature their normalized histogram[13].

4. Contour Point Distribution Histogram is applied on the binarized internalmorphological gradient of the input image. Given n concentric disks centeredat the image centroid, we calculate for each disk the percentage of gradientpixels in it [?].

5. Orientation Histogram is computed on grayscale data. We first computethe orientation map using a 11 × 11 mask for determining the dominantorientation of each pixel. The feature vector consists of the normalizedhistogram of dominant orientations.

6. Lobe Descriptor is calculated by over-segmenting the scanned images andextracting the median value of convexity, elongation etc. parameters of thesegmented lobes.

Note that tour final list of features were selected in a process of evaluatinglast year’s features and observing the shortcomings of the system (e.g. the specieswith more errors), and adding more features thought to remedy the shortcoming.For instance as a result we added more features (e.g. lobe descriptor).

In the final training of the classifiers, we used all of the selected featuresforming a 116-dimensional feature vector for scans and pseudo-scans; but discardmost shape descriptors for photos. In particular, for automatic segmentation, itwas common that the shape was not preserved, so no shape descriptors wereused; while for human assisted segmentation, we discarded only those shapedescriptors expecting a clean boundary.

6 Classifiers Training

6.1 Training and Testing Data

The ImageClef database contains 126 tree species and the images for each speciesare contributed by various people, some of the images being taken from the sameindividual plant. A straightforward cross-validation done over the whole trainingset thus results in splitting the often similar images of the same plant, intotraining and test sets. We believe this is the main reason for the over-fitting thatwe experienced in last year’s system. Consider that the images of an individualplant, often captured with the same lighting and camera parameters, are splitinto training and test sets. In this case, color and texture descriptors remainquite similar between two images, leading to overfitting.

This year, we took a different approach and divided the database intotraining and testing, rather than using cross-validation that may result in arandom split of very similar images. In short, we put all of the images of anindividual plant either in training or testing. Specifically, for each of the species,if the species images contained two or more individual plants, one of them (theone with the least number of pictures) is separated for testing while all others

Sabanci-Okan System at ImageClef 2012 Plant Identification Competition 9

are selected for training. In total 5163 scan and scan-like images are used fortraining and 1526 scan and pseudo-scan images and all photos (1733 images) areused for testing during our experiments.

Note that this approach has the disadvantage that some of the specieshaving images from a single individual plant (XXX), did not have anyrepresentative images in test data. However, in terms of feature selection, wethought that this shoud not have drastic effects, as features can be said to bespecies-independent.

6.2 Classification

We trained two classifiers, one for scans (scan and pseudo-scan categories) andone for photos. Depending on whether the photos were automatically or manuallysegmented only changed the resulting category of the run as automatic or humanassisted.

For the classifier, we used an SVM using the radial basis function kernel.The input to the classifier was the feature vector constructed according to imagetype (scans or photo), obtained from scan and pseudo-scan images. Notice thatwe only used these images as training data, for both the scan classifier and thephoto classifier, as many features would not be cleanly extracted from photos.

This year, the ambiguity resolution step, where we used a furtherclassifier trained to distinguish between the top-5 choices, is skipped, as it wasobserved that it was significantly extending the test stage without significantimprovement.

6.3 Feature Selection

Having separated our training set into two, training and testing, we evaluatedour last year’s features [6] by training on the training set and testing on the testset, rather than doing cross-validation on the original training set. This processgave us the chance of observing the relative merits of different features, but moreimportantly, we got a chance of seeing the errors made and added new featuresto better discriminate between very similar plants.

The results of the tests showing individual features length and contribu-tion are shown in Table 1 and Table 2, for shape and texture features respectively.As seen in these tables, cross-validation accuracies are almost always higher thantest set accuracies, and especially so for texture features.

7 Experimental Results

During experiments, we considered the scan and pseudo-scans as one group andtreated them using the same methodology, while for photos we employed differentsegmentation algorithms. We evaluated the efficiency of our features and thecreated classifiers on the scan group. On the other hand, we divided the photosinto 4 categories as per the provided categorization [7]: i) picked leaf ii) leaf

10 Yanikoglu et al.

Feature Name Length Cross Val. Acc.(%) Test Acc.(%)

FFT 50 49.95 46.26Lobe Descriptor 7 48.65 43.97

Regional Moments 7 48.58 41.74Area Width Factor 13 49.89 38.79

Shape Statistics 4 39.66 34.34Contour Pnt. Dist. Hist. 7 33.16 30.14

Ang. Code Hist. 10 33.76 26.61Morph. Elongation 1 9.93 11.66Convex Area Ratio 1 11.50 11.60Convex Per. Ratio 1 12.36 11.14

Compactness 1 11.47 9.24All features 95 77.44 67.89

Table 1: Individual shape feature accuracies

Feature Name Length Cross Val. Acc.(%) Test Acc.(%)

Gabor 8 26.26 15.60Orientation Hist. 6 38.81 24.64

Smoothness 1 7.22 5.44All features 15 66.26 33.42

Table 2: Individual texture feature accuracies

iii) branch iv) leafage, containing roughly a picked single leaf on a variety ofbackgrounds, a leaf which may be hanging from the branch, and plant foliagewith or without branches.

Figure 6 shows the results given in Tables 1 and 2 as graphs, for scanand pseudo-scan images. It also displays the top-8 accuracies for each feature.We considered the top-k accuracy since the identification performance is stillrelatively low, yet users of such a system may be willing to peruse a short list ofsuggestions to select the correct plant.

Recognition results for photos for different categories for the automaticsegmentation method is given in Table 3. As expected, photos perform muchworse compared to scan categories, and different photo categories differ indifficulty level, picked leaf images being the easiest, pointing to the difficultyof segmentation.

Photo type Accuracy (%)

Picked Leaf 36.47Leaf 15.49

Branch 8.33Leafage 4.43All types 15.06

Table 3: Recognition accuracies for photos using automatic segmentation.

Sabanci-Okan System at ImageClef 2012 Plant Identification Competition 11

Finally, using all the features (117-long feature vector), we obtained a85.9% accuracy on cross-validation tests and 66.1% accuracy over the test data.Note that our accuracies are measured as average accuracy over the test images,while the official scoring function computes an average accross users people whohave collected the images

8 ImageCLEF2012 Results and Discussion

This year the organizers categorized automatic and manual or human-assistedruns separately, while this information was not clear in last year’s results.According to the official results (Table 4), our runs achieved 1st place inoverall classification, for both automatic and human-assisted categories. We alsoobtained significantly better results on the scan category, compared to the nextbest system in this category. Note that we only give a partial results table,including only the best automatic systems in addition to our two systems.

Table 4: Partial plant retrieval results showing the top-performing systemsordered according to mean performances.

RunAuto./Manual

Scan Scan-like Photo Mean

Sabanci Okan-run2 Manual 0.58 0.55 0.22 0.45Sabanci Okan-run1 Automatic 0.58 0.55 0.16 0.43INRIA Imedia plantnet run1 Automatic 0.49 0.54 0.22 0.42INRIA Imedia plantnet run2 Automatic 0.39 0.59 0.21 0.40LSYS DYNI run 3 Automatic 0.41 0.42 0.32 0.38ARTELAB run 1 Automatic 0.40 0.37 0.14 0.30

References

1. E. Aptoula and S. Lefevre. A basin morphology approach to colour imagesegmentation by region merging. In Proceedings of the Asian Conference inComputer Vision, volume 4843, pages 935–944, Tokyo, Japan, November 2007.

2. E. Aptoula and S. Lefevre. On the morphological processing of hue. Image andVision Computing, 27(9):1394–1401, August 2009.

3. A. R. Backes and O. M. Bruno. Shape classification using complex network andmulti-scale fractal dimension. Pattern Recognition Letters, 31(1):44 – 51, 2010.

4. S. Beucher and F. Meyer. The morphological approach to segmentation: thewatershed transformation. In E. R. Dougherty, editor, Mathematical Morphologyin Image Processing, pages 433–482. Dekker, New York, 1993.

5. O. M. Bruno, R. O. Plotze, M. Falvo, and M. Castro. Fractal dimension appliedto plant identification. Information Sciences, 178(12):2722 – 2733, 2008.

6. H. Goeau, P. Bonnet, A. Joly, N. Boujemaa, D. Barthelemy, J. Molino,P. Birnbaum, E. Mouysset, and M. Picard. The clef 2011 plant image classificationtask. In CLEF 2011 working notes, Amsterdam, The Netherlands, 2011.

12 Yanikoglu et al.

7. H. Goeau, P. Bonnet, A. Joly, I. Yahiaoui, D. Barthelemy, N. Boujemaa, , andJ. Molino. The imageclef 2012 plant identification task. In CLEF 2011 workingnotes, Rome, Italy, 2012.

8. H. Kebapci, B. Yanikoglu, and G. Unal. Plant image retrieval using color, shapeand texture features. The Computer Journal, 53(1):1–16, April 2010.

9. F. Lin, C. Zheng, X. Wang, and Q. Man. Multiple classification of plant leavesbased on gabor transform and lbp operator. In International Conference onIntelligent Computing, pages 432–439, Shanghai, China, 2008.

10. N. Otsu. A threshold selection method from gray-level histograms. IEEETransactions on Systems, Man and Cybernetics, 9(1):62–66, 1979.

11. P. Soille. Constrained connectivity for hierarchical image partitioning andsimplification. IEEE Transactions on Pattern Analysis and Machine Intelligence,30(7):1132–1145, July 2008.

12. Z. Wang, Z. Chi, and D. Feng. Shape based leaf image retrieval. IEE Proceedings- Vision, Image and Signal Processing, 150(1):34–43, Feb 2003.

13. Z. Wang, Z. Chi, and D. Feng. Shape based leaf image retrieval. IEE Proceedingsin Vision, Image and Signal Processing, 150(1):34–43, February 2003.

14. J. Weber, E. Aptoula, and S. Lefevre. Extension of quasi-flat zones to color images.Computer Vision and Image Understanding, 2012. Under review.

15. D. M. Woebbecke, G. E. Meyer, K. Von Bargen, and D. A. Mortensen. Plantspecies identification, size, and enumeration using machine vision techniques onnear-binary images. In Optics in Agriculture and Forestry, volume 1836, pages208–219, Boston, USA, 1993. SPIE.

16. Itheri Yahiaoui, Nicolas Herve, and Nozha Boujemaa. Shape-based image retrievalin botanical collections. In PCM, pages 357–364, 2006.

17. Berrin Yanikoglu, Erchan Aptoula, and Caglar Tirkaz. Sabanci-okansystem at imageclef 2011: Plant identification task. In CLEF (NotebookPapers/Labs/Workshop), 2011.

18. S. Yonekawa, N. Sakai, and O. Kitani. Identification of idealized leaf types usingsimple dimensionless shape factors by image analysis. Transactions of the ASAE,39:1525–2533, 1996.

Sabanci-Okan System at ImageClef 2012 Plant Identification Competition 13

(a)

(b)

Fig. 6: Classification accuracies using shape and texture features for scan-pseudo-scan images. The blue lines show the classification performances during trainingusing cross validation on the training data. The red and black lines show thetop-1 and top-8 classification accuracies respectively on the separate reservedtest data.