FACE DETECTION: A COMPARISON BETWEEN HISTOGRAM THRESHOLDING AND NEURAL

NETWORKS

JAMAL AHMAD DARGHAM

ýrýý . ._....,.

ýýIVERS11 s

THESIS SUBMITTED FOR THE FULFILMENT OF THE DEGREE OF DOCTOR PHILOSOPHY

SCHOOL OF ENGINEERING AND INFORMATION TECHNOLOGY

UNIVERSITI MALAYSIA SABAH 2008

DECLARATION

I hereby declare that the material in this thesis is my own except for quotations, excerpts, quotations, summaries and references, which have been dully acknowledged.

1 JULY 2008 PERPi1S tn' ,, U,

UNiVERS(Ti MALAYS IA JAMAL AHMAD DARGHAM

PS99-008-081

ii

UNIVERSITI MALAYSIA SABAH

BORANG PPENGESAHAN STATUS TESIS

7UDUL: FACE DETECTION: A COMPARISON BETWEEN HISTOGRAM THRESHOLDING AND NEURAL NETWORKS

IJAZAH: DOKTOR FALSAFAH (IMAGE PROCESSING)

Saya, JAMAL AHMAD DARGHAM mengaku membenarkan tesis doctor falsafah ini disimpan di Perpustakaan Universiti Malaysia Sabah dengan syarat-syarat kegunaan seperti berikut:

1. Tesis adalah hakmilik Universiti Malaysia Sabah 2. Perpustakaan Universiti Malaysia Sabah dibenarkan membuat salinan untuk

tujuan pengajian saya 3. Perpustakaan dibenarkan membuat salinan tesis ini sebagai bahan pertukaran

antara institusi pengajian tinggi 4. TIDAK TERHAD

Pt ii :' US TA KAr'y UNIVERSlTI MAtAYSI. A

Penulis: JAMAL AHMAD DARGHAM

Disahkan oleh

(TANDATANGAN PUSTAKAWAN)

Alamat: 4D Luyang Apartment

Luyang, 88300 Kota Kinabalu.

-ý"'ý (Penyelia: Prof. Madya Dr. Ali Chekima)

Tarikh: 10 Julai 2008

CATATAN: Tesus dimakksudkan sebagai tesis Ijazah Doktor Falsafah dan Sarjana

secara penyelidikan atau disertasi bagi pengajian secara kerja kursus dan

penyelidikan, atau laporan Projeck Sarjana Muda (LPSM)

CERTIFICATION

TITLE : FACE DETECTION: A COMPARISON BETWEEN HISTOGRAM THRESHOLDING AND NEURAL NETWORKS

DEGREE : DOCTOR OF PHILOSOPHY (IMAGE PROCESSING)

DATE OF VIVA : 17 MARCH 2008

DECLARED BY

SUPERVISO (Associate fessor Dr. Ali Chekima)

III

ACKNOWLEDGMENTS

I am very grateful to Allah subhanahu wataala for giving me the physical and mental strength to complete this thesis. I am very grateful for my parents for their love, support and encouragement. I would also to thank my wife and two daughters for their love and understanding throughout the process of completing this thesis. My thank goes also to my supervisor and friend Associate Professor Dr. Ali Chekima for his advice and guidance not forgetting his patience. My sincere thank to my undergraduate students who volunteer to be included in the database. Lastly but not least, my sincere gratitude to the Dean and my colleagues at the School of Engineering and Information Technology, Universiti Malaysia Sabah as well as to the staff and management of Universiti Malaysia Sabah for their help and assistance.

iv

ABSTRACT

Face Detection; A Comparison Between Histogram Thresholding and Neural Networks

Face detection is an important process in many applications such as face recognition, person identification and tracking, and access control. The technique used for face detection depends on how a face is modelled. In this thesis, a face is defined as a skin region and a lips region that meet certain geometrical criteria. Thus, the face detection system has three main components: a skin detection module, a lips detection module, and a face verification module. Multi-layer perceptron (MLP) neural networks and histogram thresholding techniques have been used for skin and lips detection. In order to test the face detection system, two databases were created. The images in the first database, called In-house, were taken under controlled environment while those in the second database, called WWW, were collected from the World Wide Web. Only the skin and the lips colour in the normalised RGB colour scheme were used for the skin and lips detection respectively. A new method for obtaining the r, g, and b components of the normalised RGB systems from the R, G, and B components of the RGB system was proposed. It was found out that the proposed method, called maximum intensity normalisation, gives higher percentage of correct skin detection than the conventional rgb colour scheme regardless of the database used or the skin detection method. Two methods were used to find the number of neurons in the hidden layer of the MLP. The first method use binary search between a minimum and a maximum values while the second method use sequential search with a stopping criteria. The effect of scale factor, facial expressions and minor occlusions with glasses on skin, lips and face detection was investigated. It was found out that, as the scale factor increases the percentage skin and lips detection error decreases. However, the percentage decrease in skin and lips detection errors depends on the intensity normalisation, the detection method and the chrominance component used. But the scale factor did not have any effect on the face detection. In general, the facial expression did not have any significant effect on skin detection. However, for lips detection, the laughing expression did give the highest lips detection error followed by smiling expression. Furthermore, the percentage increase in lips detection error as a result of the facial expression depends on the intensity normalisation, the detection method and the chrominance component used. As for face detection, the facial expression has a negative effect on the correct face detection especially at scale factor of 3. Although, the minor occlusion increases the skin detection error it has no significant effect on the performance of face detection.

V

ABSTRAK

Pengesan wajah merupakan proses penting dalam banyak aplikasi seperti pengesanan wajah, pengenalan diri dan pengesanan, dan akses kawalan. Teknik yang digunakan dalam pengesan wajah bergantung kepada wajah dimodelkan. Dalam Desertasi im, wajah dimode/kan sebagai bahagian kulit dan bahagian bibir yang memenuhi beberapa kreteria geomatrik. Oleh itu, system pengesan wajah mempunai tiga komponen utama: Modul pengesan kulit, modul pengesan bibir, dan modul pengasahan wajah. Rangkaiaan "multi-layer perceptron" (MLP) neuro dan teknik-teknik diambang histogram telah digunakan bagi mengesan kulit dan bibir. Bagi menguji sistem pengesan wajah, dua pangkalan data telah dibina. Imej di dalam pengkalan data yang pertama, dinamakan "In-house" telah diambil dalam persekitaran yang terkawal sementara yang berada di dalam pangkalan data yang kedua, dinamakan "WWW" telah dikumpul daripada Rangkaian Web Sedunia "World- Wide Web': Hanya warna kulit and bibir dalam skema warn rgb telah digunakan begi mengesan kulit dna bibir. Satu kaedah baru, dinamakan "maximum intensity normalisation'. bagi memperoleh komponen-komponen rgb daripada komponen- komponen RGB telah dicadangan. Didapati bahawa kaedah yang dicandankan member peratusan betul yang tingi daripada skema rgb konvensional tidak kira apa pangkalan data digunakan atau kaedah pengesan kulit. Dua kaedah telah digunakan bagi mencari bilangan neuron dalam lapisan tersembunyl "hidden layer" : Kaedah pertama menggunakan pencarian secaraa binary di antara nilai-nilai minimum dan maksimum sementara kaedah yang kedua menggunakan pencarian secara sequen dengan kreteria untuk berhenti. Kesan factor scala, eksperasi wajah dan sedikit aklusi-aklusi dengan cermin di atas kulit, pengesan bibir dan wajah telah diuji kaji. Didapati bahawa, semakin meningkat peratus factor scala, ralat pengesan kuilt dan bibir menurun. Walau bagaimanapun, peratus ralat menurun dalam pengesan kulit dan bibir bergantung kepada skema warna, kaedah pengesan dan komponen "chrominance" yang digunakan. Tetapi factor scala tidak member kesan apa-apa kepada pengesan wajah. Pada keseluruhanya, ekspresi wajah tidak member kesan yang besar atau bermakna kepada pengesan kulit. Walau bagaimanapun, bagi pengesan bibir, ekspresi ketawa te/ah member kesan ralat yang tinggi diikuti oleh ekspresi senyuman. Walau bagaimanapun, bagi pengesan bibir, peratus ralat meningkat bergantung kepada skema warna, kaedah pengesan dan komponen warna "chrominance" yang digunakan. Dan bagi pengesan wajah, ekspresl wajah member kesan negative terutamanya pada factor scala 3. Walaupun, aklusi kecil akan meningkatkan ralat pengesan kulit /a tidak member kesan yang signifikan kepada prestasi pengesan wajah

VI

TABLE OF CONTENTS

DECLARATION

CONFIRMATION

ACKNOWLEDGMENTS

ABSTRACT

ABSTRAK

TABLE OF CONTENTS

LIST OF TABLES

LIST OF FIGURES

LIST OF ABBREVIATIONS

LIST OF SYMBOLS

KEYWORDS

CHAPTER 1 INTRODUCTION

Page

ii

III

iv

V VI

vii XI

xv

xxi

xxii xxv

1

1.1 OVERVIEW OF FACE PROCESSING SYSTEMS 1 1.2 CHALLENGES OF FACE DETECTION 2 1.3 STATE OF THE ART IN FACE DETECTION 3 1.4 OBJECTIVES OF THE THESIS 4 1.5 MAIN CONTRIBUTIONS 4 1.6 ORGANISATION OF THE THESIS 5

CHAPTER 2 OVERVIEW OF FACE DETECTION TECHNIQUES 8

2.1 CLASSIFICATION OF FACE PROCESSING SYSTEMS 8 2.2 REVIEW OF FACE DETECTION SYSTEMS 9

2.2.1 Template-Based 9 2.2.2 Appearance-Based 10 2.2.3 Features Invariant 14 2.2.4 Knowledge-Based 16 2.2.5 Combination of More Than One Representation 16

2.3 SUMMARY 17 2.4 THE PROPOSED SYSTEM 19

CHAPTER 3 ANALYSIS OF SKIN AND LIPS COLOUR DISTRIBUTIONS 21

3.1 INTRODUCTION TO COLOUR SYSTEMS 21 3.1.1 The RGB Colour Space 21 3.1.2 The Normalised RGB Colour Space 21 3.1.3 The XYZ Colour Space 22 3.1.4 The Normalised XYZ Colour Space 22 3.1.5 The U*V*W* Colour Space 23 3.1.6 The L*a*b* Colour Space 23

vii

3.1.7 The I1I2I3 Colour Space 24 3.1.8 The HSI Colour Space 24 3.1.9 The YUV Colour Space 24 3.1.10 The YCrCb Colour Space 25 3.1.11 The YIQ Colour Space 25

3.2 THE SELECTED COLOUR SPACE 25 3.3 THE DATABASES 27 3.3.1 The In-House Database 28 3.3.2 The WWW Database 30 3.4 MANUAL SEGMENTATION 31 3.5 A MODIFIED NORMALISED RGB COLOUR SPACE 32 3.6 DATA PREPARATION 33 3.7 ANALYSIS OF SKIN AND NON-SKIN DISTRIBUTIONS 34 3.8 ANALYSIS OF LIPS AND NON-LIPS DISTRIBUTIONS 39 3.9 ANALYSIS OF LIPS AND SKIN DISTRIBUTIONS 41 3.10 SUMMARY 42

CHAPTER 4 SKIN DETECTION USING HISTOGRAM THRESHOLDING 45

4.1 INTRODUCTION 45 4.2 REVIEW OF SKIN DETECTION METHODS 46

4.2.1 Non-Parametric Models 47 4.2.2 Semi-Parametric Models 50 4.2.3 Parametric Models 52 4.2.4 Combined Techniques 53

4.3 HISTOGRAM THRESHOLDING METHODS 54 4.4 PROPOSED MODIFICATIONS TO THE THRESHOLDING METHOD 56

4.4.1 Scanning Window 56 4.4.2 Moving Window 57

4.4 PERFORMANCE METRICS 57 4.5 FINDING THE THRESHOLDS VALUES 59 4.6 SKIN DETECTION 64

4.6.1 The Effect of Chrominance Components on Skin Detection 65 4.6.2 The Effect of the Normalisation Method 70 4.6.3 Effect of the Pixel Selection Method on Skin Detection 72 4.6.4 The Effect of the Database 75 4.6.5 The Effect of Scaling 75 4.6.6 Effect of the Facial Expressions 77 4.6.7 Effect of Partial Occlusion with Glasses 78 4.6.8 Images of Best and Worst Skin Detection Error 80

4.7 SUMMARY 80

CHAPTER 5 SKIN DETECTION USING NEURAL NETWORKS 83

5.1 INTRODUCTION TO NEURAL NETWORKS 83 5.2 DETERMINING THE NETWORK PROPERTIES 84 5.3 FINDING THE NUMBER OF NEURONS IN THE HIDDEN LAYER 85

5.3.1 In-House Database 89 5.3.2 WWW Database 91

5.4 TRAINING THE NETWORKS 92 5.4.1 Training Data for the In-House Database 92 5.4.2 Training Data for the WWW Database 93

viii

5.4.3 Training Performance for the In-house Database 95 5.4.4 Training Performance for the WWW Database 96

5.5 SKIN DETECTION RESULTS AND ANALYSIS 97 5.5.1 Thresholding the Output of the Neural Networks 97 5.5.2 The Effect of the Intensity Normalisation Methods 98

5.5.2.1 The In-House Database 98 5.5.2.2 The WWW Database 99

5.5.3 Effect of the Training Data Population 101 5.5.4 Effect of Scale, Facial Expressions and Partial Occlusion 104

5.6 SUMMARY 108

CHAPTER 6 LIPS DETECTION USING HISTOGRAM THRESHOLDING 109

6.1 REVIEW OF LIPS DETECTION METHODS 109 6.2 HISTOGRAM THRESHOLDING METHODS 110 6.3 LIPS DETECTION 114

6.3.1 The Effect of the Chrominance Component on Lips Detection 116 6.3.2 Effects of the Intensity Normalisation Method 117 6.3.3 Effects of the Pixel Selection Method 119 6.3.4 Effect of the Database 120 6.3.5 Effect of Scaling 121 6.3.6 Effect of the Facial Expression 122 6.3.7 Sample Images 123

6.4 LIPS AND SKIN SEGMENTATION 126 6.4.1 Skin and Lips Histograms 126 6.4.2 Finding the Optimum Threshold Values 127 6.4.3 Lips Detection 131 6.4.3.1 Effect of Threshold Selection Method 132 6.4.3.2 Effect of the Intensity Normalisation Method 134 6.4.3.3 Effect of Scale on Lips Selection 137 6.4.3.4 Effect of Facial Expression 138 6.4.3.5 Effect of the Database 139 6.4.3.6 Sample Images 141

6.5 SUMMARY 141

CHAPTER 7 LIPS DETECTION USING NEURAL NETWORKS 144

7.1 TRAINING AND VALIDATION DATA 144 7.2 NEURAL NETWORKS STRUCTURES 145

7.2.1 Finding the Number of Neurons in the Hidden Layer 145 7.2.2 The In-House Database 147 7.2.3 The WWW Database 149

7.3 TRAINING THE NEURAL NETWORKS 150 7.4 LIPS DETECTION 152

7.4.1 The Effect of the Intensity Normalisation Method 152 7.4.2 The Effect of the Combination of Chrominance Components 153 7.4.3 Effect of the Scale Factor on Lips Detection 154 7.4.4 Effect of Facial Expressions on Lips Detection 156

7.5 LIPS DETECTION: A COMPARISON BETWEEN HISTOGRAM THRESHOLDING AND MLP NEURAL NETWORKS 157

7.6 SUMMARY 157

ix

CHAPTER 8 FACE DETECTION 159

8.1 INTRODUCTION 159 8.2 SKIN DETECTION MODULE 159

8.2.1 Opening and Closing 160 8.2.2 Connected Component Labelling 162 8.2.3 Size Filtering 162

8.3 LIPS DETECTION MODULES 163 8.4 FACE DETECTION 164

8.4.1 Face Detection Using Histogram Thresholding 167 8.4.2 Face Detection Using Neural Networks 168

8.5 ANALYSIS OF FACE DETECTION SYSTEMS 169 8.5.1 Effect of Scale Factor on Face Detection 170 8.5.2 Effect of Facial Expressions on Face Detection 171 8.5.3 Effect of Minor Occlusion With Glasses on Face Detection 171 8.5.4 Performance Comparisons of Histogram and Neural Networks Based

Systems 172 8.6 PERFORMANCE COMPARISON WITH EXISTING SYSTEMS 174 8.7 SUMMARY 175

CHAPTER 9 CONCLUSION 177

9.1 INTRODUCTION 177 9.2 SKIN DETECTION 178 9.3 LIPS DETECTION 179 9.4 FACE DETECTION 180 9.5 FUTURE WORKS 181

REFERENCES 183

GLOSSARY 188

Appendix A Skin and Non-Skin Histograms Distributions 189

Appendix B Lips and Non-Lips Histograms Distributions 207

Appendix C Lips and Skin Histograms Distributions 216

Appendix D Relationship Between the Threshold Value and the Error Index for the Skin and Non-Skin 228

Appendix E Relationship Between the Threshold Value and the Error Index for the Lips and Non-Lips. The Selected Threshold Value is the Chrominance Value that Minimizes the Error Index. 237

Appendix F Relationship between the Threshold Value and the Error Index for the Lips and Non-Lips. The Threshold Value is Selected When Skin Error Equals Lips Error 249

Appendix G Average Percentage Skin Segmentation Error Networks Trained Using Single Chrominance Component and Using a Combination of two Chrominance Components 261

X

LIST OF TABLES

Page

Table 2.1 Classification of Face Detection Methods 17

Table 3.1 Description of Common Face Databases 28

Table 3.2 Descriptions of the Types of Images in the In-house 29 Database

Table 3.3 Classification of the Skin and Non-skin Histogram 37 Distributions According to the Amount of Overlap Between the Two Distributions

Table 3.4 Classification of the Lips and Non-lips Histogram 40 Distributions According to the Amount of Overlap between the Two Distributions

Table 3.5 Classification of the Lips and Skin Histogram Distributions 42 According to the Amount of Overlap between the Two Distributions

Table 4.1 Upper and Lower Thresholds Values for Several 63 Chrominance Components for Segmenting the In-House and WWW Databases into Skin and Non-Skin Regions

Table 4.2 Expected Percentage Skin Segmentation Error for a 64 Number of Chrominance Components using Single and Double Thresholds on the In-house and the WWW Databases

Table 4.3 Percentage Skin Segmentation Error for a Number of 65 Chrominance Components using Double Thresholds on Images of the In-house Database with Neutral Expressions at Scale factor of 1

Table 4.4 Percentage Skin Segmentation Error for a Number of 67 Chrominance Components Using Double Thresholds on the WWW Database

Table 4.5 Percentage of Skin Detection Error for the WWW Database 68 using Equations 4.24 and 4.25 with Pixel Intensity Normalisation Method

Table 4.6 Comparison of the Lowest Percentage Skin Detection Error 72 given by Different Intensity Normalization methods for the

XI

In-House and the WWW Databases

Table 5.1 Parameters Used for Finding the Number of Neurons in the 88 Hidden Layer for Networks Used for Skin Detection

Table 5.2 Training Data Used to Find the Number of Neurons in the 88 Hidden Layer for the Neural Networks Used for Skin Detection with a Single Chrominance Component

Table 5.3 Finding the Number of Neurons in the Hidden Layer for 90 Skin Detection on the In-House Database

Table 5.4 Network Structures and the Corresponding Chrominance 90 Component used for Skin Detection on the In-House Database

Table 5.5 Network Structures and the Corresponding Chrominance 92 Component used for Skin Detection on the In-House Database

Table 5.6 Number of Training and Validation Samples for Training the 93 Neural Networks for Skin Detection on the In-house Database

Table 5.7 Number of Training and Validation Samples for Training the 94 Neural Networks for Skin Detection on the WWW Database using a Single Chrominance Component

Table 5.8 Description of the Five Methods used for Selecting the 95 Images used for Training and Validation of the Neural Networks for the WWW Database

Table 5.9 Training Performance for the Skin Detection Networks 96 Trained on the In-house Database

Table 5.10 Training Performance for the Skin Detection Networks 97 Trained on the WWW Database Using a Single Chrominance Component

Table 6.1 Upper and Lower Thresholds Values for Several 113 Chrominance Components for Segmenting the In-House and WWW Databases Into Lips and Non-Lips Regions

Table 6.2 Expected Lips Segmentation Error for a Number of 115 Chrominance Components

Table 6.3 Percentage Lips Segmentation Error for a Number of 116 Chrominance Components Using Double Thresholds for Neutral Expressions at Scale 1 Images of the In-house

XII

Database

Table 6.4 Percentage Lips Segmentation Error for a Number of 117 Chrominance Components Using Double Thresholds for the WWW Database

Table 6.5 Upper and Lower Thresholds Values Selected as the 128 Minimum and the Maximum of the Sum of the False Acceptance Rate and the False Rejection Rate

Table 6.6 Threshold Value Selected when the False Acceptance Rate 129 Equals the False Rejection Rate

Table 6.7 The Expected Percentage Lips Detection Error for the Tmin 131 Threshold

Table 6.8 The Expected Percentage Lips Detection Error for the 132 Tequal Threshold

Table 6.9 Combination of Chrominance Components Used for Skin 132 Detection for Both Databases and for Both Intensity Normalisation Methods

Table 6.10 Description of the Intensity Normalisation Methods used for 135 Lips Detection

Table 6.11 Combination of Chrominance Components that Gave the 136 Lowest Lips Detection Error

Table 7.1 Number of pixels used for the Training and Validation of 144 the Neural Networks for Lips Detection on the In-house and WWW Databases

Table 7.2 Network Structures Used for Lips Detection for the In- 147 house Database for Several Combinations of Chrominance Components

Table 7.3 Network Structures Used for Lips Detection on the WWW 150 Database for Several Combinations of Chrominance Components

Table 7.4 Network Training Performance for Networks Used for Lips 151 Detection on the In-house Database for Several Combinations of Chrominance Components

Table 7.5 Network Training Performance for Networks Used for Lips 151 Detection on the WWW Database for Several Combinations of Chrominance Components using Maximum Intensity

Table 7.6 Network Training Performance for Networks Used for Lips 152 Detection on the WWW Database for Several Combinations of Chrominance Components using Pixel Intensity

XIII

Normalisation

Table 7.7 Combination of Chrominance Component that Gave the 154 Lowest Percentage Lips Detection Error

Table 7.8 Average Percentage Decreases in the Percentage of Lips 157 Detection Error as a Result of the Increase in the Scale Factor

Table 8.1 The Combination of Chrominance Components Used for the 160 Skin Detection Module

Table 8.2 The Rules for Erosion and Dilation Processes 160

Table 8.3 Smallest and Largest Ratio of the Skin Regions to the Area 162 of the Image for the In-house and WWW Databases

Table 8.4 Description of the Four Lips Detection Modules 164

Table 8.5 Smallest and Largest Ratios of the Lips Regions to the Area 164 of the Image for the In-house and WWW Databases

Table 8.6 The Minimum and Maximum Ratio of the Face Candidate 167 Size to the Size of the Lips Candidate for the In-house and WWW Databases

Table 8.7 Performance of the Four Face Detection Systems Using 168 Histogram Thresholding

Table 8.8 Chrominance Component and Intensity Normalisation 170 Methods Used for Skin and Lips Detection Modules for Face Detection using Neural Networks

Table 8.9 Performance of the Face Detection Systems using Neural 171 Networks

Table 8.10 Performance Comparison Between the Proposed Systems 175 and an Existing System

XIV

LIST OF FIGURES

Page

Figure 1.1 Relationship between the Various Face Processing Tasks 3

Figure 2.1 Neural Network Architecture of Li, S. Z. et al. 11

Figure 2.2 Block Diagram of the Initially Proposed Face Detection 19 System

Figure 2.3 Block Diagram of the Finally Proposed Face Detection 19 System

Figure 3.1 Samples Images from the In-house Database 30

Figure 3.2 Samples Images from the WWW Database 31

Figure 3.3 A Sample of Original and Manually Segmented Images 31

Figure 3.4 A Sample Image From the In-House (top) and WWW 33 (bottom) Databases with Maximum and Pixel Intensity Normalisation Methods

Figure 3.5 Skin and Non-skin Pixels Extraction Process 35

Figure 3.6 Skin and Non-Skin Extraction Flowchart 36

Figure 3.7 Classification of Skin and Non-Skin Histograms into Two 38 Categories

Figure 3.8 Effect of the Database on the Skin and Non-skin 39 Distribution.

Figure 3.9 Classifications of Lips and Non-Lips Histograms into Two 40 Categories

Figure 3.10 Effect of the Database on the lips and Non-lips 41 Distributions.

Figure 3.11 Effect of the Database on the lips and Skin Distributions 43

Figure 4.1 Classification of Skin and Non-Skin histograms into Two 55 Types: One Requiring Two Threshold values and One Threshold Value

Figure 4.2 Flowchart of the Procedure for Finding Threshold Value 61

xv

Figure 4.3 Relationship Between the Error Index and the Threshold 62 Value

Figure 4.4 Effect of the Normalisation Method on the Percentage of 69 Skin Detection Error

Figure 4.5 Effects of the Intensity Normalisation Method on Skin 70 Detection Error for the In-House Database

Figure 4.6 Skin Detection using both Maximum and Pixel Intensity 71 Normalisation Methods

Figure 4.7 Effect of the Pixel Selection Method on the Percentage 73 Skin Detection Error for the In-house Database

Figure 4.8 Effect of the Pixel Selection Method on the Percentage 74 Skin Detection Error for the WWW Database

Figure 4.9 Percentage Skin Detection Errors for a Number of 76 Chrominance Components for the In-House and WWW Databases

Figure 4.10 Effect of the Scaling on the Percentage of Skin Detection 77

Figure 4.11 Effect of the Facial Expression on the Percentage of Skin 78 Detection Error for Scale Factor of 1

Figure 4.12 Effect of partial Occlusion with Glasses on the 79 Percentage Skin Detection Error for Scale Factor 1

Figure 4.13 Samples of Skin Detection Images of the In-House 80 Database with Neutral Expression at Scale factor of 1

Figure 5.1 Network Structure for Skin Detection Using (a) a Single 85 Chrominance and (b) Two Chrominance Components

Figure 5.2 Flowchart for the Procedure to Find the Number of 87 Neurons in the Hidden Layer for Skin Detection

Figure 5.3 Number of Trained Networks (top) and the Mean 89 Squared Error (bottom) Versus the Number of Neurons in the Hidden Layer for the In-House Database

Figure 5.4 Number of Trained Networks (top) and the Mean 91 Squared Error (bottom) Versus the Number of Neurons in the Hidden Layer for the WWW Database

Figure 5.5 Determining the Optimal Threshold Value for 98 Thresholding the Output of the Neural Networks

xvi

Figure 5.6 Percentage Skin Detection Error for Neural Networks 100 Trained using a single Chrominance Component on the In-house Database

Figure 5.7 Percentage Skin Detection Error for Neural Networks 101 Trained using Two Chrominance Components on the In- house Database

Figure 5.8 Effect of Intensity Normalisation Methods on the 102 Percentage Skin Detection Error for Neural Networks Trained using a Single Chrominance Component on the WWW Database

Figure 5.9 Effect of Intensity Normalisation Methods on the 103 Percentage Skin Detection Error for Neural Networks Trained using Two Chrominance Components on the WWW Database

Figure 5.10 Effect of Training Data Population on Skin Detection 104 Error on the In-house Database

Figure 5.11 Effect of Training Data on Skin Detection Error using a 105 Single Chrominance Component on the WWW Database

Figure 5.12 Effect of Training Data Selection Method on the 106 Percentage Skin Detection Error When a Combination of Two Chrominance Components was Used for the WWW Database

Figure 5.13 Effect of Scale, Facial Expressions, and Partial Occlusion 107 on the Percentage Skin Detection Error

Figure 6.1 Classification of Lips and Non-Lips histograms into Two 111 Types

Figure 6.2 Relationships Between the Estimated Lips Segmentation 114 Error and the Threshold Value

Figure 6.3 Effects of the Normalisation Method on the Percentage 118 of Lips Detection Error

Figure 6.4 Effect of the Intensity Normalisation Method on the 119 Percentage Lips Detection Error Under Different Scale, Facial Expressions and Minor Occlusions for the In- House Database

Figure 6.5 Effect of the Pixel Selection Method on the Percentage of 120 Lips Detection Error for the In-house Database

Figure 6.6 Effect of the Pixel Selection Method on the Percentage of 121

xvii

Lips Detection Error for the WWW Database

Figure 6.7 Effect of the Database on the Percentage of Lips 122 Detection Error

Figure 6.8 Effect of Scaling on the Percentage of Lips Detection 123 Error for the In-house Database for Several Chrominance Components

Figure 6.9 Effect of the Facial Expression on the Percentage of Lips 124 Detection Error for the In-house Database for Several Chrominance Components

Figure 6.10 Performance of the Lips Detection Algorithm on Neutral 125 Expression at Scale 1 Images of the In-House Database

Figure 6.11 Performance of the Lips Detection Algorithm on the 125 WWW Database

Figure 6.12 Skin and Lips Histogram Distributions for the Normalised 127 r-g Chrominance Component for the In-house and WWW Databases using Maximum Intensity and Pixel Intensity Normalisation

Figure 6.13 Chrominance Component that Give Similar Threshold 130 Values (top) and Different Threshold Values (bottom) When Using Two Different Criteria for the Threshold Finding Algorithms

Figure 6.14 Percentage Lips Segmentation Error using Maximum 133 Intensity Normalisation Method on both Databases

Figure 6.15 Percentage Lips Segmentation Error using Pixel Intensity 134 Normalisation Method on both Databases

Figure 6.16 Percentage Lips Detection Error for Different 136 Combination of Intensity Normalisation Methods for the WWW Database

Figure 6.17 Percentage Lips Detection Error for Different 137 Combination of Intensity Normalisation Methods for the In-house Database

Figure 6.18 Effect of Scale Factor on the Lips Detection Error with 138 Tmin Threshold

Figure 6.19 Effect of Facial Expressions on Lips Detection with 139 Tequal Threshold

Figure 6.20 Effect of the Database on the Lips Detection Error with 140 Tmin Threshold

xviii

Figure 6.21 Images that Gave the Lowest and Highest Lips Detection 141 Error for Tequal and Tmin Thresholds Respectively for Scale factor 1 with Neutral Expression of the In-House Database

Figure 6.22 Images that Gave the Lowest (top) and Highest 142 (bottom) Lips Detection Error for Tequal (left) and Tmin (right) Thresholds for Maximum Intensity for the WWW Database

Figure 6.23 Neutral Expression Images at Scale Factor of 1 of the In- 143 House Database Sorted According to the Percentage Lips Detection Error

Figure 7.1 Flowchart for Finding the Number of Neurons in the 148 Hidden Layer for Lips

Figure 7.2 Relationship between the Number of Neurons in the 149 Hidden Layer and the Mean Percentage Correct Detection

Figure 7.3 Relationship between the Number of Neurons in the 150 Hidden Layer and the Mean Percentage Correct Detection and the Mean Squared Error for the WWW Database

Figure 7.4 Effects of Intensity Normalisation on the Mean 153 Percentage lips Detection Error for the In-House Database

Figure 7.5 Effects of Intensity Normalisation on the Mean 154 Percentage lips Detection Error for the WWW Database

Figure 7.6 Effects of Scale Factor on the Percentage Lips Detection 155 Error for the In-house Database

Figure 7.7 Effect of the Facial Expression on the Percentage of Lips 156 Detection Error for the In-House Database

Figure 7.8 A Comparison Between the Histogram Thresholding and 158 MLP Neural Network for Lips Detection

Figure 8.1 Block Diagram of the Skin Detection Module Used by All 161 Face Detection Systems

Figure 8.2 An Example of Skin Detection Process on the In-House 163 Database

xix

Figure 8.3 Block Diagram of the Lips Detection Module When a 165 Single Intensity Normalisation Method or a Single Threshold Selection Method is Used

Figure 8.4 Block Diagram of the Lips Detection Module When Two 165 Intensity Normalization Methods are Used

Figure 8.5 Block Diagram of the Lips Detection Module When Two 166 Threshold Selection Methods are Used

Figure 8.6 An Example of Lips Detection Process on the In-House 166 Database

Figure 8.7 Performances of the Four Face Detection Systems on the 168 In-house Database

Figure 8.8 Performances of the Four Face Detection Systems on the 169 WWW Database

Figure 8.9 Effect of Scaling on the Percentage of Correct Face 171 Detection

Figure 8.10 Effect of Facial Expressions on Face Detection for 172 Different Scale Factors

Figure 8.11 Effect of Minor Occlusions with Glasses on the 173 Performance of Face Detection Systems

Figure 8.12 Face Detection Comparisons Between Histogram 174 Thresholding and Neural Network Systems

Figure 8.13 Face Detection Comparisons Between Histogram 176 Thresholding and Neural Network Systems on Test Images

xx

LIST OF ABBREVIATIONS

CIE

EI

EM

FAR

FLD

FRR

GMM

LDNN

MAP

ML

MLP

NSE

PCA

PDF

PSE

RCE

ROC

SDNN

SE

SNoW

SOM

Commission Internationale de I'Eclairage

Error Index

Expectation Maximization

False Acceptance Rate

Fisher's Linear Discrimination

False Rejection Rate

Gaussian Mixture Model

Lips Detection Neural Network

Maximum a Posteriori

Maximum Likelihood

Multi-Layer Perceptron

Non-Skin Error

Principal Component Analysis

Probability Density Function

Percentage Segmentation Error

Restricted Coulomb Energy

Receiver Operating Characteristics

Skin Distinction Neural Network

Skin Error

Sparse Network of Winnows

Self-Organising Map

xxi

LIST OF SYMBOLS

T, ̀

T` h

Tc

NES

J NEns

%C

k5

! -S, T

b

B(X, Y)

b(x, y)

C

Cl

Lower Threshold for Chrominance Component C

Higher Threshold for Chrominance Component C

Threshold for Chrominance Component C

The Number of Skin Pixels Incorrectly Classified as Non-Skin

The Number of Non-Skin Pixels Incorrectly Classified as Skin

Percent Correct Classification

The Mahalanobis Distance from the Vector X to the Mean

Vector MS

A Standard Threshold The Blue Chrominance Component of the rgb Colour Space

The Value of the Blue Chrominance Component at the x, y Coordinates

The Value of the Normalised Blue Chrominance Component at the x, y Coordinates

A Given Chrominance Component

Class 1

C11 and C22 Correct Classification

C12 False Rejection

C2 Class 2

C21 Cb

Cii

C,

CS

Cs

f;

False Acceptance

The Blue Chrominance of the YCbCr Colour Space.

Cost Function

The Red Chrominance of the YCbCr Colour Space.

The Covariance Matrix of the Skin Chrominance

The Covariance Matrix Of The Skin Chrominance

The Joint Probability Distribution Function

xxii

9

G(X, y)

g(x, y)

g/b

M

M(x, y)

MS

N

NC1

NC2

NCC1

NCC2

NEC1

NEC2

N'

NM(X, Y)

NP(x, y)

NS(c)

Ns(X)

Ns'(c)

Ns'(X)

0(x, y)

P(C)

P(C/S)

P(S)

P(S/C)

P(S')

The Green Chrominance Component of the rgb Colour Space

The Value of the Green Chrominance Component at the x, y

Coordinates

The Value of the Normalised Green Chrominance Component

at the x, y Coordinates

The Ratio of the Green and Blue Chrominance Components of

the rgb Colour Space.

The Total Number of Images in the Database

Pixel at the x, y Coordinates in the Manually Segmented Image

The Mean Vector of the Skin Chrominance

Total Number of Pixels in the Image (equal to NC1 + NC2)

Number of Pixels of Class 1 in the Image

Number of Pixels of Class 2 in the Image

Number of Class 1 Pixels Correctly Classified

Number of Class 2 Pixels Correctly Classified

Number of Class 1 Pixels Incorrectly Classified

Number of Class 2 Pixels Incorrectly Classified

The Number of Pixels in Image J

Pixel at the x, y Coordinates in the Normalised Image using

Maximum Intensity Normalisation Method

Pixel at the x, y Coordinates in the Normalised Image using

Pixel Intensity Normalisation Method

The Number of Skin Pixel in Colour C

The Number of Skin Pixel in Colour X

The Number of Non-Skin Pixel in Colour C

The Number of Non-Skin Pixel in Colour X

Pixel at the x, y Coordinates in the Output (Segmented) Image

The Probability Of Colour C Occurring In an Image.

The a Priori Probability of a Pixel P(X, Y) being Skin

The Estimated Skin Probability

The Probability of a Pixel with colour C being a Skin

The Estimated Non-Skin Probability

xxiii

REFERENCES

Acharya T. & Ray A. K. 2005. Image Processings: Principles and Applications New Jersey: John Wileys and Sons.

Albiol A., Torres L. & Delp E. J. 2001. Optimum Colour Spaces for Skin Detection. Proceedings of the International Conference on Image Processing. Tessaloniki. 122 -124.

Albiol A., Torres L. & Delp E. J. 2001. An Unsupervised Color Image Segmentation Algorithm For Face Detection Applications. Proceedings of the International Conference on Image Processing. Tessaloniki. 681 - 684.

Anagnostopoulos C., Anagnostopoulos I., Vergados D., Papaleonidopoulos I., Kayafas E., Loumos V. & Stasinopoulos G. 2002. A Probabilistic Neural Network for Face Detection on Segmented Skin Areas Based On Fuzzy Rules. Proceedings of the 11th Mediterranean Electrotechnical Conference. Cairo. 493 - 497.

Bergasa L. M., Mazo M., Gardel A., Sotelo M. A. & Boquete L. 2000. Unsupervised and Adaptive Gaussian Skin-color model. Image and Vision Computing. 18(12): 987-1003.

Brand J. D. & Mason J. S. D. 2001. Skin Probability Map and Its Use In Face Detection. Proceedings of the International Conference on image processing. Thessaloniki 1034 -1037.

Brown D. A., Craw I. & Lewthwaite J. 2001. A SOM Based Approach to Skin Detection with Application in Real Time Systems. Proceeding of the British Machine Vision Conference. Manchester. 491-500

Butler D., Sridharan S. & Chandran V. 2002. Chromatic Colour Spaces For Skin Detection using GMMS. Proceedings of the International conference on Acoustics, Speech, and Signa/ Processing. Orlando. 3620 -3623.

Chai D. & Bouzerdoum A. 2000. A Bayesian Approach to Skin Colour Classification in YCbCr Colour Space. Proceedings of the IEEE Region Ten Conference (TENCON2000), Kuala Lumpur, 421-424.

Chang T. C., Huang T. S. & Novak C. 1994. Facial Feature Extraction From Color Images. Proceedings of the Lt IAPR International. Conference on pattern Recognition Computer Vision & Image Processing. Vol. 2,39 - 43

Chiang C. C., Tai W. K., Yang M. T., Huang Y. T. & Huang C. 3.2003. A Novel Method for Detecting Lips, Eyes, and Faces in Real-Time. Real-Time Imaging. Vol (9), 277-287, Elsevier.

183

Cotton S. 1995. Colour, colour spaces and the human visual system. Technical report CSR-95-03, Department of Computer Science, University of Birmingham

Eveno N., Caplier A. & Coulon P-Y. 2003, lumping Snakes and Parametric Model for Lip Segmentation. Proceedings of the International Conference on Image Processing. Vol. 2,867-870

Fleck M., Forsyth A. & Bregler C. 1996. Finding Naked People. Proceeding of the 4th European Conference on Computer Vision. Cambridge. 593-602.

Fu L. M. 1994. Neural Networks in Computer Intelligence. McGraw Hill

Gevers, T. 1990. New Approach to Segmentation of Colour Images. Msc Thesis, University of Amsterdam.

Gevers, T. 2001. Robust histogram construction from color invariants. Proceedings of the Eighth IEEE Computer Vision International Conference on ICCV 2001. 615 - 620.

Gomez E., Travieso C. M., Briceno 3. C. & Ferrer M. A. 2002. Biometric Identification System By Lip Shape. Proceedings of the 36th Annual International Carnahan Conference on Security Technology. 39 - 42

Gomez G. 2002. On Selecting Colour Components for Skin Detection. Proceedings of the 16 th International Conference on Pattern Recognition. 961 -964

Gonzales R. C., Woods R. E. & Eddins S. L. 2004. Digital Image Processing, lud Edition. New Jersey: Prentice Hall.

Greenspan H., Goldberger 3. & Eshet I. 2001. Mixture Model For Face-Color Modelling and Segmentation. Pattern Recognition Letters. 22(14): 1525- 1536.

Haddadnia J., Faez K. & Ahmadi M. 2002. N-Feature Neural Network Human Face Recognition. Proceedings of the 15th International Conference on Vision Interface. Calgary.

Hsu R. L., Abdel-Mottaleb M. & Jain A. K. 2002. Face Detection In Color Images, IEEE Transaction on Pattern Analysis and Machine Intelligence. 24(5): 696- 706.

Jones M. J. & Rehg J. M. 1998. Statistical Color Models with Application to Skin Detection. Compaq Cambridge Research Lab Technical Report, No. 98-11.

Kim J. B., Moon C. H. & Kim H. J. 2002. Efficient Face Extraction Using Skin-Color Model and a Neural Network, Lecture Notes in Computer Science, 538-543.

Kotropoulos C. & Pitas I. 1997. Rule-based Face Detection in Frontal Views. Proceedings of the International Conference on Acoustics, Speech and Signal Processing. Vol. 4,122-126.

184

Kotropoulos C., Bassiou N., Kosmidis T. & Pitas I. 2001. Frontal Face Detection Using Support Vector Machines and Back-Propagation Neural Networks. Proceeding of the Scandinavian Conference on Image Analysis. Bergen.

Kruppa H., Bauer M. A. & Schiele B. 2002. Skin Patch Detection in Real-World Images. Proceedings of the DA GM Symposium 2002. Zurich, Switzerland.

Lawrence S., Lee G. C., Tsoi A. C. & Back A. D. 1997. Face Recognition: A Convolutional Neural Network Approach. IEEE Transaction on Neural Networks, Special Issue on Neural Networks and Pattern Recognition. 8(1) : 98-113.

Li S. Z., Lu J. W. & Chan K. L. 2000. Face Detection Using Neural Learning with Optimized Feature Set. Proceedings of the Fourth Asian Conference on Computer Vision. Taipei.

Li Y., Gong S., Sherrah J. & Liddell H. 2000. Multi-View Face Detection Using Support Vector Machines And Eigenspace Modelling. Proceedings of the International Conference on Know/edge-based Intelligent Engineering Systems and Allied Technologies. Brighton. 241-245.

Lievin M. & Luthon F. 2000. A Hierarchical Segmentation Algorithm for Face Analysis Application for Lipreading. Proceedings of the IEEE International Conference on Multimedia and Expo. 1085-1088

Liew A. W. C., Leung S. H., & Lau W. H. 2000, Lip Contour Extraction Using A Deformable Model. Proceedings of the International Conference on Image Processing. Vol. 2,255 - 258

McKenna S., Gong S. & Collins J. J. 1996. Face Tracking and Pose Representation. Proceeding of the British Machine Vision Conference. Edinburgh, Scotland.

Mitsukura Y., Fukumi E. & Akamatsu N. 2000. A Design of Face Detection System by Using Lip Detection Neural Network and Skin Distinction Neural Network. Proceedings of the International Conference on Systems, Man, and Cybernetics. 2789 -2793.

Moreno F., Andrade-Cetto J. & Sanfeliu A. 2001. Localization of Human Faces Fusing Colour Segmentation and Depth from Stereo. Proceedings of the 8th IEEE International Conference on Emerging Technologies and Factory Automation. Antibes. 527-535.

Nefian A. 1999. A Hidden Markov Model-Based Approach for Face Detection and Recognition. PhD. Thesis, Georgia Institute of Technology, USA.

Pavlovic V. & Garg A. 2001. Efficient Detection of Objects and Attributes Using Boosting. Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition.

Peng Z., Ai H., Hong W., Liang L. & Xu G. 2003. Multi-cue-based face and facial feature detection on video segments. Journal of Computer Science and Technology. Vol. 18(2), 241-246.

185

Rowley H. A., Baluja S. & Kanade T. 1998. Rotational Invariant Neural Network Based Face Detection. IEEE Transaction On pattern Analysis and Machine Intelligence. 38-44.

Sadeghi M., Kittler J. & Messer K. 2002. Modelling And Segmentation Of Lip Area In Face Images. Proceedings of IEE Vision, Image and Signa/ Processing. Vol. 149(3), 179 - 184

Sangwine S. J. & Horne R. E. N., 1998. The Colour Image Processing Handbook. London: Chapman and Hall.

Sigal L., Sclaroff S. & Athitsos V. 2000. Estimation and Prediction of Evolving Colour Distributions for Skin Segmentation under Varying Illumination. Proceedings of the Computer Vision and Pattern Recognition (CVPR'OO). vol 2,152 - 159

Skapura D. M. 1995. Building Neural Networks. New York: Addison-Wesley.

Skarbek W. & Koschan A. 1994. Colour Image Segmentation -A Survey. Technical report, Technical University of Berlin.

Son L. P., Chai D. & Bouzerdoum A. 2001. A Universal And Robust Human Skin Color Model Using Neural Networks. Proceedings of the International Joint Conference on Neural Networks. 2844 -2849.

Terrillon J. C. & Akamatsu S. 1999. Comparative Performance of Different Chrominance Spaces for Color Segmentation and Detection of Human Faces in Complex Scene Images. Proceedings of the 12th Conference on Vision Interface. 180-187.

Troje N. F. & Vetter T. 1996. Representations of human faces. Technical Report, Max Planck Institute for Biological Cybernetics, Tübingen, Germany.

Valentin D., Abdi H. & O'Toole A. J. 1994. Categorization and Identification of Human Face Images by Neural Networks: A Review of Linear Auto- Associator and Principal Component Approaches. Journal of Biological Systems. 2: 413-429

Wang J-G. & Sung E. 1999. Frontal-View Face Detection and Facial Feature Extraction Using Color and Morphological Operation. Pattern Recognition Letters. 20(10) 1053-1068.

Wark T. & Sridharan S. 1998. A Syntactic Approach to Automatic Lip Feature Extraction for Speaker Identification. Proceedings of ICASSP-98. pp. 3693- 3696.

Wei F. & Li P. F. 2001. A Novel Face Segmentation Algorithm. Proceedings International Conferences on Info-tech and Info-net. Beijing, 550 -556.

Yang M. H., Roth D. & Ahuja N. 2000. A SNoW-Based Face Detector, Advances in Neural Information Processing Systems. S. A. Solla, T. K. Leen and K. -R. Muller (eds), pp. 855-861, MIT Press.

186

Yang M. H., Ahuja N. & Kriegman D. 2002. Detecting Faces in Images: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence. 24(1): 34- 58.

Yin X., Dong G. & Ming X. 2001. Hand Image Segmentation Using Color And RCE Neural Network. Robotics and Autonomous Systems 34(4): 235-250

Yip A. & Sinha P. 2001. Role of Color in Face Recognition. MIT AIM-2001-035.

Zarit B. D., Super B. 3. & Quek F. K. H. 1999. Comparison Of Five Color Models In Skin Pixel Classification. Proceedings International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems. 58 -63

Zhang Q. & Izquierdo E. 2006. Multi-feature based face detection. Proceedings of the International Conference on Visual Information Engineering. VIE 2006. pp. 572-576.

Zhao W., Chellappa R., Rosenfeld A. & Phillips J. 2000. Face Recognition: A Literature Survey. Technical Report, CS-TR4167, University of Maryland.

187