“i hereby declare that i have read through this report ...eprints.utem.edu.my/19980/1/fruit...

“I hereby declare that I have read through this report entitle “Fruit Sorting By Machine

Vision Technique” and found that it has comply the partial fulfilment for awarding the

degree of Bachelor of Mechatronics Engineering.

Signature : ..........................................................................................

Supervisor’s Name : ..........................................................................................

Date : ..........................................................................................

FRUIT SORTING BY MACHINE VISION TECHNIQUE

KHOR CHIN WE

A report submitted in partial fulfilment of the requirements for the degree of

Bachelor of Mechatronics Engineering

Faculty of Electrical Engineering

UNIVERSITI TEKNIKAL MALAYSIA MELAKA

2016

iii

I declare that this report entitle “Fruit Sorting By Machine Vision Technique” is the

result of my own research except as cited in the references. The report has not been

accepted for any degree and is not concurrently submitted in candidature of any other

degree.

Signature : ..........................................................................................

Name : ..........................................................................................

Date : ..........................................................................................

KHOR CHIN WE

2/6/2016

iv

To my beloved mother and father

v

ACKNOWLEDGEMENT

First, I would like to thank to my supervisor, Pn. Nursabillilah Mohd Ali who

accept me as her FYP student to carry out her research with the title of ‘Fruit Sorting by

Machine Vision’. Pn. Nursabillilah gives me a lot of valuable advices and guides me

throughout the whole semester. Without her guidance and persistent help, I would not be

able to complete this Final Year Project Report 1 and Final Year Project Report 2.

Besides, special thanks my Machine Vision subject lecturer, Dr. Muhammad

Herman bin Jamaluddin who spend time to teach us patiently and provide us a lot of useful

information. Also, my grateful thanks to those all my friends who are willing to help me in

the understanding of image processing subject. Last, not forget to give my great

appreciation my beloved parents who mentally support and encourage me. Without helps

of particular that I mentioned above, I would face many difficulties in completing my Final

Year Report 1 and Final Year Report 2.

vi

ABSTRACT

Nowadays, automation system such as machine vision system is widely used in

many industrial. Fruit sorting by machine vision technique is a potential automation system

which is able to obtain their target in term of productivity and quality. However, there are

several problems such as camera resolution, illumination and performance that need to be

considered for using the machine vision technique. Hence, the objectives of this project is

to design a high accuracy algorithm that is able to classify apples based on color and size.

Furthermore, the performance of fruit sorting system is needed to be evaluated and

optimized by manipulating the illumination, thresholding value and camera distance.

MATLAB is an analysis tool that has been used in this research project. There are three

experiments such as HSV value comparison, thresholding value performance comparison

and the different camera distance performance comparison. The experiments are aim to

define the tolerance limits of hue value and number of pixels. The tolerance limits are used

for the purpose of categorizing the type of apple. At the end of the experiment, the analysis

and discussion explained about the performances of each variation of parameters. The

problems such as low illumination condition and the limitation of different thresholding

value and distance of camera are described as well in the analysis and discussion. As

conclusion, the performance in term of accuracy and precision of the designed algorithm

has been evaluated and optimized at the end of this project.

vii

ABSTRAK

Pada masa kini, sistem automasi seperti sistem penglihatan mesin digunakan

semakin luas dalam pelbagai industri. Sistem pembahagian buah-buahan dengan

mengunakan teknik penglihatan mesin adalah satu sistem automasi yang berpotensi dan

bermampu untuk mendapatkan meningkatkan produktiviti dan kualiti dalam sesebuah

industri. Walau bagaimanapun, terdapat beberapa masalah seperti resolusi kamera,

pencahayaan dan prestasi yang perlu dipertimbangkan semasa menggunakan teknik

penglihatan mesin. Oleh itu, projeck penyelidik ini bertujuan untuk merekabentuk sebuah

algoritma yang berketepatan tinggi dan mampu membezakan epal berdasarkan warna dan

saiz. Tambahan pula, prestasi sistem pembahagian buah-buahan dapat dioptimumkan

dengan memanipulasi pencahayaan, nilai ambang dan jarak kamera. MATLAB adalah alat

analisis yang telah digunakan dalam projek penyelidikan ini. Terdapat tiga eksperimen

seperti HSV perbandingan, prestasi perbandingan dan jarak camera perbandingan telah

dilaksanakan untuk menentukan batas-batas toleransi nilai warna dan bilangan piksel. Had

toleransi digunakan untuk mengkategorikan jenis-jenis epal. Bahagian analisis dan

perbincangan telah menjelaskan bahawa tentang prestasi setiap perubahan parameter.

Masalah-masalah seperti pencahayaan rendah, had nilai ambang yang berbeza dan jarak

kamera berbeza telah diterangkan serta dalam analisis dan perbincangan. Kesimpulannya,

prestasi dari segi ketepatan algoritma telah dinilai dan dioptimumkan pada akhir projek ini.

viii

TABLE OF CONTENTS

CHAPTER TITLE PAGE

ACKNOWLEDGEMENT v

ABSTRACT vi

ABSTRAK vii

TABLE OF CONTENTS viii

LIST OF TABLES xi

LIST OF FIGURES xii

LIST OF APPENDICES xv

1 INTRODUCTION 1

1.1 Introduction 1

1.2 Motivation 2

1.3 Problem Statement 4

1.4 Objectives 5

1.5 Scopes 5

1.6 Thesis Organization 6

2 LITERATURE REVIEW 7

2.1 Fruit Sorting and Grading System 7

2.2 Background of Machine Vision 8

2.3 Components of Machine Vision System 9

2.3.1 Lighting 9

ix

2.3.2 Frame Grabber 10

2.3.3 Software - MATLAB 11

2.3.4 Cameras 11

2.3.5 Image Processing 12

2.3.6 Example of Image Processing Algorithm 18

2.4 Summary of Image Processing Technique 23

3 METHODOLOGY 24

3.1 Description of Research Project 25

3.2 Description of Fruit Sorting System Flow 27

3.3 Component Used 29

3.3.1 Digital Camera 29

3.3.2 Microcontroller (Arduino UNO) 30

3.3.3 DC Gear Motor (SPG30) 31

3.3.4 Single Channel 5V Relay 32

3.3.5 Light Bulb 33

3.4 Materials Used 34

3.4.1 Apples 34

3.5 Prototype of Fruit Sorting System 34

3.6 Analysis Tools 36

3.6.1 MATLAB 36

3.7 Experiment Setup 37

3.7.1 Experiment 1:Comparison of HSV 37

3.7.2 Experiment 2:Thresholding Value Comparison 38

3.7.3 Experiment 3:Distance of Camera 40

4 RESULT AND DISCUSSION 41

4.1 Experiment 1: Comparison of HSV 41

x

4.1.1 Precision Analysis 41

4.1.2 Performance Evaluation 45

4.1.3 Hue Tolerance Setting 46

4.2 Experiment 2: Thresholding Value Comparison 50

4.2.1 Image Thresholding Analysis 50


4.3 Experiment 3: Comparison of Distance of Camera 55

4.3.1 Number of Pixels Tolerance Setting 55


5 CONCLUSION AND RECOMMENDATION 62

5.1 Conclusion 62

5.2 Recommendation 63

REFERENCES 64

APPENDICES 67

xi

LIST OF TABLES

TABLE TITLE PAGE

2.1 Characteristics of CCD and CMOS Cameras [19] 12

2.2 Summary of Image Processing Technique 23

3.1 Properties of Labeled Regions [46] 31

4.1 Results in different TH value 51

4.2 Detectability (Red Apple) in different TH value 52

4.3 Detectability (Green Apple) in different TH value 53

xii

LIST OF FIGURES

FIGURE TITLE PAGE

1.1 The number of robots in use in the five largest global markets,

projected to 2017 [3]

2

1.2 The efficiency of implementing manual facilities and automated

facilities [4]

3

2.1 Automated fruit sorting and grading system [16] 10

2.2 Image Processing Main Steps [5] 12

2.3 Brightness matrix adjustment [19] 13

2.4 Contrast matrix adjustment [19] 14

2.5 Cropping Image in matrix mode [19] 15

2.6 Image Zooming in matrix mode [19] 15

2.7 Rotation of Image in matrix mode [19] 16

2.8 Image Dilation in matrix mode [19] 17

2.9 Image Erosion in matrix mode [19] 17

2.10 Image captured from mobile device is converted into L*a*b 18

2.11 Typical segmentation techniques (a) Thresholding, (b) edge-based

segmentation and (c) region based segmentation [5] age

19

2.12 Standard deviation measurement superimposed on each object in

the im

19

2.13 Pipeline of detection algorithm [14] 20

2.14 Image extracting flow of fruit sorting system [13] 21

xiii

2.15 Block diagram of thresholding process [5] 22

3.1 Research project flow chart 26

3.2 Fruit Sorting System flow chart 28

3.3 CMOS Camera 29

3.4 Arduino UNO 30

3.5 DC Gear Motor 31

3.6 Single Channel 5V Relay 32

3.7 Fluorescent light bulbs 33

3.8 Incandescent light bulbs 33

3.9 Type of apples 34

3.10 Conveyer belt 35

3.11 Inspection box 35

3.12 Assembled prototype 35

3.13 MATLAB Computer Software 36

3.14 Experiment setup 37

3.15 Installation of light bulb 38

3.16 Tracking process 39

3.17 Distance variation 40

4.1 HSV color space in a conical object 42

4.2 Low precision (left) and high precision (high) 43

4.3 Standard deviation in different illuminations 44

4.4 Overall performance in different illuminations 45

4.5 Control limit of hue value (Red) 46

4.6 Control limit of hue value (Green) 47

4.7 Rotten and yellowish red apples 48

4.8 Ripe apple versus fresh apple 49

xiv

4.9 Red and Green apples hue value region 51

4.10 Accuracy of red apple detection 54

4.11 Accuracy of green apple detection 54

4.12 Number of pixels in camera position 20cm (Small apple) 56

4.13 Number of pixels in camera position 20cm (Big apple) 56



4.16 Number of pixels in camera position 30cm (small apple) 58




4.20 Number of pixels in camera position 40 cm (Small apple) 60

4.21 Number of pixels in camera position 40 cm (Big apple) 60

4.22 Overall performance in different camera positions 61

xv

LIST OF APPENDICES

APPENDIX TITLE PAGE

A Program code 69

B Gantt chart for FYP1 and FYP2 73

C Table of Result 74

D Publication 81

1

CHAPTER 1

1. INTRODUCTION

This chapter review the introduction of automation fruit sorting system. The

motivation, problem statement, objective and scope about this research project are discussed

in this chapter. Besides, the thesis organization is descripted at the last part of this chapter.

1.1 Introduction

There are several handling steps of fruit process in fruit processing industries. The

fruit after collecting from farm, it will undergo the process of washing, sorting, grading,

packing, transporting and storage before going to sell in market [1]. Fruit sorting and grading

can be considered as the heaviest processing task compared to the other steps. Normally,

more time and effort will be invested in this process. This process can be either operated in

manual sorting method or automatic sorting method. Manual fruit sorting is a method

operated by human in order to identify and separate fruit manually while automatic fruit

sorting is a method can be operated by using the technique of machine vision. One of the

advantages of machine vision is high precision [2]. Machine vision can be carried to get

measurements without any touch on the surface on objects, so there is no damage or

dangerous on the fragile parts. Besides that, machine vision can operate continuously and the

operating and maintenance costs are low. The high flexibility of visual system becomes the

reason for people to choose machine vision system rather than optical sensors.

2

1.2 Motivation

Manpower contributed by labour is flexible and fast. Meanwhile, machine vision is

type of automatic system which is dependable and precise. Many manufacturing industries

have tried to implement the automated system to reduce the use of manpower.

The Figure 1.1 shows the number of robots in use in the five largest global markets.

From the graph, the demand of robots from 4 countries except Japan shows the trend of

increase from year 2012 to 2017. According to the German-based International Federation of

Robotics, China is already the world’s largest market for industrial robots [3]. These trends

will keep increasing in the next few years. Hence, the trend obviously indicate that automatic

fruit sorting method is more preferable choice for fruit industrial compared to manual sorting

method.

Figure 1.1: The number of robots in use in the five largest global markets, projected to

2017 [3]

3

The Figure 1.2 represents the differences in term of profit, cost, and sales between

manual operating system and automated system. The sales of both systems are the same but

the operating cost of automated system decrease drastically while the profit increases by time.

The trends clearly indicate that most of the industries choose automation system rather that

manual system.

Figure 1.2: The efficiency of implementing manual facilities and automated facilities

[4]

The reason of using machine vision system that can be able to increase the profit is

high productivity. However, some industries hire more labours in order to seek and achieve

higher productivity but normally manual fruit sorting causes unwanted problems that may

directly influence the production result. Hence, training is basically to be provided for labours

in order to produce skillful, experience and quality worker but high investment cost on

labours will highly reduce a company profit.

Machine vision technique is a reliable system to be implemented in fruit sorting for

fruit industries. It is time-saving and high efficient system which is able to significantly hit

the production target in fruit industries. The designed system not only can operate

continuously day and night without any tiredness, the production output quality is also hugely

trustable. The capability of fruit sorting by machine vision is truly high compare to human

vision.

4

Besides, the visual results generally are highly dependent on labours mental and

physical fitness. Humans are not able to have fully concentration while making many

decisions within a certain time. Human decisions are variable and difficult to be standardized.

In addition, the task of visualizing fruit is quite tedious. It probably causes the problems of

difficulty to hire workers and labours turnover. Therefore, fruit sorting by using the technique

of machine vision is recommended system to be implemented.

1.3 Problem Statement

Machine vision technology is highly used in factory automation. It is a system that

camera captures products’ picture and interprets them to find defects and categorize them.

Nowadays, camera has become common in our life. Camera has been transformed from

analogue to digital camera and even the mobile phone has the function of high quality camera.

Thus, with this huge change of camera, it has become a selective device to detect object or

feature.

There are thousand types of camera in the market. Different cameras have different

image processing quality. Camera quality is a constraint that should be considered in machine

vision system. Normally, high Mega-Pixels of camera will cause a challenge in having too

much information. In real system operating time, high resolution camera will cause

unacceptable processing time [5]. Large image sizes require longer time to compute and

collect data from camera. Thus, selecting an appropriate type of camera based on the

requirement is able to improve a vision system in term of efficiency.

Besides, another problem should be overcome is the illumination conditions that

surround the camera. The colour detection and ability to distinguish an object are can be

affected by the surrounding environment [5]. The surrounding lighting condition should be

standardized in order to get precise analysis result. In addition, reflection of light on an object

causes analysis errors during image processing.

For an algorithm design, the accuracy is important in measuring the size and colour of

apples. A good algorithm is able to reduce the percentage of the errors such as systematic

5

error and random error from happening. Thus, the result consistency is highly maintained

with a good design of image processing algorithm.

1.4 Objectives

The objectives of the research project are:

1. To design and develop an algorithm for fruit classification based on size and colour.

2. To evaluate the system performance for fruit sorting and classification in term of

accuracy and precision.

3. To optimize the system performance by manipulating the illumination, thresholding value

and distance of camera.

1.5 Scopes

The scopes of the research project are listed as the following:

1. Apple is chosen as the analysis material for this fruit sorting system.

2. Algorithms are developed based on the size and colour of apple.

3. Algorithms are designed by using MATLAB computer software.

4. Analysis is based on the data of static image.

6

1.6 Thesis Organization

This report is the study about the fruit sorting by using the machine vision technique.

In this report, the chapter 1 will cover about the motivation and problem statement for

designing an image processing algorithm of fruit sorting system. Besides that, chapter 1 will

state about the objective and scope. For chapter 2, it will cover the theoretical background

and basic principle of machine vision. The previous related research of automatic fruit sorting

system and machine vision are also covered in chapter 2 and the methods and techniques in

designing an image processing algorithm of fruit sorting system will be discussed in chapter 3.

Finally, the result and conclusion will be covered in the last two chapters which are chapter 4

and chapter 5.

7

CHAPTER 2

2. LITERATURE REVIEW

This chapter presents the overview about the preview research project which related

to fruit sorting and grading system. Furthermore, the methods and theories of image

processing which related to fruit sorting process are reviewed and analyzed. The constraints

of applying machine vision technique are described in this chapter as well.

2.1 Fruit Sorting and Grading System

The demand of fruit based on the quality is getting high in market. Sorting and

grading are the important process to maintain the quality of fruit [6]. Many machine vision

systems have been proposed for agricultural applications such as fruit grading, fruit maturity

analyzing and fruit sorting. Machine vision has been used in the research of mango inspection

and grading system, mango maturity prediction and vegetable automation sorting by machine

vision [7]. The fruits such as citrus fruits, apple and strawberry have been used.

A distributed computer machine vision system for automated inspection and grading

of fruits was presented to control the quality of fruits [8]. The automation system is designed

for food industries so that the quality of food is able to meet the international standards.

Generally, maturity of fruit needs experts to be justified. Manual sorting is time cosuming,

laborious process and diffult to judge by human according to fruit maturity. Maturity

prediction system is used to predict the maturity of harvested mango [9]. The system is

design to solve the problem of manual sorting system.

8

For some small agro-industrial companies, it is burdensome to procure a high cost

sorting machine in order to maintain and increase their product quality. A portable smart

sorting and grading machine by using computer vision was presented to solve the problem of

heavy built-up system [9].

Grading and sorting system by using machine vision is very useful in maintaining the

consistency, uniformity and depletion of time. An integral automation industrial fruit and

vegetable system and the research is aim to meet the market standards, classified according to

quality levels based on the maturity, size, weight, density and skin defect [10]. To analyzing

the image, algorithm should be designed. There are different kinds of fruits that can be

different by applying the image processing technique. Image processing is widely used in

different agriculture products such as apple, tomatoes, mango, strawberry, cherries, orange,

lemon and banana was presented [11]. Besides, the application analysis of machine vision

technology in agricultural inspection consist various kind of problems [12]. The problems

such as surrounding light intensity should be taken serious to get a precise result.

2.2 Background of Machine Vision

Machine Vision (MV) is the technology and method that provide digital imaging

information from the environment and use it for analysis purpose. The analysis method by

using MV can be applied in many fields such as industrial inspection, medical visualization,

law enforcement and artistic effect. MV is highly demanded in many fields due to speed,

consistency and rest period. Machine Vision can process hundreds or thousands pieces of

information in a few seconds. Whereas human operative system cannot process information

as fast as machine vision based system. Besides that, the consistency of output process by

human is truly unstable compared to machine vision system. Humans are not able to work 24

hours per day but machine vision system can operate continuously with stop unless

maintenance.

Machine Vision is the method about digital processing image. Digital processing

image define as the process from a given input image in order to obtain a required output

image. There are basically 3 categories of digital image processing which are image

processes, image analysis and computer vision. For image processes, it is used in the

9

application of camera. The camera will capture a scene and the internal processor will

process the data of input image. The image will undergo noise removal, sharpening,

brightness adjustment and etc. Finally, the output image will be stored in the memory and it

has been ready for photo printing. For image analysis, the image after capturing or recording,

it will be undergoes the analysis process. The analysis process is basically running based on

the features, shape and texture of an object. The application of image analysis can be used in

object recognition and segmentation. Computer vision is basically the high level image

process which is used the computer system and digital processing image system. For this

digital processing image category, it needs to design a set of program based on requirements

and used for scene understanding and autonomous navigation. Normally, this kind of

technique has been applied for face recognition, fingerprint system and object tracking

system.

2.3 Components of Machine Vision System

The basic components of a machine vision system consist of a camera, a computer

and a lighting system [13]. The computer should be equipped with an image acquisition

board. Besides, computer software is needed for transmitting electronic signals to computers,

acquiring images and performing storage and processing of the images.

2.3.1 Lighting

The lighting unit selection is dependable for different applications. With a suitable-

chosen lighting system, recognition and analysis process will be taken easier in the optimal

way [14]. Also, eliminating the tedious image processing procedures is one of the reasons to

choose a best lighting system.

Due to the high sensitivity of of image processing to illumination, constant

environment conditions is important in achieving a robust performance of algorithm. Hence,

methods of controlling the environment conditions is to prevent the camera and objects from

“i hereby declare that i have read through this report ...eprints.utem.edu.my/19980/1/fruit...

Documents