universiti putra malaysia development of …psasir.upm.edu.my/7301/1/fk_2008_88a.pdfuniversiti putra...
TRANSCRIPT
UNIVERSITI PUTRA MALAYSIA
DEVELOPMENT OF A NEURAL NETWORK-BASED CAMERA FOR
TOMATO HARVESTING ROBOTS
AZIZ BIN MAMAT
FK 2008 88
DEVELOPMENT OF A NEURAL NETWORK-BASED CAMERA FOR TOMATO HARVESTING ROBOTS
by
AZIZ BIN MAMAT
Thesis Submitted to the School of Graduate Studies, University Putra Malaysia, in Fulfilment of the requirements for the Degree of Master Of Science
December, 2008
Dedicated to my parents,
Hj. Mamat Bin Hj. Awang and Hjh. Habsah Binti Hj. Awang Kechik
My beloved wife,
Hjh. Sazilawati Binti Abdul Aziz
My beloved daughter and sons,
Nurul Afina Binti Aziz
Zaim Raziqin Bin Aziz
Zaim Wafiuddin Bin Aziz
and
friends
with love.
i
Abstract of thesis presented to the Senate of Universiti Putra Malaysia in fulfilment of the requirement for the degree of Master of Science.
DEVELOPMENT OF A NEURAL NETWORK-BASED CAMERA FOR TOMATO HARVESTING ROBOTS
By
AZIZ BIN MAMAT
December 2008
Chairman : Professor Wan Ishak Wan Ismail, PhD
Faculty : Engineering
Automated tomato harvesting robots were rapidly developed recently. Most of the
designs were more focused on positioning of the end of robotic arm by using various
methods such as combination of the sensor and vision system. This project
concentrated on the artificial intelligent via the Neural Network, in order to provide a
better decision making system for tomato harvesting robot. The objective of this
study was to develop 3 degree of freedom tomato harvesting robotic system complete
with gripper and motion program. The development of software for tomato pattern
identification, determination of the X and Y coordinates from web camera captured
and the determination of the tomato and tomato ripeness using decision making from
Neural Network also become the main objective. The approach is to detect the
desired object using vision system attached to the cylindrical automation system and
perform image analysis. These features will serve as inputs to a neural net, which
will be trained with a set of predetermined ripe tomato. The output is a command for
harvester arm to make the movement for harvesting. The position determination was
done with a conversion of the distance in pixel into a distance in metric unit (mm) of
ii
the tomato image. Whereas the depth of the tomato distance (z direction) was done
by moving the actuator system towards the calculated tomato position until the
object sensor senses the present of the tomato. AWIsoft07 software was developed
to view the harvester vision, display the captured image analysis on the harvester
vision, and display the numerical analysis output and neural network output.
The harvester system with 3 degree of freedoms (3DOF) equips with specially
designed tomato gripper named as AWI2007 Tomato Harvesting Robot was
developed in order to realize the data from the AWISoft07 developed software.
Several calibrations were made to ensure the accuracy of the AWI2007 Tomato
Harvesting Robot. The AWIsoft07 and AWI2007 Tomato Harvesting Robot were
subjected to several harvesting tests under the laboratory environment. The
harvesting result shows the ability of the software and the harvester. Consequently,
AWI2007 Tomato Harvesting Robot with the camera vision was able to recognize
the tomato ripeness intelligently via neural network analysis and moved to the
harvesting position. These situations provided new improvements for tomato
harvesting system compared to the previous findings. Therefore the application of
the neural network based on camera vision was successful perform as artificial
intelligent for tomato harvesting robotic system.
iii
Abstrak tesis dikemukan kepada Senat Universiti Putra Malaysia sebagai memnuhi keperluan untuk ijazah Master Sains
MEMBINA NEURAL NETWORK BERASASKAN KAMERA UNTUK ROBOT PENUAIAN TOMATO
Oleh
AZIZ BIN MAMAT
Disember 2008
Pengerusi : Professor Wan Ishak Wan Ismail, PhD
Fakulti : Kejuruteraan
Robot penuaian tomato yang beroperasi secara outomatik telah dibina secara rancak
kebelakangan ini. Kebanyakan rekabentuknya memfokuskan kepada penentuan
kedudukan pengesan hujung lengan robot dengan menggunakan berbagai kaedah
seperti kombinasi sensor dan sistem penglihatan. Projek ini menumpukan kepada
kepintaran tiruan melalui Jaringan Neural, dalam menyediakan sistem pembuatan
keputusan untuk robot penuaian tomato. Objektif kajian ini adalah membangunkan
robot penuaian tomato dengan tiga darjah kebebasan, lengkap dengan penggengam
dan program pergerakan. Pembangunan perisisan untuk identifikasi bentuk tomato,
menentukan koodinat X dan Y daripada rakaman imej kamera sesawang dan
menentukan tomato serta kemasakan tomato, menggunakan pembuat keputusan
daripada Jaringan Neural yang juga menjadi objektif utama. Pendekatannya ialah
untuk mengesan objek yang perlukan menggunakan sistem penglihatan yang
dipasangkan pada sistem automasi selinder dan menghasilkan analisa imej. Ciri-ciri
ini akan berfungsi sebagai masukan kepada jaringan neural, yang mana akan dilatih
dengan satu set tomato masak yang telah dikenalpasti. Keluarannya ialah arahan
iv
kepada lengan penuai bagi menghasilkan gerakan untuk penuaian. Penentuan
kedudukan dilakukan dengan menukarkan jarak dalam unit pixel kepada jarak dalam
unit metrik (mm) untuk imej tomato. Sebaliknya jarak kedalaman tomato (arah z)
dilakukan dengan menggerakkan lengan penggerak sistem kearah kedudukan tomato
yang telah dikira kedudukannya sehingga sensor objek mengesan kehadiran tomato.
Perisian AWIsoft07 berkeupayaan mempamirkan pandangan penuai, dan
memaparkan analisa imej yang diperolehi daripada pandangan penuai, memaparkan
keluaran analisis berangka dan keluaran jaringan neural.
Sistem penuai slinder dengan tiga darjah kebebasan dengan penggenggam tomato
yang rekabentuk khas dinamakan sebagai AWI2007 Robot Penuaian Tomato telah
dibangunkan demi merealisasikan data daripada perisian AWIsoft07. Beberapa
kalibrasi telah dilakukan untuk memastikan ketepatan AWI2007 Robot Penuaian
Tomato. Perisian AWIsoft07 dan AWI2007 Robot Penuaian Tomato telah melalui
beberapa ujian penuaian di bawah kesekitaran makmal. Keputusan penuaian
menunjukkan kebolehan perisisan dan penuai. Sesungguhnya, AWI2007 Robot
Penuaian Tomato berkeupayaan untuk mengenali kemasakan buah tomato dan
bergerak kekedudukan penuaian secara pintar. Keadaan ini telah menghasilkan
suatu penambahbaikan untuk sistem penuaian tomato berbanding dengan hasil
dapatan sebelumnya. Kesimpulannya aplikasi jaringan neural berasakan penglihatan
kamera telah berjaya dihasilkan sebagai kepintaran tiruan untuk sistem robot
penuaian tomato.
v
ACKNOWLEDGEMENTS
The author would like to thank ALLAH s.w.t the most gracious and merciful for
giving the opportunity and the ability to finish this project.
Special great appreciation and gratitude is expressed to Professor Dr. Ir Wan Ishak
Wan Ismail who keeps excellent supervision, help, continuous encouragement,
guidance and numerous discussions throughout this project. Great appreciation is
also expressed to Associate Professor Dr. Datin Napsiah Ismail and Associate
Professor Dr. Abdul Rahman Ramli for their advice, help valuable remarks and
encouragement all the way throughout this project.
Honestly appreciation is expressed to Politeknik Kota Bharu, Kementerian Pangajian
Tinggi, Department of the Civil Servant and Government of Malaysia for their
support and sponsorship throughout this study.
The deepest appreciation is expressed to Mr Jasman Jaafar, Mr. Khairuddin Abdul
Rahman, Mr. Zakiria Bin Ismail, Mr. Abd. Rahman Abdullah and all the wonderful
people who were involved in this project directly or indirectly for giving the help,
guidance, advice, support and encouragement throughout the course of this study.
Sincere gratitude and appreciation is also addressed to my beloved wife, kids, family
and friends for their full support, help, and encouragement in continuing my study.
vi
I certify that a Thesis Examination Committee has met on 2 December 2008 to conduct the final examination of Aziz bin Mamat on his thesis entitled "Development of a Neural Network-Based Camera for Tomato Harvesting Robots" in accordance with Universities and University Colleges Act 1971 and the Constitution of the Universiti Putra Malaysia [P.U.(A)106] 15 March 1998. The Committee recommends that the student be awarded the Master of Science.
Members of the Thesis Examination Committee were as follows: Ishak Aris, PhD Associate Professor Faculty of Engineering Universiti Putra Malaysia (Chairman) Ir. Desa Ahmad, PhD Professor and Ir Faculty of Engineering Universiti Putra Malaysia (Internal Examiner) Megat Mohamad Hamdan, PhD Professor Faculty of Engineering Universiti Pertahanan Malaysia (Internal Examiner) Mohd Marzuki Mustafa, PhD Professor Faculty of Engineering Universiti Kebangsaan Malaysia (External Examiner)
_________________________ BUJANG KIM HUAT, PhD Professor and Deputy Dean School of Graduate Studies Universiti Putra Malaysia Date: 19 March 2009
vii
This thesis was submitted to the Senate of Universiti Putra Malaysia and has been accepted as fulfilment of the requirement for the degree of Master of Science. The members of Supervisory Committee were as follows: Wan Ishak Wan Ismail, PhD Profesor and Ir Faculty of Engineering Universiti Putra Malaysia (Chairman) Napsiah Ismail, PhD Associate Professor and Datin Faculty of Engineering Universiti Putra Malaysia (Member) Abdul Rahman Ramli, PhD Associate Professor Faculty of Engineering Universiti Putra Malaysia (Member)
_______________________________ HASANAH MOHD GHAZALI, PhD Professor and Dean School of Graduate Studies Universiti Putra Malaysia Date: 9 April 2009
viii
DECLARATION
I declare that the thesis is my original work except for quotations and citations which have been duly acknowledged. I also declare that it has not been previously, and is not concurrently, submitted for any other degree at Universiti Putra Malaysia or at any other institution.
__________________ AZIZ BIN MAMAT Date : 30 January 2009
ix
TABLE OF CONTENTS Page
ABSTRACT ii ABSTRAK iv ACKNOWLEDGEMENTS vi APPROVAL vii DECLARATION ix TABLE OF CONTENTS x LIST OF TABLES xiii LIST OF FIGURES xiv LIST OF ABBREVIATIONS xvi
CHAPTER
1.0 INTRODUCTION 1
1.1 Overview 1 1.2 Problem Statements 3 1.3 Research Scope And Limitations 5 1.4 Objectives 6 1.5 Thesis Layout 6
2.0 LITERATURE REVIEW 8
2.1 Anatomy of the tomato 8 2.2 Malaysian tomato variety 10 2.2.1 Tomato grading and maturity 10 2.3 Color and spectrum 10 2.3.1 Color in outdoor scenes 10 2.4 Camera and robot vision 12 2.5 Neural Network 15 2.6 Harvesting robot in greenhouse 24 2.7 Open Fields harvesting robot 25 2.8 Existing tomato harvesting robot 26
3.0 METHODOLOGY 34
3.1 Preliminary study 34 3.1.1 Image acquisition 34 3.2 Digital Image Processing 37 3.2.1 Image enhancement 37 3.2.2 Color image segmentation 39
x
3.2.3 Mathematical operation on the image and object edge detection
39
3.2.4 Objects indexing 41 3.3 Artificial Neural Network (ANN) 44 3.3.1 Learning mechanism 45 3.4 AWIsoft07 software 47 3.4.1 Overview of the AWIsoft07 software 48 3.4.2 Digital Image Analysis using AWIsoft07
Software 54
3.4.3 Neural Network Analysis using AWIsoft07 software
58
3.4.4 AWIsoft07 software controlling the AWI2007 Tomato Harvesting Robot
59
3.5 AWI2007 Tomato Harvesting Robot Development 61 3.5.1 Design concept 61 3.5.2 Tomato gripper design 61 3.5.3 AWI2007 Tomato Harvesting Robot Design 63 3.5.4 Mobile unit design 64 3.5.5 The X axis direction design mechanism 65 3.5.6 The Y direction design mechanism 66 3.5.7 The Z direction design mechanism 67 3.5.8 The Motor Driver Determination 69 3.5.9 The support system 75 3.6 AWI2007 Tomato Harvesting Robot Calibration 76 3.6.1 Pixel image distance calibration 76 3.6.2 X axis rotation angle calibration 77 3.6.3 Y axis vertical distance calibration 78
4.0 RESULTS AND DISCUSSION 80
4.1 Digital image pixel value calibration 80 4.2 X axis rotation angle calibration 81 4.3 Y axis displacement calibration 83 4.4 Neural Network Analysis 85 4.5 AWIsoft07 Result Display 88 4.6 Robot Motion According To The Tomato
Coordinate 90
4.7 Harvesting Experimental 90 4.8 Problems And Suggestions 92 4.8.1 Effect of the changes of the light intensity
on AWIsoft07 analysis 92
4.8.2 Effect of the changes of the webcam parameter on AWIsoft07 analysis
93
4.8.3 AWI2007 Tomato Harvesting Robot modification.
93
xi
4.8.4 Modification of the communication system 94 4.9 Research Benchmark 95
5.0
CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE RESEARCH 96
5.1 Conclusion 96 5.2 Recommendations for future research 99
REFERENCES 100
APPENDICES BIODATA OF STUDENT LIST OF PUBLICATIONS
xii
LIST OF TABLES
Table Page
2.1 Errors in calculating L*a*b* 12
4.1 RGB average value 86
4.2 Results of Neural Network analysis on five selected colors 87
4.3 Result of the tomato harvesting using AWI2007 Tomato Harvesting Robot.
91
xiii
LIST OF FIGURES Figure Page
2.1 Estimate of L*a*b* values on the basis of RGB measurements.
11
2.2 A sketch of the experimental setup.
13
2.3 A sketch of a laser guidance system.
13
2.4 Configuration of the vision system.
15
2.5 Three-layer back-propagation neural network. 18
2.6 Architecture of the neural network used for the analysis.
22
2.7 Grape harvesting end-effector
25
2.8 Tomato Robot harvester situated in green house
27
2.9 Block diagram of 3-D Vision sensor
28
2.10 End effectors (First prototype).
29
2.11 End effectors (Second prototype).
30
2.12 Motion of suction pad and check valve.
30
2.13 Method of harvesting.
32
3.1 The overall structure flowchart 35
3.2 Tomato color plate sampling 36
3.3 Overall Digital Image Processing Flowchart 38
3.4 Image adding technique
40
3.5 Image subtraction technique
40
3.6 Image threshold
42
3.7 Boundary and Centroid determination.
44
3.8 The determination of the 10 by 10 pixels from tomato image center coordinate.
45
3.9 The flowchart of the neural network sub program 46
3.10 Tomato color plate with three-layer back-propagation neural network
48
xiv
3.11 AWIsoft07 Software
49
3.12 AWIsoft07 software Control Panel:Tomato Calibration Mode menu flowchart
50
3.13 AWIsoft07 software Control Panel:Image Acquisitions menu flowchart
51
3.14 Control Panel Operation
52
3.15 Operation of the MODE B
53
3.16 Image Acquisition control button 54
3.17 AWI2007 Preview window
55
3.18 Image display from the output of the digital image processing
56
3.19 Numerical Output From Digital Image Processing
57
3.20 Neural Network analysis window
58
3.21 Robot motion flowchart sub program 60
3.22 Fingers of the Tomato gripper model 62
3.23 The assembly of the X,Y and Z axis motor 64
3.24 Mobile unit assembly model
65
3.25 The assembly of the X axis motor
66
3.26 The assembly of the Y axis motor
68
3.27 The assembly of the Z axis motor
69
3.28 Portion of the main helical shaft 71
3.29 Force diagram the main helical shaft 71
3.30 Measurement of the webcam distance
77
3.31 Calibration of the rotational angle for X axis
78
3.32 Calibration of the vertical movement on Y axis
79
4.1 Calibration of the pixel distance converted to mm distance
81
4.2 Calibration of the X axis rotation angle to time of X axis motor activation
82
xv
4.3 Calibration of the time of the Y axis motor activated and pointer travel distance measured
84
4.4 The measurement of the pointer travel distance of the Y axis motor activated according to the 10 second time interval
85
4.5 The result from the AWIsoft07 89
4.6 Webcam setting parameter
94
xvi
LIST OF ABBREVIATONS θ The threshold value
α The learning rate.
δk(p) The error gradient at neuron k in the output layer at iteration p.
∂yk (p) The output of neuron k in the output layer at iteration p.
∂xk (p) The net weighted at neuron k in the output layer at iteration p
Δwjk(p) The output layer weight correction at hidden layer j.
2D Two dimensions
3D Three dimensions
ANN Artificial neural network
CCD Couple charge diode
CW Clockwise
CCW Counter clockwise
d The camera distance (mm)
d.c. Direct current
DOF Degree of freedom
ek (p) The error of neuron k in the output layer at iteration p
Ec Calibration error
Er The percentage of error
GUI Graphical User Interface
IGES Improved gray-scale
MSE Mean square of error
MT Metric ton
n The number of neuron input
xvii
NN Neural network
P The distance in pixels
PSD Position-sensitive device
Q The distance in millimeters (mm).
R2 Coefficient of regression
RCCW The counter clockwise X axis angle of rotation in degrees
RCW The clockwise X axis angle of rotation in degrees
RGB Red, Green and Blue color space
TCW The time given to activate the X axis motor in second for clockwise rotation.
TCCW The time given to activate the X axis motor in second for counter clockwise rotation.
TDW The downward time given to activate Y axis motor, in second.
TUW The upward time given to activate Y axis motor, in second.
TV Television
UPM Universiti Putra Malaysia
U.S.A United State of America
USB Universal serial bus
Volt Voltage
wi The weight of input i
wjk(p) The output layer weight.at hidden layer j.
wjk (p + 1) The output layer updating weight at hidden layer j.
xi The value of input i
X The net weighted input to the neuron
yd.k(p) The desired output of neuron k at iteration p.
yj(p) The output of neuron j in the hidden layer p
xviii
xix
yk(p) The actual output of neuron k at iteration p
Y The output of the neuron
YDW The displacement distance in downward direction in millimeter
YUW The displacement distance in upward direction in millimeter
CHAPTER 1
INTRODUCTION
1.1 Overview
Lycopersicum esculentum which is known as tomato was classified under fruity
vegetables. Tomato was originated from South America and widely produced in the
tropical environment. Cameron Highland was the main tomato production area in
Malaysia, and for the low land tomato widely produced in Johor (MOA, 2006).
Consumption of the tomato in Malaysia can be categorized into fresh and process
tomato. The process tomato mainly can be divided into tomato puree, tomato paste
and concentrated tomato. Processed tomato also produced tomato sauce and
ketchup. Ministry of Agriculture in Malaysia reported that in year 2001, Malaysia
imported 8,588.59 metric ton (MT) (RM20,605,256) tomato puree, tomato paste and
concentrated tomato, where as in 2002 (until April) Malaysia imported 2,248.52245
MT(RM5,183,535) (MOA, 2006). Malaysia also exported 12,295 MT fresh tomato
in 2000 and stand 20th place world ranking for tomato exporter. Malaysia exported
7,517MT tomato sauce and ketchup which is 1% of the world market (MOA, 2006).
The decrement in import and increment in export, indicate that the tomato production
in Malaysia showed a positive indicator for expansion.
To produce tomato in a large volume needs a large area of land. The best tomato
cultivation is on the high land area cultivated with the high land variety tomato.
However the highland area is limited, therefore tomato production must be done in
the low land area with the cultivation of the good variety lowland tomato. The
modern production techniques must be practiced with controlled environment.
Under the green house production system, the possibility to apply the automation and
robotic system become more relevant especially when the rising cost of labor and the
difficulties to employ the labor for agricultural work.
Recently many researchers develop the robot in the way of fully autonomous and
move like the human being. The vision system was developed to represent the
human eyes by using capture device such as camera. The capture devices were used,
ranging from an expensive video camera, couple charge diode (CCD) camera, digital
camera, to the low-priced web camera. The different camera will give the different
sensitivity and resolution. The sensitivity of the capture device also depends on the
environment condition, especially to the light intensity. Normally the captured
image especially when using web camera will displayed in the Red, Green and Blue
(RGB) color space. The digital image processing is able to reduce the effect light
intensity to the captured images. The digital image processing also can be applied to
identify the pattern of the object in the image.
In the new era of the robot development, artificial intelligent becomes one of the
most important tools to make robot becomes intelligent. The artificial intelligent is
defined as "the study of the computations that make it possible to perceive, reason,
and act", includes an expert systems, fuzzy logic and artificial neural network
(Winston, 1993).
Traditionally, tomato harvesting was done manually by picking the fruit by hand or
cut by knife or scissor. Traditional harvesting method is normally facing the
2
problems of labors and time consuming. Therefore the paradigm change needed to be
done in this area. Mechanization or automation in the harvesting seems to be a
solution for those problems. The physical characteristics of the tomato plant, such as
soft and brittle stem are constraints to the mechanization application. The fruit itself
is covered by soft layer skin. Consequently, the application of mechanization for
harvesting tomato may damage the tomato stem and bruise the tomato fruit.
Therefore the best way to make the paradigm change is using automation in
harvesting. The advantages of automation are
i) Tomato fruit will avoid from bruise during harvesting.
ii) The tomato plant will not be damaged during harvesting.
iii) The usage of labor will be reduced.
iv) The tomato is possible to be graded during harvesting.
v) The robot is capable of working non stop (due to the power supply life
time).
Further more, there are several disadvantages for automation, such as higher initial
capital needed and good technical knowledge to operate. The cost for maintenance is
also high when using automation.
1.2 Problem Statements
The introduction of the robot in the bioproduction environments not only saves labor
but will help raise productivity. The developments of the agriculture robots were
raise according to the needs. For examples, an autonomous six-legged walking robot
was developed to explore and sample in rugged for outdoor environments
3
(Hoffman, 1991). The three degree of freedom (DOF) agricultural harvester robot
was designed to perform the oil palm harvesting (Bouketir, 1999). The agricultural
harvester robot was equipped with CCD camera for red object identification and an
algorithm was also developed in order to transfer two dimensional image plane to the
base of the three dimensional coordinate frame. However no report was made on the
intelligent robot component. The 'robot eye' system for agriculture robot to predict
actual distance of the target object using videogrammetry technique and triangulation
method was developed toward the intelligent robot (Mohd, 2002).
The evolution of the robot works under the greenhouse environment also grows in
the similar way. The mushroom harvesting robot was developed by Reed et.al.,
(1995) to find the relation between detachment torque and cap diameter and the
relation between torque and twist angle detachment. It was reported that an available
torque of 200Nmm combined with a rotational angle of 180˚ should be necessary.
Van et. al. (2006) reported on the de-leafing cucumber plant using an autonomous
robot. In a number of cases one leaf was reported successfully removed. The vision
system and manipulator control was reported perform relatively well (Van et. al.,
2006; Van et. al., 2002).
The research on the strawberry harvesting robot was reported by Satou et. al, (1996),
and the cherry tomato harvester robot (Fujiura et. al., 1990; Subrata et. al., 1995)
were developed by researches. The robot was guided with machine vision (a 3-D
vision sensor) and the harvesting algorithm. The end-effectors for the tomato
harvesting robot also were developed by Monta et. al. (1998). The end-effectors
with two parallel plate fingers and a suction pad was reported successfully harvest
4