image processing based detection & size …...accurate segmentation and detection of fruit in...
TRANSCRIPT
Image Processing Based Detection & Size Estimation of Fruit on Mango
Tree Canopies
Santi Kumari Behera
Assistant Professor, Department of Computer Science & Engineering,
VSSUT, Burla, Odisha, India
Shrabani Sangita
Research Scholar, Department of Computer Science & Engineering,
VSSUT, Burla, Odisha India.
Prabira Kumar Sethy
Assistant Professor, Department of Electronics,
Sambalpur University, Odisha India.
Amiya Kumar Rath
Professor, Department of Computer Science & Engineering,
VSSUT, Burla, Odisha India.
Abstract National fruit mango contributes a major part in national
growth. Due to the flavour, nutrition & taste mango is one of
the popular fruit. For increment in profitability of agricultural
industry the proper detection, grading and packaging of fruit
are very important. Image processing techniques hold the
promise of enabling rapid and accurate fruit crop yield
predictions in the field. The key to fulfilling this promise is
accurate segmentation and detection of fruit in images of tree
canopies. Fruit measure in image processing can be used to
estimate the size of fruit which used to inform management
decisions at an individual tree level. Here an image processing
based technique is developed to detect and measure the size of
each mango successfully, with the help of Randomizes Hough
Transform (RHT) the fruit us detected properly. The on-tree
fruit is detected with 98% accuracy and it measures each fruit
with 3.07% average error.
Keywords:Image Processing,Colour Thresholding, Detection,
Measurement, Randomized Hough Transform (RHT)
Introduction The king of fruit mango is a tropical fruit which consumed
worldwide. As mango is a seasonal fruit, it collected from the
tree on that season and transported to different location. But
there has been a decrease in production of good quality fruits,
due to improper cultivation, manual inspection, and lack of
maintenance, postharvest losses in handling and processing,
lack knowledge in quick quality evaluation techniques. Also,
rising labour costs, shortage of skilled laborers and the need to
enhance generation forms have all put pressure on producers
and processors for the demand of a rapid, economic,
consistent and nod-destructive inspection method. In such a
scenario, automation can reduce the costs by promoting
production efficiency. The on tree fruit detection, counting,
grading and packaging of mangoes, all are the important part
of good mango production. It is a tedious job and difficult for
the farmer to always maintain constant attentiveness.
On-tree estimation of fruit measure is valuable for the
expectation of estimation of harvest yield and can advise
pressing material (plate embed) buying and showcasing
courses of action. For physiological investigations, estimation
of the span of the individual fruit after some time permits
estimation of fruit development rate and its reaction to the
ailment and agronomic conditions. Blunder is happen at the
season of estimating the extent of each fruit on-tree, due to
natural product covering way.
In past many research work has been carried for detecting,
localizing & counting of on tree fruit [1-3] by implementing
KNN, SVM, ANN & texture analysis. It is easily detected
fruit but sometimes the problem is occurs due to overlapping,
shape or colour. An image processing technique is developed
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 4 (2018) Spl.© Research India Publications. http://www.ripublication.com
6
to estimate the volume and mass of the post-harvest fruit [4]
& weight is the estimate with the help of water displacement
method (WDM). For identifying specific curves like ellipse
the RHT (Round Hough Transform) is used [5-6]. In image
processing technique the colour analysis is very important, it
is helpful for segmentation [7-9]. Many types of colour model
are present like RGB, L*a*b, CMYK, HSV, YCbCr.
Detecting shape and estimating volume is one of the important
aspects of research [10-13], which help to grade fruit. The
shape of fruit help to analyze, whether it is circular, elliptical
or any other type of shape. After the shape analysis, the
volume measurement is an easy task. The favorable feature of
our paper is measuring the size of mango from open-field
images rather the post-harvest grading as previous research.
Materials & Methods The primal recognition technique, that human visual system
adapts to distinguish between green and magenta color mango
fruits and to identify fruits amidst background leaves, is the
color recognition technique. Therefore, a novel approach is
proposed using image processing for performing three
functions: 1) color space segmentation for background
removal 2) Detection of Mango 3) size measurement of
mango.
Figure 1: System Overview for Measurement of Mango Size.
L*a*b color space segmentation
The colour threshold app which helps to threshold colour
images by manipulating the colour components of the
images based on different colour spaces. This is chosen
to removing the background.
Figure 2: Colour thresholding for L*a*b* colour space.
Figure 3: Colour thresholding for L*a*b colour space.
To separate the mango from background, a pixel-based
division was led. The arranged RGB picture was then changed
over into CIE L*a*b* shading space to isolates the organic
product shape in the L* channel. This approach isolate the
picture into two classes of pixels, so that the intra-class
change is insignificant, making a double picture. Due to their
Measure
Size of
Fruit
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 4 (2018) Spl.© Research India Publications. http://www.ripublication.com
7
convexity and smoothness, the reflectance of mango organic
product is superior to anything the encompassing leaves or
branches and were expelled utilizing shading limit in light of
a* and b* channels, in first case Fig.2 especially for the green
mango the value of L*a*b* should be 5<= L<=85,-
40<=a<=20,-40<=b<=35 and in second case Fig.3 specially
for other colour mango the value of L*a*b* should be
0<L<100,10<a<60,-30<=b<=70.Then the morphological
activity of territory opening was connected to evacuate little
protests, trailed by shutting to fill gaps in leafy foods pixels
were dealt with as zero.
Detection of Mango Fruit
After the color thresholding process the image is converted to
binary image, then we use the strel function for structuring the
objects. Then detect the edge and use Randomized Hough
Transform (RHT) for detect the elliptical part because mango
is elliptical by shape. So we can detect the mango fruit.
Elliptical Fitting for Reorganization of Complete Fruit
As mango natural product has an inexact circle form, a basic
technique for elliptical fitting in light of picture minute used to
recognize if the imaged organic product has an entire shape.
This strategy for estimating ellipticity has been accounted for
to yield preferred exactness over different techniques [14-15]
and to have a decent mistake resistance (i.e., resilience of
flawed circle shapes, concerning mango natural product).
Three criteria were utilized to judge whether the organic
product form was finished:
1 Ellipse region: an entire natural product at a camera
separation of around 15cm-50cm ought to have a zone of
1000 to 1800 pixels. Littler patches could be foundation or
deficient organic product, while bigger patches could be
bunched natural product.
2 Area (A): the proportion of genuine associated segment (a
division aftereffect of a natural product) measure (A) to
Ascertain circle region in light of the fitted ellipse Major Axis
a and Minor Axis b, defined by:
Area (A )= π×a×b (1)
3 Bounding box length versus Ellipse real length: the length
of a bounding box simply epitomizing the natural product was
utilized as a part of estimation of the organic product length.
The real pivot of the fitted ellipse is typically bigger than the
length of the bounding box, in any case, thick stalk closes that
were not evacuated by the line filter could bring about an
overestimation of natural product length. Accordingly, if the
length of bounding box was 4 pixels bigger than the ellipse
major axis a, the protest was rejected from measure
estimation.
(a) (b) (c)
Figure 4: Ellipse fitting and find Major Axis and Minor Axis
(a) Elliptical area detected (b) Total Height and Width of the
fruit (c) red part is the major axis and green part is the minor
axis.
Size Estimation of Each Mango
Once an associated segment was perceived as a totally natural
product, a bounding box was made, being the littlest rectangle
containing the segment, from that we found the mango height
and width in pixel, the pixel is converted to ‘cm’, which will
be helpful for the measurement. As the shape is an elliptical
area so here we have to calculate the major axis (a) and the
minor axis (b), as we found the total height and width so after
dividing by 2 we found the major axis (a) and the minor axis
(b) Fig.4, so we can find the area of the fruit.
a=Width/2 (Major Axis) (2)
b=Height/2 (Minor Axis) (3)
Area (A )= π×a×b (4)
Mango Fruit Size Estimation
Figure 5: Block diagram for Mango fruit size estimation.
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 4 (2018) Spl.© Research India Publications. http://www.ripublication.com
8
Step 1: First we take the image which is taken by the
conventional RGB camera in a healthy weather.
Step 2: The colour threshold app which helps to threshold
colour images by manipulating the colour components of the
images based on different colour spaces. This is chosen to
removing the background.
Step 3: For removing the small objects from a particular
image, we use morphological operation. Here we use strel
function for creating structuring element.
Step 4: Basically for analyzing lines and curves in an image
the Hough Transform is used. But in case of detecting specific
curves like ellipse, it always faces the problem. So for
detecting ellipse, we use Randomized Hough Transform.
Step 5: The label block help to identify each object which is
connected to the background. The background pixels are
denoted as 0 in colour black and the object is in colour white.
The first object pixel is denoted as 1 then the next pixel is 2
and it continues as its number of the object. It helps to label
each object separately.
Step 6: After separate each object, we count each pixel of the
object and create a box for every object. We analyze its area
and count the pixel as width and height. Then we can find
each object’s height and width. (Major Axis & Minor
Axis).Here the pixel value is converted to ‘cm’ unit.
Result & Discussion We validate the proposed method in two ways (a) the sample
image having only one mango fruit (b) the sample images
having more than one mango fruit (Bunch). Here we take 10
number of the sample images having single mango fruit and
5 number of the sample images which contain more than one
mango fruit. The RGB image of on-tree mango as input,
then the colour thresholding process is occurs where we use
L*a*b colour space for remove background, as it is an on-
tree operation it cannot remove all the background.Then the
morphological binary operation occurs it help to remove the
small points. Then use RHT to identifying fruits then we
labeled each fruit by a bounding box, then we find major axis
and minor axis value then we calculate the area. The fruit
sample is collected from a mango garden for analysis. Prior
to validation by algorithm we measure the fruit size by
measurement tape, shown in Fig. 6. The 10 number of
samples having single mango fruit is validated by the
proposed algorithm and implied 3.07% error. Fig.10 contains
the graphical representation of Estimation of Actual and
Estimated Size of Single Mango Fruit The another way to
evaluate the performance of proposed algorithm is by taking
sample images having more than one mango fruit. Fig.8 (a)
cotain 2 number of mago fruit which manual measurement
area is 107.186 & 193.956 and algorithmic measurement
area is 106.06438 &189.50 which implied 1.0468%
&2.297% error. Fig.8(b) contain 4 number of mango fruit
which manual measurement is 049.25, 87.948, 164.9025,
106.794 and algorithmic measurement area is
39.29,79.18,142.32,96.147 which implied 20.22%,9.9695%,
13.6945%,15.2653% error. Fig 8(c) contain 2 number of
mango fruit which manual measurement area is 74.598 &
70.6725 and algorithmic measurement area is 69.91 & 66.94
which implied 6.284% & 5.2814% error. Fig.8(d) contain 4
number of mango fruit which manual measurement area is
84.808,84.82,118.72,115.9029 and algorithmic measurement
area is 70.769,69.90,110.8285,99.25 which implied
16.5657%,17.5902%,6.6494%,14.368% error. Fig.8(e)
contain 5 number of mango fruit which manual measurement
area is 108.364,90.69,70.6725,103.653,126.546 which
implied 5.2826%, 2.9331%, 11.2441%, 7.6409%,8.7909%
error. Fig.12 contains the graphical representation of
Comparison of Manual and Estimated size Measurement of
Each Mango of a Bunch. In Table 1 and Table 2 all the
information about fruit size is present.
Figure 6: Sample of measurement (a) measurement of Major
Axis (b) Measurement of Minor Axis.
Sample image of single mango fruit
(a) (b)
(c) (d)
(e) (f)
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 4 (2018) Spl.© Research India Publications. http://www.ripublication.com
9
(g) (h)
(i) (j)
Figure 7: Sample images for single mango fruit
Sample images of bunch mango fruit
(a) (b)
(c) (d)
(e)
Figure 8: Sample images of bunch mango fruit
Size Measurement of Fruit
Workflow of proposed method
(Single Mango Size Estimation)
(c)
(c) (d)
(e) (f)
(g)
Figure 9: Process of size estimation (a) Original Image (b)
L*a*b Image (c) Binarized Image (d) Spur removed
Image (e) RHT applied Image (f) Bounding Box for each
Mango fruit (g) Measure the Major and Minor Axis.
(a) (b)
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 4 (2018) Spl.© Research India Publications. http://www.ripublication.com
10
Table 1: Actual and Estimated Size of Single Mango Fruit.
0
20
40
60
80
100
120
140
160
Actual Estimated Error
Figure 10: Graphical Representation of Estimation of
Actual and Estimated Size of Single Mango Fruit
Workflow of proposed method
(Size Estimation Of Bunch Mango)
(a) (b)
(c) (d)
(e) (f) (g) (h)
[a= 3.1584 , b= 3.96] [a = 3.49, b = 7.35]
(i) (j) (k) (l)
[a=5.808, b=7.8016] [a=3.8026, b=8.052]
Figure 11(a-l): Process for size estimation of Mango Fruit in
Bunch (a) Original Image (b) Back ground Removed Image
(c) Label Each Mango of the Bunch (d) RHT applied Image
(e) Bounding Box of 1st Mango (f) Measurement of size of 1st
Mango (g) Bounding Box of 2nd Mango (h) Measurement of
size of 2nd Mango (i) Bounding Box of 3rd Mango (j)
Measurement of size of 3rd Mango (k)Bounding Box of 4th
Mango (l) Measurement of size of 4th Mango.
Input
Image
Actual Size Estimated Size % of Error in
Area cm2
Major
Axis(a
) cm
Minor
Axis(b
) cm
Are
a
cm2
Major
Axis(a)
cm
Minor
Axis(b
) cm
Are
a
cm2
(Estimated Area~
Actual Area)
/Actual Area × 100%
1 4.25 5 66.
74
3.8016 5.1744 61.7
86
7.4228%
2 4.75 4.5 67.
138
4.85 4.479 68.2
7
1.6861%
3 4.75 4.5 67.
138
4.68 4.548 66.8
85
0.3768%
4 7 5.5 120
.92
8.5 4.5 116.
81
3.52%
5 1.375 3.35 14.46
1.848 2.49 14.505
0.31%
6 1.375 3.35 14.
46
1.716 2.584 13.9
2
0.03%
7 4.5 5.5 77.
739
7
4.487 5.356 75.4
85
2.9003%
8 4.5 5.5 77.
7397
4.458 5.356 74.9
97
3.5281%
9 2.35 2.5 18.
453
2.14 2.487 16.7
2
9.4%
10 7.5 6 141.34
5
7.452 6.135 143.6
1.5307%
Average Error 3.07%
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 4 (2018) Spl.© Research India Publications. http://www.ripublication.com
11
Table 2: Comparison of Manual and Estimated size
Measurement of Each Mango of a Bunch
(a)
(b)
(c)
(d)
(e)
Figure 12: Graphical representation of Comparison of
Manual and Estimated size Measurement of Each Mango of a
Bunch
I
n
pu
t
I
m
ag
e
Fr
uit
Numb
er
Estimated Actual % of Error
in
Measurement of Area
M
ajor
Ax
is(a)(
cm
)
Mi
nor
Ax
is(b)
(c
m)
Ar
ea (c
m2
)
Majo
r Axis
(a)
(cm)
Mino
r Axis
(b)
(cm)
Area
(cm2
)
(Estimated
Area ~ Actual Area) /
Actual Area
× 100%
Aver
age Error
1 1 5.016
6.732
106.0
64
5.25 6.5 107.186
1.0468% 1.67
21%
2 5.87
10.27
8
189.5
0
6.5 9.5 193.956
2.2974%
2 1 3.1
589
3.9
6
39.
29
4 3.9 49.2
5
20.2274%
2 3.4
3
7.3
5
79.
18
4 7 87.9
48
9.9695% 14.7
89
3 5.808
7.801
6
142.3
2
6 8.75 164.9025
13.6945%
4 3.802
8.052
96.14
7
4.25 8.5 113.4686
25
15.2653%
3 1 4.75
4.686
69.91
4.75 5 74.5987
6.2852%
2 4.8
48
4.1
12
66.
94
5 4.5 70.6
725
5.2814 5.78
3
4 1 3.168
7.112
70.76
9
3.75 7.2 84.82
16.5657%
2 3.1
28
7.1
15
69.
90
3.75 7.2 84.8
2
17.5902% 13.7
9
3 4.3
56
8.1 11
0.8
258
4.5 8.4 118.
72
6.6494%
4 3.9
6
7.9
8
99.
25
4.5 8.2 115.
9029
14.368%
5 1 5.52
5.924
102.6
4
5.75 6 108.3645
5.2826%
2 5.1
96
5.3
94
88.
03
5.25 5.5 90.6
9
2.933
3 4.3
46
4
4.6 62.
72
6
4.5 5 70.6
725
11.2441% 7.11
7
4 5.264
5.79
195.7
33
5.5 6 103.653
7.6409%
5 5.396
6.848
115.8
01
1
5.75 7 126.546
8.4909%
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 4 (2018) Spl.© Research India Publications. http://www.ripublication.com
12
Conclusions & Future Work Here we proposed the Image Processing technique for on tree
fruit detection and measure each fruit size. In this paper, we
use colour thresholding, RHT & region detecting. The main
advantage of this paper is we detect every colour of mango
fruit and don’t face any problem regarding background.
The future work may be extended to improve in size
measurement section and try to measure the weight of each
fruit and analyse mature immature, which will be helpful for
the farmers.
References
[1] Gongal, A., et al. "Sensors and systems for fruit
detection and localization: A review." Computers and
Electronics in Agriculture 116 (2015): 8-19
[2] Payne, Alison B., et al. "Estimation of mango crop
yield using image analysis–segmentation method."
Computers and electronics in agriculture 91 (2013): 57-
64.
[3] Qureshi, W. S., et al. "Machine vision for counting fruit
on mango tree canopies." Precision Agriculture 18.2
(2017): 224-244.
[4] Omid, M., M. Khojastehnazhand, and A.
Tabatabaeefar. "Estimating volume and mass of citrus
fruits by image processing technique." Journal of food
Engineering 100.2 (2010): 315-321.
[5] Sethy, Prabira Kumar, Shwetapadma Panda, Santi
Kumari Behera, and Amiya Kumar Rath. "On Tree
Detection, Counting & Post-Harvest grading of fruits
Based on Image Processing and Machine Learning
Approach-A Review."
[6] Basca, Cosmin A., Mihai Talos, and Remus Brad.
"Randomized hough transform for ellipse detection
with result clustering." Computer as a Tool, 2005.
EUROCON 2005. The International Conference on.
Vol. 2. IEEE, 2005.
[7] Mustafa, Nur Badariah Ahmad, et al. "Classification of
fruits using Probabilistic Neural Networks-
Improvement using color features." TENCON 2011-
2011 IEEE Region 10 Conference. IEEE, 2011.
[8] Mendoza, Fernando, Petr Dejmek, and José M.
Aguilera. "Calibrated color measurements of
agricultural foods using image analysis." Postharvest
Biology and Technology 41.3 (2006): 285-295.
[9] Yam, Kit L., and Spyridon E. Papadakis. "A simple
digital imaging method for measuring and analyzing
color of food surfaces." Journal of food engineering
61.1 (2004): 137-142.
[10] Spreer, Wolfram, and Joachim Müller. "Estimating the
mass of mango fruit (Mangifera indica, cv. Chok Anan)
from its geometric dimensions by optical
measurement." Computers and electronics in
agriculture 75.1 (2011): 125-131.
[11] Moreda, G. P., Ortiz-Cañavate, J., García-Ramos, F. J.,
& Ruiz-Altisent, M. (2009). Non-destructive
technologies for fruit and vegetable size determination–
a review. Journal of Food Engineering, 92(2), 119-136.
[12] Wang, Ta Yuan, and Sing Kiong Nguang. "Low cost
sensor for volume and surface area computation of axi-
symmetric agricultural products." Journal of Food
Engineering 79.3 (2007): 870-877.
[13] Wang, Zhenglin, Kerry B. Walsh, and Brijesh Verma.
"On-Tree Mango Fruit Size Estimation Using RGB-D
Images." Sensors17.12 (2017): 2738.
[14] Rosin, Paul L. "Measuring shape: ellipticity,
rectangularity, and triangularity." Machine Vision and
Applications 14.3 (2003): 172-184.
[15] Stojmenovic, Milos, and Amiya Nayak. "Direct ellipse
fitting and measuring based on shape boundaries."
Pacific-Rim Symposium on Image and Video
Technology. Springer, Berlin, Heidelberg, 2007.
International Journal of Applied Engineering Research ISSN 0973-4562 Volume 13, Number 4 (2018) Spl.© Research India Publications. http://www.ripublication.com
13