monitoring surface cleanliness of manufactured metal parts
TRANSCRIPT
IN DEGREE PROJECT MECHANICAL ENGINEERING,SECOND CYCLE, 30 CREDITS
, STOCKHOLM SWEDEN 2018
Monitoring surface cleanliness of manufactured metal parts using camera technique
EMIR POZZI COLAKOVIC
KTH ROYAL INSTITUTE OF TECHNOLOGYSCHOOL OF INDUSTRIAL ENGINEERING AND MANAGEMENT
Master of Science Thesis
KTH School of Industrial Engineering and Management
Department of Production Engineering
Division of Manufacturing and Metrology Systems
SE-100 44 STOCKHOLM
Monitoring surface cleanliness of
manufactured metal parts using
camera technique
Emir Pozzi Colakovic
-2-
Abstract
Technical surface cleanliness is a mandatory requirement for many production lines. It is part of
the quality control process to ensure that the surfaces of the manufactured components are free
of contaminations and are ready for the next step in the production line. The project has taken
into account two different requirements related to the component cleanliness: first, the
cleanliness is measured as the level of contamination on a sample and then, the particles sizes are
determined. An important factor in the cleanliness is the presence of particles larger than a
certain limit, called the critical particles, which have to be detected. This thesis is inspired from an
issue SCANIA has (2018) in its production plant in Södertälje, Stockholm. SCANIA has a quality
control system that analyzes the contamination level of few components through a time
consuming process that takes hours and requires expensive microscopes, human intervention and
a dedicated measurement room in the plant. The aim of this thesis is to investigate an alternative
technical cleanliness monitoring method through the image analysis of the contamination
samples. Through a simple digital camera, pictures from the contaminated samples are taken,
processed and analyzed in order to obtain the cleanliness level and the particles size of the
samples. In contract to the current solution, the proposed method has the possibility of being
implemented in the production line, providing larger sampling rate.
-3-
Sammanfattning
Teknisk ytrenhet är ett obligatoriskt krav för många produktionslinor. Det är en del av
kvalitetsstyrningsprocessen för att säkerställa att ytorna på de tillverkade komponenterna är fria
från föroreningar och är redo för nästa steg i produktionslinan. Projektet har tagit hänsyn till två
olika krav relaterade till komponents renhet: för det första mäts renheten som föroreningsnivå på
ett prov och sedan bestäms partikelstorlekarna. En viktig faktor vid renhetsbestämning är
närvaron av partiklar som är större än en viss gräns, som kallas de kritiska partiklarna, som måste
detekteras. Avhandlingen är inspirerad av ett problem SCANIA har (2018) i sin
produktionsanläggning i Södertälje, Stockholm. SCANIA har ett kvalitetskontrollsystem som
analyserar föroreningsnivån av få komponenter genom en tidskrävande process vilken tar timmar
och kräver dyra mikroskop, mänskligt ingripande och ett dedikerat mätrum i anläggningen. Syftet
med denna avhandling är att undersöka en alternativ teknisk renhetsövervakningsmetod genom
bildanalys av förorenade proverna. Med en enkel digitalkamera tas bilder på de förorenade
proverna, bearbetas och analyseras för att uppnå renhetsnivån och partikelstorleken på proven. I
överensstämmelse med den nuvarande lösningen finns det möjlighet att implementera den
föreslagna metoden i produktionslinan, vilket ger större samplingsfrekvens.
-4-
Table of Contents
Abstract ........................................................................................................................................................................... 2
1 Introduction .......................................................................................................................................................... 5
1.1 Technical cleanliness - a background ....................................................................................................... 5
1.2 Motivation and problem statement .......................................................................................................... 6
1.3 Project Description ..................................................................................................................................... 6
1.4 Previous Work ............................................................................................................................................. 7
2 Theoretical Background ...................................................................................................................................... 7
2.1 Digital Image ................................................................................................................................................ 7
2.2 Image Segmentation ................................................................................................................................... 7
2.2.1 Image Histogram................................................................................................................................ 8
2.2.2 Thresholding ....................................................................................................................................... 8
2.2.3 Edge Detection .................................................................................................................................. 9
3 Method .................................................................................................................................................................10
3.1 Setup ............................................................................................................................................................10
3.2 Image Processing.......................................................................................................................................11
3.2.1 Background Removal ......................................................................................................................11
3.2.2 Binary Conversion ...........................................................................................................................12
3.3 Image Analysis ...........................................................................................................................................13
3.3.1 Scale Conversion ..............................................................................................................................14
3.3.2 Particles Detection ...........................................................................................................................16
4 Results and validation ........................................................................................................................................18
5 Discussion ...........................................................................................................................................................19
5.1 Threshold ...................................................................................................................................................19
5.2 Structural Element ....................................................................................................................................20
6 Future Work ........................................................................................................................................................21
7 Conclusion ...........................................................................................................................................................22
7.1 Particles Type .............................................................................................................................................23
7.2 Execution Time .........................................................................................................................................23
List of Figures ..............................................................................................................................................................24
Appendix A ..................................................................................................................................................................25
Appendix B ...................................................................................................................................................................28
Bibliography .................................................................................................................................................................29
-5-
1 Introduction
1.1 Technical cleanliness - a background
Technical cleanliness is a fundamental requirement for many components in various industries.
Many engineering fields such as aerospace, automotive and electrical engineering require high
levels of cleanliness. In these industries, it is necessary to assure that the components are free of
contaminations to increase the component’s lifetime, reliability and performance and decrease the
risk of failure and function loss. Contaminations usually occur during the manufacturing of the
component as a result of all the machining processes such as drilling, cutting, and milling. It can
also appear during bad handling and storing stages. Particle size plays an important role in
automotive industry. Particles larger than a certain level can damage the final product or affect its
functionality by e.g. blocking a fluid flow through a tube (Gommel, et al.). Large particles can
damage diesel injection systems, this can cause an overstressed or short circuit in a printed circuit,
blocking nozzles, congesting valves, damaging bearings (Dzetzit, et al., 2017). Contaminations
can cause problems in electrical engineering applications as they may cause short circuits
(Piscani, 2017). Technical cleanliness inspection starts with the particle collection from the
manufactured parts. The collected samples are then inspected through a microscope according
to the international standards where the particles detected and measured (Piscani, 2017).
A cleanliness inspection process is done in two steps: sample preparation and inspection.
Sample preparation: there are two ways to prepare a sample presenting the component’s
contamination. In the first approach, the manufactured part is washed and the
contamination is collected through a membrane filter. The dried filter is then used as a
sample for the next step. The second approach is done manually through an operator
who applies an adhesive membrane on the component surface. The collected samples are
then ready to be inspected with a microscope. (Piscani, 2017)
Inspection: It starts by placing the sample/dried membrane on the microscope to capture
images. The images are later analyzed and evaluated by microscope’s image processing
software. The particles are detected as dark areas against a lighter background. Particles
are measured through different parameters such as maximum dimension or equivalent
circle diameter, equivalent circle area. Measuring the particles permits to classify them in
groups and make comparisons. (Piscani, 2017)
The relevant contaminants on a part are mainly metallic due to their hardness and conductivity
and will be the focus of the technical cleanliness of manufactured metal parts. These particles
appear as dark on the picture and mostly generated by all machining operations on the
component (Kohli, et al., 2015).
-6-
1.2 Motivation and problem statement
The motivation behind developing a new technique to operate the technical cleanliness
inspection started with a problem SCANIA faced today regarding the cleanliness inspection of
cylinder heads for their truck engines. Today at Scania, the cleanliness monitoring is an off-line
procedure where samples are first collected from a few components (3 out of 1000 cylinder head
per day) and then they are submitted to a laboratory for inspection. Three different sampling
methods are used: stamps, tape and mobile particle vacuum extraction unit. The methods are all
operator-dependent as both the preparation and inspection are done through visual judgement.
In order to inspect the critical particles, the operator uses a microscope. For SCANIA critical
particles are larger than 600μm. The particle collection is done in batches of 3 cylinder heads and
takes from 3 to 4 hours in the laboratory. The measurement of particles takes another 4 hours
leading to a time consuming process. In an 8 hours shift working day where around 1000 cylinder
heads are manufactured, only 3 cylinder heads are measured in SCANIA. These factors make the
current solution adopted by SCANIA inefficient. First, there is a low number of measurements,
the system is therefore not reactive and cannot give immediate feedbacks in case of issues. The
quality control routine is not an in-line process and requires taking the component out of the
production line. A dedicated room and particular microscope are used making this solution
expensive and the output is an operator-dependent visual judgement.
1.3 Project Description
Particles samples are in form of dark contamination on a bright background. The main goal of
this project is to develop an inspection method based on using camera systems and image
processing algorithms to substitute the human visual inspection with an automatic procedure.
The method captures pictures with a digital camera, processes it and analyzes the quantity and the
type of the particles. It permits to complete an inspection in a few seconds, in a reliable and more
standardized way than the current solutions. The proposed method makes it possible to analyze
each component fast and give an immediate feedback. Moreover, the developed algorithms make
it possible to collect much more data in a structured way. For a complete automation, the
solution could be combined with an automatic sample collection phase.
In order to obtain digital pictures as an input for the software a small portable digital microscope
was used. Numerous samples have been tested on the algorithm during the development of the
software in MATLAB. The software was able to give two outputs out of a particular image taken
from the microscope of a contamination stamp:
1. Output1: The sample image is Clean or Not Clean
2. Output2: Critical Particles Presence
In order to do that the software takes in the image, operates some processing as cutting the
picture, removing shades to obtain a homogenous background and converts the image to a binary
matrix of groups of 1s and 0s in which each value defines the presence or absence of the
contamination in the surface. As a result, the software measures and detects critical particles.
-7-
1.4 Previous Work
The literature review showed a lack of image analysis applications for technical cleanliness.
However, few studies were done in terms of particle characterization (Vecchio, et al., 2012). In
‘Detection of Surface Cleanliness based on Image recognition’ (Yanyan, et al.) an image analysis
method was described to detect oil and nick contaminations and defects on industrial touch
switches for an objective and more accurate evaluation of cleanliness with no human
interference. As mentioned in the paper, the surface cleanliness are mostly dominated by naked-
eye observation, weight testing and solvent testing methods. The paper presents a method of
using two different degrees of light to detect separately both types of surface objects.
Image analysis has been also applied for quality control of paddy (SenthilRajan). The main
approach utilized in the paper is to classify paddy types through analyzing the gray-scale image
and dividing the detected areas into their dimension by the number of pixels in the area,
categorizing the paddy for its overall dimension. In this sense, the paper presents a similar idea of
having an in-line quality control system based on image analysis that measures detected objects
dimensions, in this case to categorize the product.
Most of image analysis applications within technical cleanliness rely on sophisticated microscopes
such as scanning electron microscope (SEM). (Shia, et al., 2003). Simpler solutions as optical
microscopes are economically advantageous but not practical for an automated and in-line
cleanliness evaluation. All of these methods are most importantly time consuming (Shia, et al.,
2003). Therefore, this thesis proposes a different method by using a simpler and less costly digital
camera for technical cleanliness.
2 Theoretical Background
Particle measurements are done on the images through the software developed in MATLAB. It is
important before explaining the software to give a theoretical background on digital images and
image processing techniques.
2.1 Digital Image
A digital image is a two-dimensional array, also defined as matrix in which each element of the
array corresponds to a pixel of the image. In colored images, the image is defined by 3 different
matrixes, big as the image itself, for each primary color, R for red, G for Green and B for blue.
Since colors are not relevant for the application, we will focus on gray-scale and binary picture. A
gray-scale image is a single matrix whose values represent an intensity varying within a range. A
gray-scale image is defined as normalized when its values are fractional, going from 0 to 1, from a
black to a white pixel. (Gonzalez, et al., 2007)
2.2 Image Segmentation
Once we have represented a digital image as a matrix, we can apply all the mathematics related to
matrixes in order to manipulate and operate an analysis on the image through image
segmentation techniques.
-8-
2.2.1 Image Histogram
Each gray-scale image can be represented with a histogram. The histogram of a digital image
which values vary from 0 to 1 is a discrete function which shows the number of pixels for each
grayscale value in the image. The values on the x-scale vary from 0 - black - to 1 – white - and the
y-values are natural numbers. An image that tends to go to darker shades will have an histogram
with high values concentrated on the left side of the histogram close to 0, and the opposite will
happen for a picture which colors are brighter. (Gonzalez, et al., 2007)
An example of an image histogram is shown in Figure 2-1. As most of the image consists of a
gray background, the histogram has a peak between 150 and 200, gray pixel values. The
histogram shows a smaller peak on the left side corresponding to the black areas pixels.
Figure 2-1, Image and related histogram
2.2.2 Thresholding
In case of a gray-scale image having an object and a background the image histogram will show
pixels intensity values grouped in two big groups as shown in Figure 2-2.
Figure 2-2, Histogram of a background and object picture (Gonzalez, et al., 2007)
The first peak on the left side of Figure 2-2 represents the background pixels while the smaller
peak on the right side presents the object pixels. The first peak has low intensity values (darker
gray levels) and the second peak covers high intensity values (lighter gray levels). To separate the
object pixels from the background pixels, we use the image histogram and define a threshold
level, T, to separate these two groups. All pixels having a value inferior to the threshold T will be
set to 0 intensity (black) and all pixels having intensity bigger than T will be set to 1 (white). This
way the grayscale image has been simplified to a matrix of 0s and 1s called binary or logical
matrix. In the images studied for this thesis, the sample corresponded to the light background
and the particles to the object with darker pixel intensity. It is possible to represent object and
-9-
background in the opposite way, interchanging 0 and 1 values. For our applications having 1
values (white pixel) for the particles permits to do calculations on the cleanliness more easily. An
example of the result of a binary conversion after defining a threshold value is shown in Figure 2-
3.
Figure 2-3, Binary conversion (MathWorks, 2018)
2.2.3 Edge Detection
It is important in many applications to be able to detect edges of objects, contours and lines. A
digital edge is a sharp transition in the pixel intensity values as shown in Figure 2-4.
Mathematically transitions in a function values are calculated through the first derivative and for a
two-dimensional function, as an image, through its gradient. (Gonzalez, et al., 2007)
Figure 2-4, Digital Edge (Daemi, 2014)
Many edge detection algorithms are described in literature and applied to image segmentation
problems (Gonzalez, et al., 2007). Among these algorithms Canny edge detector is broadly used
for various applications. The Canny edge detector has 4 steps (Canny, 1986):
1. It smooths the image by Gaussian filter. The image is blurred to remove the noise.
2. It calculates the Gradient magnitude and direction. The gradient is the vector that
indicated the direction of maximum growth in a two-dimensional function. As edges are
sharp intensity changes, gradients will be perpendicular to the edges and their direction
will define the local edge direction.
3. Applying non-maxima suppression to the gradient magnitude image.
4. Using double thresholding and connectivity analysis to detect and link the edges.
Threshold the image to reduce false edges.
An example of Canny edge detection can be seen in Figure 2-5. The edge detection is used in the
software for the scale conversion part.
-10-
Figure 2-5, Canny edge detection of the image scale
3 Method
The algorithm is explained in all its steps defining the role of each built-in function used in
MATLAB.
3.1 Setup
The pictures were taken with a digital camera:
1. A Dino-Lite Edge Digital Microscope am7915mzt was used to take the sample pictures
(Figure 3-1).
Figure 3-1, Digital Microscope used for the project (Dino-Lite)
2. The scale of the images was defined as a segment on the picture and a corresponding
value in mm already set in the program. Pictures should be taken with a fixed zoom level
and distance to the object. The developed algorithm can be calibrated for different scales
before being applied into the quality control.
3. Images with resolution of 2560x1920 were captured by the camera.
The developed software consists of two functions: calibration and regular application. The
calibration is done once to convert the pixel values to millimeters since the size of the critical
particles are expressed in micrometers by using the calibration function. Once that is done, the
regular application function which lacks the scale conversion part will be used for monitoring.
-11-
3.2 Image Processing
The software starts with some standard and common image processing operations to treat the
image and prepare it for an optimal analysis. The image is read, converted to grayscale as color do
not matter for our application, every possibly existing text on the picture is removed and it is
normalized in order to have fractional values of the pixel intensities that would be otherwise
integers bigger than 1. The normalization process is essential to the black and white conversion
as it makes easier to define a threshold and apply it to convert the picture in binary values. The
picture is finally blurred through Gaussian filter to remove some noise. It has been observed that
blurring interferes with the edge detection needed for the scale conversion, therefore a separate
grayscale image is saved as Igray_line to be later used in the program for that purpose. The
output of these preliminary operations can be seen in Figure 3-2, grayscale image of the sample.
Figure 3-2, Grayscale image
The picture used for calibrating must have a scale (bottom left of the picture). The picture could
be saved from the digital microscope with the scale shown in a white external frame. This would
critically facilitate the edge detection on the scale lines for future work.
3.2.1 Background Removal
After some initial processing operations, the software removes any possible shade in the image.
This process appeared to be the slowest in the program. The operation is done defining a
structural element ‘se’ used by the functions ‘imbothat’. The functions operates the bottom-hat
filtering on the grayscale image, doing a morphological closing of the image and automatically
subtracting the result from the original image (MathWorks, 2018). The filter permits to obtain a
more homogenous image in which no shades are present. Shades could result in darker areas with
no particles in the image to be detected as particles. Different structural element dimension
values have been tested on different images before concluding that 150 pixel gives reasonably
good results.
-12-
Figure 3-3, Background removed image
For visualization purposes, the image can be contrasted. No additional information is being
added in the process. It is possible to see in Figure 3-3 the result of the background removal
operation.
3.2.2 Binary Conversion
A critical part of the program is the conversion of the image from grayscale and fractional values
of pixel intensities to black and white with logical values of 0 and 1. MATLAB has a built-in
functions in order to find optimal thresholds for different pictures called ‘graythresh’ that utilized
Otsu’s method, which chooses the threshold that minimizes the interclass variance of the black
and white pixels (Otsu, 1979). It has been observed that percentage values of the threshold level
give better results than the level by itself. Starting from a 0.95 percentage of the level and
decreasing of 0.05, 0.8 appeared through different tests on different images to give the best
detection. Once an optimal threshold level is defined it is possible to convert the image into
binary values.
In order to convert the image into binary values it is common to use an inequation. For each
matrix element in which the pixel intensity is lower than 0.8*level we define a new element with a
‘true’ value corresponding to 1 in the pixel. The image shown from the matrix will appear white
in that point. The opposite will happen if the inequation is not satisfied. Figure 3-4 shows the
result of the image binary conversion.
-13-
Figure 3-4, Binary image
3.3 Image Analysis
Once we have obtained a processed binary image we are able to conduct an analysis for particle
detection. Particles are clusters of 1-values in the logical matrix. Through the ‘regionprops’
function in MATLAB we are able to find areas and related dimensions of these groups of the
same values. The regionprops function takes a binary image as input and gives different
properties of the image in a struct variable (MathWorks, 2018).
In order to avoid having the scale value being detected a particle it is cropped out of the picture
setting all pixel values in the region to 0 (BWc(end-150:end,270:500)=zeros()). Among all
the possible parameters of regionprops the most relevant for our application are:
1. Area. Actual number of pixels in the region, returned as a scalar value.
2. Pixel List. Locations of pixels in the region.
3. Major Axis Length. Length (in pixels) of the major axis of the ellipse that has the same
normalized second central moments as the region.
4. Minor Axis Length. Length (in pixels) of the minor axis of the ellipse that has the same
normalized second central moments as the region. (MathWorks, 2018)
It is possible to have an initial particle detection without scale detection around the median value
of the particle area: the program calculates the median value of all the areas, find the index values
of all areas bigger than the median value (ind=area>m), then plots all the pixels in these areas as
red dots on the original image. The result is shown in Figure 3-5:
-14-
Figure 3-5, Visualization of particles bigger than the median
It is visible in the image how the region around the scale value has not been detected as
contamination as we set the matrix values in that area to be zero.
3.3.1 Scale Conversion
In order to find the critical particles it is necessary to be able to express each measure done
through the regionprops function in mm or μm instead of pixels. To do so it is possible to use
the scale plotted on the bottom left of the picture. Each line has a length corresponding to the
value in mm in the box, 1.0 in case Figure 3-6. If we are able to calculate the length of these lines
in pixels, we can define a correspondence between the two units of measurement.
First, we need to define a region of interest. The pixel values have been found through the
MATLAB image plot interface that permits to see the position of each pixel pointed on the
picture. Once we have a region of interest around the scale (L-shaped, with the two lines) we
apply the canny edge detection to have defined pixels in a binary image for the edges of the lines.
The values in the canny edge function, threshold and sigma, were found testing different values
on different images.
Each line of the scale appears to be as a thin area and therefore two lines are detected for each
line of the scale as in Figure 3-6.
-15-
Figure 3-6, Edge detection to the scale
As we are interested in the scale solely we need a way to remove all the particle edges detected.
One method is to determinate all the areas through regionprops, calculate the maximum which
will be the scale itself and delete the index values of all the others. The result is shown in Figure3-
7:
Figure 3-7, Edge detected scale
It is possible to calculate the length in pixels of the scale lines by determining the indexes of the
top pixel at the highest position and an estimation of the bottom pixel. The bottom pixel position
will be calculated as an average as the bottom line of the scale has a certain thickness which
causes the edge detection to find two different parallel lines. In order to find the indexes the
‘find’ function will be used. The ‘find’ function determines ‘returns a vector containing the linear
indices of each nonzero element in the array’, nonzero values in the scale picture are all the edge
pixels (MathWorks, 2018). As the index values grow from top to bottom in a matrix the top-pixel
will correspond to the minimum y-value. The bottom value will be found in a similar way but as
an average of minimum and maximum y-value of all the nonzero elements indexes. In order to
simplify the process the images is cropped as shown in Figure 3-8:
-16-
Figure 3-8, Cropped region of interest of the scale
As the scale consists of two lines it is possible to calculate the lines length in pixels along both
and get a better evaluation through averaging the two values. Finally, the conversion will be
determined as:
um_pixel=1000*scale_of_image/scale_length;
3.3.2 Particles Detection
The particles measures can finally be expressed in micrometers and compared to the CPM -
Critical Particle Dimension.
MajorAxisLength = MajorAxisLength*um_pixel;
MinorAxisLength = MinorAxisLength*um_pixel;
CPMidx = find(MajorAxisLength > CPM | MinorAxisLength > CPM);
In order to define the particle as critical the code requires one of the two dimensions to exceed
the critical value. In a two-dimensional environment, particles can appear as granules, with
comparable dimensions or fibers, with a predominant dimension over the other. In case textile
contamination (appearing as fiber) is not an issue and does not need to be detected, it is possible
to implement in the code the recognitions of these kind of particles. As textile particles have a
predominant dimension, their length, and it is possible to set a value of 10% in the maximum and
minimum dimension ratio of the particle/fiber so that everything that is not a granule, with
comparable maximum and minimum dimension and ratio of the two value close to 1, is not
considered. A case of detected textile or hair is shown in Figure 3-9:
-17-
Figure 3-9, Detected textile contamination
Critical particles can be finally plotted on the original image as shown in Figure 3-10:
Figure 3-10, Detected critical particles
The critical particle detection is not the only output as in addition the cleanliness is measured.
The cleanliness can be defined as the percentage of the particle area on the whole area. This is
easily calculated through the binary image as each pixel of value of 1 represents a point in in a
particle area: by summing all these points and calculating the percentage on the image area we
have a definition of cleanliness. This definition of cleanliness is dependent from the zoom level
of the image. By standardizing the image acquisition, we can have comparable cleanliness levels.
-18-
4 Results and validation
The software has been tested on many different pictures taken following the requisite of having
standardized pictures with a fixed microscope height and zoom level. All images were tested on
the ‘operating’ version of the function with no scale detection. The function takes as input the
conversion value of the calibration functions therefore normal pictures do not need a scale
represented as previously shown in the Method section. Critical particles are highlighted as red
clusters on the original image. In case no particles are detected, the software gives a negative
output. Some of the results will be shown below: the original image taken with the digital
microscope will be on the left and the software result showing the particle detected on the right.
In case the software does not find any particle, nothing will be highlighted. More results can be
found in the Appendix B
-19-
Figure 4-1, Particle Detection, Software Results
In order to validate the software a possibility is to compare the software results to more classic
methods utilizing microscope. Unfortunately, this has not been possible for the project. For
future work, this would definitely be a great line of improvement for the project and it could help
to have a better cleanliness evaluation in addition to finding critical particles, as both evaluations
are independent and important for quality control.
5 Discussion
In order to have a meaningful particle detection it was necessary to experiment on certain
parameter values. Among these, the parameters that affected the program the most were the
threshold value for the binary conversion and the structural element dimension for background
removal.
5.1 Threshold
It is hard to establish in many cases if a darker area is a simple shade or an actual particle. The
software should be validated through comparison with a trained eye quality control operator. The
main factor it is possible to experiment with in order to avoid the detection of darker areas as
particles is the threshold level. It is shown below how the threshold influences the shades
detection over increasing values of the Otsu’s threshold level from the optimal value of 80%
(original- 80-85-90-95-100).
-20-
Figure 5-1, Effect of varying the threshold
value on the particle detection
It can be seen how in these kind of non-homogenous pictures with great variance in the pixel
intensities the software can struggle to recognize particles. Higher threshold values make dark
regions to appear as particles: these regions are then converted to the binary image as particle
areas and considered by the ‘regionprops’ functions as clusters. Images corresponding to values
close to 100% of the Otsu’s threshold value how the software produces a wrong detection.
Testing the software on the highest number of different images permitted to find the right set of
values in order to have a reasonable detection.
5.2 Structural Element
The second other important element playing a fundamental role in a functioning detection is the
structural element dimension utilized for background removal. Small values of the structural
elements that would generate a faster code go against the particle detection aim as they filter out
big particles. Very high values of the structural element dimension though aggregate separate
particles in one resulting in false positive – critical particles that are actually two separate
noncritical particles. Various tests on different types of images (clean, very contaminated,
homogenous, full of shades) suggested a final value around 150 pixels. Above it is shown the
effect of the structural element diameter variation on the binary conversion and therefore the
particle detection from values going from 30 to 150 increasing each time of 30.
-21-
Figure 5-2, Effect of varying the structural element
dimension on image binary conversion
6 Future Work
The scale value is manually set in the software. As long as pictures are taken in a standardized
way this should not be a big concern, as the scale value would not change. In case images are
taken through different zoom level and/or camera positions, the scale would change and the
software should be able to detect the scale value automatically. One way could be applying a
OCR detection able to analyze the region of interest including the scale value – for example ‘5
mm’ – turning into the numerical value the software needs in order to perform the conversion of
pixels into micrometers. A less sophisticated method could be adding the numerical scale value to
each image in a text file or similar. This method would not require an OCR detection which
appears often to be inaccurate.
-22-
Another interesting possibility for this software is to apply machine-learning techniques to find
patterns in the particles shapes and appearance and build a database around them. In order to do
so the software should include new parameters able to define a particle shape and dimension.
Such system could give the possibility for root causes identification, patterns identification and
therefore to make predictions in the components cleanliness level.
In the project, the main particle dimension utilized was the biggest length of the particle, also
called Feret diameter. The dimension can be seen as the result of a measurement done with a
caliber or the mean value of the distance between pairs of parallel tangents to a projected outline
of the particle (ASTMF187716). In order to detect the critical particles the Feret diameter is
sufficient but for a better particle characterization, we need more parameters. Some additional
measures to define a particle and somehow measure its shape have been taken from the ASTM
F1877−16 ‘Standard Practice for Characterization of Particles’ and are:
1. ECD, equivalent circle diameter, for size measurement.
2. AR (aspect ratio), for shape evaluation, defined as the ration between the maximum and
minimum diameter.
3. Roundness R, for shape evaluation, varying from 0 to 1 with 1 for perfect circle.
4. Form factor FF, similar to the roundness but based on the particle perimeter instead of
the area.
The particle color is a limitation in the projects as only dark particles can be detected. It is
possible to use different degrees of light to detect different colors of contaminations, mainly dark
and light (Yanyan, et al.). The illumination method can be changed too. Instead of having the
light source coming from the microscope the sample could be enlightened from the bottom in
order to have a more homogenous picture and more accuracy in the particle detection as there
will not be shades in the pictures. As the background removal steps are the most time-consuming
in the software, better light would require less background removal and it would make the
software faster.
7 Conclusion
Having the issue of detecting critical particles on metal components surface, the software shows
the capacity to detect critical particles in a few seconds compared to time-consuming operations
through non-digital microscope. Moreover, the new method permits to collect data, inspect more
components and gives the possibility of an automatized in-line quality control. Compared to the
previous situation, a technical cleanliness quality system based on image analysis is able to reduce
a work done in hours to a 12 seconds operation. The new method offers the possibility:
To inspect more components
To collect a big quantity of data
To automatize the monitoring process
-23-
7.1 Particles Type
It is clear that the software recognizes particles as the darkest regions in the image. In case of
light critical particles, the software would not succeed. However, most of the critical
contamination is metallic which appear dark through a camera picture (Kohli, et al., 2015).
7.2 Execution Time
The software takes around 12 seconds per execution. The bottleneck is in few lines: the
background removal part takes around 75% of the execution time, as showing in Figure 6-1. The
bigger the structural element the longer the time. As previously explained low values of the ‘se’ –
structural element - are impossible to use. In order to improve the time efficiency of the code, in
case that appears to be an issue, is it necessary to find an alternative for background removal.
Figure 7-1, Time required in percentage by the program steps
10% - Image Processing 75% -
Background
Removal
15% - Particle Detection
-24-
List of Figures
Figure 2-1, Image and related histogram ................................................................................................................... 8
Figure 2-2, Histogram of a background and object picture (Gonzalez, et al., 2007).......................................... 8
Figure 2-3, Binary conversion (MathWorks, 2018) .................................................................................................. 9
Figure 2-4, Digital Edge (Daemi, 2014) ..................................................................................................................... 9
Figure 2-5, Canny edge detection of the image scale .............................................................................................10
Figure 3-1, Digital Microscope used for the project (Dino-Lite) ........................................................................10
Figure 3-2, Grayscale image .......................................................................................................................................11
Figure 3-3, Background removed image ..................................................................................................................12
Figure 3-4, Binary image .............................................................................................................................................13
Figure 3-5, Visualization of particles bigger than the median ..............................................................................14
Figure 3-6, Edge detection to the scale ....................................................................................................................15
Figure 3-7, Edge detected scale .................................................................................................................................15
Figure 3-8, Cropped region of interest of the scale ...............................................................................................16
Figure 3-9, Detected textile contamination .............................................................................................................17
Figure 3-10, Detected critical particles .....................................................................................................................17
Figure 4-1, Particle Detection, Software Results ....................................................................................................19
Figure 5-1, Effect of varying the threshold value on the particle detection ......................................................20
Figure 5-2, Effect of varying the structural element dimension on image binary conversion ........................21
Figure 6-1, Time required in percentage by the program steps ...........................................................................23
-25-
Appendix A
function ImageAnalysisCleanliness
% OUTPUT 1: Cleanliness.
%Once the picture is processed and converted to binary the cleanliness can
%be calculated as the percentage of the particle area. The number of dirt
%particles is calculated too.
% OUTPUT 2: Maximum particle dimension.
%The programs takes in 2560x1920 pictures with the scale put in the bottom
%left and some info on the top of the image which are cropped out.
%*****SET INITIAL DATA*****
CPM = 600; %Critical particle dimension in um. Chosen by the USER based on
the application
scale_of_image=1; %Value in mm. It could be detected through OCR for a more
%automized process
%*****IMAGE PROCESSING*****
% 1. Loading picture, grayscale conversion, normalization picture
I = imread('Test6.jpg');
Igray = rgb2gray(I);
Igray = Igray(60:end,:); %the image needs to be cropped initially
Igray = im2double(Igray);
Igray = (Igray-min(Igray(:)))/(min(Igray(:))+max(Igray(:)));
Igray_line=Igray; %use this later for line detection as we can't blur
Igray = imgaussfilt(Igray,2);
figure(1),imshow(Igray); title('Grayscale Picture');
% 2. Background removal, uniforming lightning
se = strel('disk',130);
B = imbothat(Igray,se); %computes the morphological closing of the image
%(using imclose)and then subtracts the original image from the result.
Bc = 1-B; %complementing the matrix
figure(2), imshow(Bc); title('No background Grayscale Picture');
% 3. Applying contrast (for visualiziation only)
se = strel('disk',10);
J = imsubtract( ima');dd(B,imtophat(B,se)) , imbothat(B,se));
figure(3), imshow(1-J); title('Contrast Enchancement');
% 4. Automatic thresholding for the BW conversion
level = graythresh(Bc); %the function uses Otsu's method for automatic
%thresholding
BW=Bc<0.8*level; %0.8 tested to be the best-important test different
%percentages of the threshold
figure(3), imshow(BW); title('BW Image');
%*****IMAGE ANALYSIS*****
% 5. Particle Detection
BWc=BW;
BWc(end-150:end,270:500)=zeros(); %crop out the scale value that would be
%detected as dirt
%***Particles Properties Definition***
-26-
s=regionprops(BWc,'Area','PixelList','MajorAxisLength','MinorAxisLength');
area=cat(1,s.Area);
MajorAxisLength=cat(1,s.MajorAxisLength); %use this value later for CPM
%comparison
MinorAxisLength=cat(1,s.MinorAxisLength);
%*************************************
m=median(area);
ind=area>m; %ind is a logical matrix in which the 1 values corresponds to
%all the clusters in s having an area bigger than the median
sm=s(ind); %******************************s=sm*****************************
PL=cat(1,sm.PixelList);
figure(4),imshow(Igray); title('Particles Detected'); hold on;
line(PL(:,1),PL(:,2),'lineStyle','none','marker','.','color','r');
% 6. Line detection finding bigger area (L-shape scale)
roi_line=Bc(1450:1780,1:350);
canny_edge=edge(roi_line,'Canny',0.2,1); %...nny',0.5,1)
figure(5), imshowpair(roi_line,canny_edge,'montage'),title('Scale Roi');
scanny=regionprops(canny_edge,'Area','PixelIdxList');
area=cat(1,scanny.Area);
mx=max(area);
ind=area>=mx; %ind is a logical matrix in which the 1 values corresponds to
scanny=scanny(ind);
Pind=scanny.PixelIdxList;
canny_c=zeros(size(roi_line));
canny_c(Pind)=1; %canny_c represent the scale L shaped area solely
%1-find the area with the maximum length with regionprops and set to 0 the
%smallers areas
figure(6), imshow(canny_c),title('Filtering out everything but the scale
pixels');
% 6.2 Find the max and min y values in the canny_c picture to obtain the
%scale length. Use function find to get the position of the upper and lower
%limit of the scale find gives the linear coordinates but we need the
%matrix ones
[Ycannyt,Xcannyt]=find(canny_c);%the values are represented in linear form
but we need them in matrix form
scale_top=min(Ycannyt); %since the values start from up to bottom in a
matrix we need the min for the top
canny_c_crop=canny_c(1:end,size(canny_c,2)*0.4:end);
figure(7), imshow(canny_c_crop),title('Cropped part in order to get bottom
value');
[Ycannyb,Xcannyb]=find(canny_c_crop);%the values are represented in linear
form but we need them in matrix form
scale_bottom= (max(Ycannyb)+min(Ycannyb))/2;
scale_top2=max(Xcannyt);
canny_c_crop2=canny_c(1:size(canny_c,1)*0.6,1:end);
figure(10), imshow(canny_c_crop2);
[Ycannyb,Xcannyb]=find(canny_c_crop2);
scale_bottom2= (max(Xcannyb)+min(Xcannyb))/2;
%The scale is calculated along the horizontal and vertical lines and then
%an average is computed
-27-
scale_length= abs(scale_bottom - scale_top);
scale_length2=abs(scale_bottom2 - scale_top2);
scale_length=(scale_length+scale_length2)/2;
um_pixel=1000*scale_of_image/scale_length; %micro meters per pixel
%each length in pixels can now be converted in micrometers and compared to
%the critical value CPM.
% 7. Counting Particles and Max particle dimension
MajorAxisLength = MajorAxisLength*um_pixel;
MinorAxisLength = MinorAxisLength*um_pixel;
CPMidx = find(MajorAxisLength > CPM | MinorAxisLength > CPM);
%where CPM is the critical particle dimension
scpm=s(CPMidx); %where s is the struct of ALL the particles properties
defined in line 49
PL=cat(1,scpm.PixelList);
number_critical_particles=size(CPMidx,1);
if number_critical_particles>0
figure(8),imshow(Igray); title('Critical Particles Detection'); hold on;
line(PL(:,1),PL(:,2),'lineStyle','none','marker','.','color','r');
else
figure(8),imshow(Igray); title('NO Critical Particles Detected')
end
%*****OUTPUTS*****
Cleanliness = 100 - (sum(BWc(:)) / (size(BWc,1)*size(BWc,2)) *100)
number_critical_particles
if number_critical_particles>0
X = ['The sample is NOT OK.'];
disp(X)
else
X = ['The sample is OK.'];
disp(X)
end
%The cleanliness measure depends from the zoom level and it can't be used
%for comparisons just like this. For example zooming in a dirt area would
%produce a picture that appears more dirty than before as the black to
%white area ratio increases.
end
-28-
Appendix B
-29-
Bibliography
1. ASTMF187716. Standard Practice for Characterization of Particles.
2. Canny, J. 1986. A Computational Approach to Edge Detection. 1986.
3. Daemi, Bita. 2014. “Image analysis for precision metrology: Verification of micro machining systems and
aerodynamic surfaces. Stockholm : s.n., 2014.
4. Dino-Lite. dino-lite.eu. [Online]
5. Dzetzit, Alexandra and Nagit, Gheorghe. 2017. Research on importance of cleanliness in
manufacturing reliable products for automotive. 2017.
6. Gommel, Udo and Kreck, Guido. A short Introduction to Cleanliness Technology:. Stuttgart : s.n.
7. Gonzalez, R. C. and Woods, R. E. 2007. Digital image processing. 2007.
8. Huang, Chengxue, et al. 2017. Technical Cleanliness. Developing a method for in-line measurements.
Stockholm : s.n., 2017.
9. Kohli, Rajiv and Mittal, K.L. 2015. Developments in Surface Contamination and Cleaning: Types of
Contamination and Contamination Resources. Volume 10. 2015.
10. MathWorks. 2018. https://se.mathworks.com/help/images/ref. [Online] 2018.
11. Otsu, N. 1979. A Threshold Selection Method from Gray-Level Histograms. 1979, Vols. IEEE
Transactions on Systems, Man, and Cybernetics, Vol. 9, 1.
12. Pall, Corporation. 2017. Component Cleanliness of Fluid Systems in Automotive. 2017.
13. Piscani, Francesco. 2017. An Introduction to Technical Cleanliness Inspection.
https://www.qualitydigest.com/inside/metrology-article/introduction-technical-cleanliness-inspection-
101117.html#. [Online] 10 11, 2017.
14. —. 2017. Cleanliness Inspection Process . https://www.olympus-ims.com/en/insight/increasing-
importance-technical-cleanliness/. [Online] 3 30, 2017.
15. SenthilRajan, A. Paddy Grade and Dirt Classification.
16. Shia, X.K., et al. 2003. Automatic recognition and evaluation of. Hong Kong : s.n., 2003.
17. Vecchio, I., et al. 2012. 3D Geometric Characterization of Particle Applied To Technical Cleanliness.
Karlsdorf-Neuthard, Germany : s.n., 2012.
18. Yanyan, H., et al. Detection of Surface Cleanliness Based on Image Recognition.
TRITA ITM-EX 2018:562
www.kth.se