automated inspection with machine vision · a large component of machine vision system design is...

Copyright © 2015. ConsulTech Engineering, PLLC. All rights reserved.

Automated Inspection

With

Machine VisionPart 2

Stanley N. Hack, D.Sc., PE

ConsulTech Engineering, PLLC

www.consultechusa.com

November 9, 2015

2Copyright © 2015. ConsulTech Engineering, PLLC. All rights reserved.

PRESENTATION GOALS

• Understanding Machine Vision and Its Uses

• Understanding Machine Vision Components

• Appreciating the Interacting Complexities of Machine Vision Components and Processing

Disclaimer:

• Most of ConsulTech Engineering’s clients regard their processes, which

include machine vision applications, as highly proprietary.

• Many of the applications presented have been devised for this presentation.


PRESENTATION SCOPE

Part I Review: CNY Engineering Expo 2014

• Machine Vision Definition

• Machine Vision Uses

• Machine Vision Components – Illumination

Part II: CNY Engineering Expo 2015

• Machine Vision Components

− Cameras and Sensors

− Optics

Part III: CNY Engineering Expo 2016

• Machine Vision Processors

• Machine Vision Software

• Machine Vision Systems

• Final Exam


MACHINE VISION DEFINTION

Machine Vision is defined as the technology and methods used to automatically inspect materials, components, and manufactured systems using image-based sensors and systems.

Key Words

• Automatic Inspection

• Materials, Components, Manufactured Systems

• Image-Based


MACHINE VISION DEFINITION

Machine Vision has the following attributes:

• The input is an image, and the output is a set of data, such as feature

existence, object type, object or feature location, and measurements.

• The system analyzes inanimate objects.

• The system has a priori knowledge of the imaged object(s), including

object shape, size, position, and attributes.

• The system performs its processing repeatedly and often rapidly.

• The imaging environment, including illumination, geometry, and

motion, is controlled by the Machine Vision system.


MACHINE VISION ENVIRONMENT

Machine Vision is unique among all of the computer imaging modalities (computer vision, image processing,

medical imaging, remote sensing) in that many aspects of the imaging environment can often be controlled

• Illumination

• Sensor positioning

• Sensor size and resolution

• Image magnification and orientation

• Optical filtering

• Reflections

A large component of Machine Vision system design is the optimization of the captured images to make the software’s job feasible, easier, and/or faster.


USES OF MACHINE VISION

• Quality Control

• Process Control

• Robotic Guidance

NDA Expiration 01 December 2007

Image courtesy of Matrox Electronic

Systems Ltd., Dorval, Quebec, Canada

Image courtesy of Matrox Electronic

Systems Ltd., Dorval, Quebec, Canada

Images courtesy of Microscan Systems, Inc.,

NERLITE® Lighting Solutions, Renton, WA



Manufacturing Quality Control

• Dimension verification

• Parts placement accuracy

• Debris detection *

• Label placement

• Label printing

• Coatings integrity

• Circuit continuity

• Color verification

• Cracks, dents, and other defects *

•••

* Little or no a priori knowledge of the imaged object(s).

Defect Inspection

Placement InspectionImages courtesy of Matrox Electronic Systems Ltd., Dorval, Quebec, Canada



Manufacturing Process Control

• Temperature control

• Cutting / grinding adjustments

• Parts sorting

• Flow and speed control

• Timing control

• Robot control

•••

Grinding Marks and Defects

Baking Temperature Control

Robot Welding


MACHINE VISION COMPONENTS

Work station image courtesy of Comark LLC, Milford, MA


Machine Vision Components

The Three Most Important Components

• Illumination

• Illumination

• Illumination

The Other Most Important Components

• Cameras and Sensors

• Optics

• Synchronization (Motion Control)

• Processing

Image courtesy of Operations Technology, Inc.,

Blairstown, NJ

Detailed in Part 1

A large component of Machine Vision

system design is the optimization of the captured images to make the software’s job feasible, easier, and/or faster.


Illumination Review

Automated Inspection with Machine Vision Part 1 summarized in two theorems:

Theorem 1

If adding shadows to a captured image helps the processing, configure the illumination to add the appropriate shadowing.

Theorem 2

If shadows in a captured image hurts the processing, configure the illumination to remove shadowing.

Figures courtesy of Illumination Technologies, Elbridge, NY


Illumination Review

Imaging Geometry

• On-axis or coaxial

• Partial bright field or directional • Back lighting

• Diffuse, dome, or cloudy day

• Dark field

• Structured

Figures from - A Practical Guide to

Machine Vision Lighting, Daryl Martin,

Advanced Illumination, Inc., 2007


Illumination Review

• Ultraviolet Spectrum ~ 100 nm – 400 nm

• Visible Spectrum ~ 390 nm – 700 nm

• Infrared Spectrum ~ 750 nm – 100 μm


Cameras and Sensors


Sensors

• CCD (Charge-Coupled Device)

o Bell Labs 1970

o Photoactive region (capacitor array)

— Each capacitor accumulates an electric charge proportional to the

light intensity at that location.

o Transmission region (shift register)

— A control circuit causes each capacitor to transfer its contents to its

neighbor (operating as a shift register)

— The last capacitor in the array dumps its charge into a charge

amplifier, which converts the charge into a voltage that is read out


Sensors

• CMOS (Complimentary Metal Oxide Semiconductor)

o First suggested in 1968-1969

o Commercialized in late 1980’s – early 1990’s

o Machine Vision-quality sensors now available from Sony, CMOSIS,

and ON Semiconductor

o Active Pixel Sensor (APS) – each pixel includes its own amplifier

o Until recently, lower image quality than CCD

o Requires less power than CCD

o Lower cost than CCD

o Less blooming than CCD

o Potential “rolling shutter” effect

o Less quantum efficiency than CCD

o Traditionally used for less demanding applications such as cell

phones and photography cameras (measurement precision not

required)


Sensors

• Bolometer

o Measures the power of incident

electromagnetic radiation via the heating

of a material with a temperature-

dependent electrical resistance

o Predominantly used in the infrared

spectrum

— Long-Wave Infrared (LWIR)

o 8 to 14 µm

o Thermal imaging

— Mid-Wave Infrared (MWIR)

o 3 to 5 μm

o Long distance tracking through the

atmosphere

— Short-Wave Infrared (SWIR)

o 0.9 to 1.7 μm

o Low light level imaging (night vision, fog,..)

Long Distance Tracking (MWIR)

Low Light Level Imaging (SWIR)

LWIR Bolometer CoreCourtesy of Xenics, NV, Leuven, Belgium


Camera Specifications

• Sensoro Sensor Size

o Shutter Control

o Sensor Resolution (Pixel Count)

o Pixel Size

o Frame Rate

o Quantum Efficiency

o Signal-to-Noise Ratio

• Trigger Capabilityo Synchronous

o Asynchronous

o Exposure Control

• Camera Typeo Area Scan

o Line Scan

o Color / Black & White

• Interface

Area Scan and Line ScanCourtesy of Edmund Optics

Asynchronous Exposure Control


Sensors – Shutter Control

• Global Shutter

o All pixels are exposed simultaneously

o Film cameras use a mechanical shutter

o Available with CCD

o Newly available with CMOS

• Rolling Shutter

o Sequential exposure start and stop for

each row of pixels

o Can create motion artifacts

o Artifact of legacy CMOS sensors

Global Shutter(Courtesy Basler AG)

Roling Shutter(Courtesy Basler AG)

Motion Artifact Due

to Rolling Shutter(courtesy Wikipedia)

Flash



• Interface

o Camera Link

— Specialized Interface

— 100 – 800 MB/S

— Up to 300 m Cable

o USB Vision

— USB-3.0

— Up to 350 MB/S


o GigE Vision

— Gigabit Ethernet

— Up to 100 MB/S


o CoaXPress

— Coax Cable

— Up to 6.25 Gb/S


o Analog (Legacy)

— RS-170 (B&W)

— NTSC, CCIR, SECAM (Color)

o Others

— FireWire ™ or IEEE-1394

— USB-2.0



• B&W

o Gray Levels (bits)

• Color

o 3 CCD

o Bayer Filter

o Color Levels (bits)

• Frame Rate

o Total frame rate

o Windowed frame rate

• Lens Mount

o C-Mount (16 mm)

o F-Mount (Nikon 35 mm)

o K-Mount (Pentax 35 mm)

o Large Format - M37 X 0.75, …

Bayer Filter



Courtesy of Point Grey Research, Inc., Richmond, Canada

Sony

CMOS

Sensors

UV Blue Green Red IR

CMOS

Sensors

Quantum Efficiency (QE) Plot



Courtesy The Optical Society of America, Washington, DC

Quantum Efficiency (QE) of Infrared Detection Sensor Materials



Film Frame Sizes

Frame Size Diagonal (mm) Width (mm) Height (mm) Aspect Ratio

35 mm (slide) 43.27 36.0 24.0 1.5 : 1

35 mm (movie) 27.2 22.0 16.0 1.375 : 1

16 mm 14.54 12.52 7.41 1.69 : 1

8 mm 5.94 4.8 3.5 1.37 : 1

Super 8 mm 7.04 5.79 4.01 1.44 : 1

IMAX 87.91 70.4 52.63 1.34 : 1

Video Frame Sizes

Frame Size Diagonal (mm) Width (mm) Height (mm) Aspect Ratio

4/3 “ 21.6 17.3 13 1.25 : 1

1 “ 16.0 12.8 9.6 4 : 3

1/1.2” 13.33 10.76 8.0 1.35 : 1

2/3 “ 11.0 8.8 6.6 4 : 3

1/2” 8.0 6.4 4.8 4 : 3

1/3” 6.0 4.8 3.6 4 : 3

1/4” 4.0 3.2 2.4 4 : 3

1/1.8” 8.93 7.18 5.32 1.35 : 1

1/2.5” 7.18 5.76 4.29 1.34 : 1

High Definition (HD) 16 : 9

Shaded areas indicate standard broadcast aspect ratios

Cinema aspect ratios: 1.85:1 - normal widescreen and 2.39:1 - anamorphic widescreen (rectangular pixels)

Television aspect ratios: 4:3 - standard definition and 16:9 - high definition



Relationship between bit depth and signal-to-noise ratio

• Bit-deptho Defines the number of discrete values of gray or of a color

o Defines the contrast resolving power

o Examples:

— Bit depth = 8-bits 256 shades of gray or color (RGB)

— Bit depth = 12-bits 4096 shades of gray or color (RGB)

• Signal-to-Noise Ratio (SNR)

o Primarily due to electronic noise

o Can include quantum noise in photon-limited systems

• Relationship

o Pixel bit depth resolution must be greater than the SNR

o Example 1: SNR = 40 dB = 1 : 10,000 > 4,096 > 256

o Example 2: SNR = 30 dB = 1 : 1,000 > 256 < 4,096


Smart Cameras

• Internal processing capabilityo CPU

o FPGA

• Graphical development software

• Uses

o Bar code reading

o Optical character recognition

o Simple detection and measure-

ment applications

National Instruments LabView™Matrox Design Assistant and IRIS Camera

Dalsa Sherlock and BOA Camera

Cognex EasyBuilder and InSight Camera


Other Sensors

Vacuum Tube - Vidicon, Plumbicon, Orthicon, Saticon, Newvicon

"Orthicon" by Tecchese - Own work. Licensed under CC BY-SA 3.0 via Commons -

https://commons.wikimedia.org/wiki/File:Orthicon.svg#/media/File:Orthicon.svg

Solid State CID (Charge Injection Device)• Every pixel in a CID array can be individually

addressed via electrical indexing of row and column electrodes

• Each pixel is read non-destructively

• Asynchronous trigger and clear

• Long integration times possible


Other Sensors

• X-Ray

• Ultra-Sound

• Electrical Impedance

• Seismic (sound waves)

• LIDAR (Light RADAR)o Time-of-Flight

o Interferometry

Courtesy of Google, Inc., Mountain View, CA


Optics

Snell's Law

(aka Snell–Descartes Law, Law of Refraction)

Defines the relationship between the angles of

incidence and refraction, when referring to light or

other waves passing through a boundary

(interface) between two different isotropic media

(uniform in all orientations), such as water, glass, or

air.

��

��

��

�

= �

�

= ��

��

where:� �angle measured from the normal of the boundary

�velocity of light in the respective medium

�wavelength of light in the respective medium

� �refractive index of the respective medium


Lens Specifications

• Lens Type

o Finite / Finite Conjugates

o Infinite / Infinite Conjugates

o Infinite / Finite Conjugates

• Size (Diameter)

• Field-of-View (FOV)

• Depth-of-Field (DOF)

• Working Distance (WD)

• Magnification

• Numerical Aperture (NA) or

f/Stop

• Chromatic Optical Aberrations

• Resolution - Modulation

Transfer Function (MTF)

• Contrast

Courtesy of Edmund Optics, Barrington, NJ


Finite / Finite Conjugate Lens Type

Single Element

F = ∗�

��M =

�

= �

��= ��

�

where:

F = effective focal length of lens system

O = object-to-lens distance

I = image-to-lens distance (image on sensor)

M = magnification

HI = half height of image

HO = half height of object

�

��

�

+

�

�



F = � ∗��

� ��M =

�

=

��

�

= ��

�

where:

F = effective focal length of lens system d = distance between elements

FI = focal length of lens closest to image M = magnification

FO = focal length of lens closest to object HI = half height of image

O = object-to-lens distance HO = half height of object

I = image-to-lens distance (image on sensor)

Two Elements



Single Element Revisited

F = ��∗��

��M =

��

��= �

��= ��

�

where:

F = effective Focal Length of lens system

WD = Working Distance (WD) or object-to-lens distance

BF = Back Focus or image-to-lens distance

M = Magnification

HS = half of Sensor Height or half height of image

HO = half of Field-of-View or half height of object

�

��

�

��+

�

��


Chromatic Optical Aberrations

Longitudinal chromatic

aberration (LCA) occurs when

different wavelengths focus at

different points along the

horizontal optical axis since the

refractive index of a glass is

wavelength dependent.

Transverse chromatic aberration

(TCA) occurs white light is used,

and the red, yellow, and blue

wavelengths focus at separate

points in a vertical plane.

where:

C = red (656.3 nm)

d = yellow light (587.6 nm)

F = blue light (486.1 nm)Mitigated using lens coatings.


Depth-of-Field

Depth-of-Field is the

increment surrounding the

working distance in which

the object is in focus.

• Dependent upon the lens

aperture (f/stop)

• Dependent upon the

sensor pixel size which

defines the Circle of

Confusion


Depth-of-Field

Increased Aperture and Constant Circle of Confusion DOF Decreases


Depth-of-Field

Numerical Aperture

NA = � sin�

Image-Space Numerical Aperture

(lens is focused to infinity)

NA� = � sin�= � sin �� !

"#~�

!

"#

f-Number

fN = #

!=

$

"%&'

where:

D = lens aperture (entrance pupil) diameter

� = index of refraction (1.0 in air)

� = half-angle of the maximum cone of light that can enter or exit the lens

NA = numerical aperture

NA� = image-space numerical aperture (lens focused to infinity)

F = lens focal length

fN = f-number of aperture


Infinite / Infinite Conjugate Lens Type

M ��

�

��

�

�(

(�

d � � + ��

where:

M = magnification

FO = focal length of object-side lens

FI = focal length of image-side lens



d = distance between lens

elements

D = lens aperture diameter

Magnification not dependent

upon back focus or

working distance


Infinite / Finite Conjugate Lens Type

M ��

�

��)*��

��+

where:

M = magnification



F = lens focal length

I = lens-to-image distance

(back focus)

� = image-side angle,

dependent upon focal

length

D = lens aperture diameter

Magnification not dependent

upon working distance


Telecentric Lens

• Single-Sided Telecentric Lenso Infinite/Finite conjugate lens

o Magnification dependent on focal length

o Magnification dependent on back-focus

• Double-Sided Telecentric Lenso Infinite/Infinite conjugate lens

o Magnification dependent on focal length only

• Useso Camera lens

o Back-light lens



Telecentric Lens

Conventional Lens / Diffuse Backlight Telecentric Lens / Telecentric Backlight


Experiment ConfigurationConventional Lens Image

Telecentric Lens Image


Resolving Power

Resolution

Number of line-pairs per unit distance

that are resolved by an imaging system.

Contrast

Relative gray-levels of black and white

objects produced by an imaging

system.

Modulation Transfer Function

(MTF)

Plot of perceived contrasts between

black and white objects verses line-

pairs per unit distance.

Limiting Resolution

Function of field-of-view, magnification,

and sensor size (pixel count).

Line-Pair Images

Resolution Depiction



Resolving PowerObject Image

Imaging Lens

Imaging Lens

Object Image

White White

BlackBlack

100%

Contrast20%

Contrast



Modulation Transfer Function (MTF)



Distortion – Spherical Aberation

%��),-)�,� �.��/�

/�0�11

where:

AD = actual distance of imaged points from center of field

PD = predicted distance of imaged points from center of field with

no distortion

Black Circles – actual locations imaged points

with distortion present

Red Circles – predicted locations of imaged

points without distortion

Light incident near the edges of the

lens come to focus too early.

Focus Range


Test Targets

EIA Gray Level Chart

IEEE Resolution Chart Depth-of-Field Test Target

USAF 1951 Resolution Chart


Test Targets

TV Color Bar PatternTV Test Pattern

Projected Distortion

Test Pattern

Fixed Frequency

Distortion Chart


Infrared Optics (Lens) Materials%

tra

nsm

itta

nce

SWIR – 0.9 to 1.7 µm

MWIR – 3 to 5 µm

LWIR – 8 to 14 µm

wavelength (μm) From R.E. Fisher, et al., Optical System Design, 2nd Ed.

Courtesy McGraw Hill, New York, NY


PART III: CNY Engineering Expo 2016

• Machine Vision Processors

• Machine Vision Software

• Machine Vision Systems


SUMMARY

• Machine Vision is defined as the technology and methods used to automatically inspect materials, components, and manufactured systems using image-based sensors and systems.

• Machine Vision is used for Quality Control, Process Control, and Robotic Guidance.

• Machine Vision Systems include the following components:

− Illuminators

− Cameras and Sensors

− Optics (Lenses)

− Synchronization (Motion Control)

− Processors

• Machine Vision Software is able to extract, highlight, and manipulate information from a captured image, but it cannot add information. In other words, if the software can’t see it, it can’t process it.

• Machine Vision Systems Engineering is required for success.


QUESTIONS?

?


REVIEW QUESTIONS and ANSWERS

1. What is Machine Vision?Machine Vision is the technology and methods used to automatically inspect

materials, components, and manufactured systems using image-based sensors and

systems.

2. What are the uses of Machine Vision?Machine Vision is used for Quality Control, Process Control, and Robotic Guidance.

3. What are the components that compose a Machine Vision System?A Machine Vision System includes illuminators, sensors (cameras), optics (lenses),

synchronizers (motion controllers and/or encoders), and processors.

4. What types of sensors are used in Machine Vision cameras?CCD, CMOS, and bolometers.

5. What are some of the parameters used to specify lenses?Size, focal length, resolving power (MTF and contrast).

6. How excited are you to attend Part III of this series?EXTREMELY!

7. How many PDHs did you earn by attending this session?


Thank You For Attending

Automated Inspection

With

Machine VisionPart 2

Stanley N. Hack, D.Sc., PE

www.consultechusa.com

November 9, 2015

automated inspection with machine vision · a large component of machine vision system design is...

Documents