anatomy skills image processing fundamentals - … 1 aaron ponti anatomy skills image processing...

35
2/24/2012 1 Aaron Ponti Anatomy Skills Image processing fundamentals Image processing and analysis circle Image processing fundamentals -- Definitions Definitions Digital image Image resolution Grayscale resolution Spatial resolution Image data types Binary (1 bit) Intensity (8 bit, gray) Indexed (8 bit, color) RGB (24 bit, color) Color spaces

Upload: lamkhanh

Post on 09-Apr-2018

266 views

Category:

Documents


2 download

TRANSCRIPT

2/24/2012

1

Aaron Ponti

Anatomy SkillsImage processing fundamentals

Image processing and analysis circle

Image processing fundamentals-- Definitions

Definitions• Digital image

• Image resolution• Grayscale resolution• Spatial resolution

• Image data types• Binary (1 bit)• Intensity (8 bit, gray)• Indexed (8 bit, color)• RGB (24 bit, color)

• Color spaces

2/24/2012

2

Definitions• Digital image

• Image resolution• Grayscale resolution• Spatial resolution

• Image data types• Binary (1 bit)• Intensity (8 bit, gray)• Indexed (8 bit, color)• RGB (24 bit, color)

• Color spaces

Digital image• A digital image is a discrete function defined over a rectangular grid

(lattice) representing the characteristics of the objects being imaged.

Sampling (digitization): example in 1D.

sample

continuous (analog) signal

sampling interval

Digital image Digital image

• In 2D images, each grid element, or pixel(picture element), is defined as a location and a value representing the characteristic of the object in that location.

• In 3D images, the pixel is called voxel(volume element).

• A 3D image is just a stack of 2D images.

2/24/2012

3

Definitions• Digital image

• Image resolution• Grayscale resolution• Spatial resolution

• Image data types• Binary (1 bit)• Intensity (8 bit, gray)• Indexed (8 bit, color)• RGB (24 bit, color)

• Color spaces

Image resolution The resolution of an image is a measure of the fidelity of the representation

of the original scene.

Resolution is related firstly to the characteristics of the imaging system and secondly to the number of pixels (i.e. spatial resolution) and the range of brightness values (i.e. grayscale resolution) that are used for digitization.

The grayscale resolution of an image is expressed as its bit depth.

The maximum number of brightness (i.e. gray) levels in an n-bit image is 2n.

Image resolution ::Grayscale resolution

Image resolution ::Grayscale resolution

2/24/2012

4

Image resolution ::Spatial resolution Image resolution summary

Definitions• Digital image

• Image resolution• Grayscale resolution• Spatial resolution

• Image data types• Binary (1 bit)• Intensity (8 bit, gray)• Indexed (8 bit, color)• RGB (24 bit, color)

• Color spaces

Image data types ::Binary

2 “gray values”: black (0) and white (1)

Binary values (0 or 1)

1 1 1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1 1 1 11 1 1 0 0 0 0 1 1 1 11 1 1 1 0 0 0 1 1 1 11 1 1 1 1 1 0 1 1 1 11 1 1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1 1 1 1

2/24/2012

5

178 181 182 182 183 183 186 181 185 185 186 185 186 …178 182 182 183 184 183 183 182 182 182 188 184 186 …181 182 184 185 184 181 183 185 187 184 183 189 186 …186 186 184 187 185 183 189 185 184 186 190 187 186 …185 182 183 189 185 187 187 186 187 187 188 190 185 …184 183 185 185 187 185 189 186 189 189 189 193 193 …183 184 187 182 184 187 189 185 190 188 193 199 198 …183 182 181 186 190 189 195 191 187 162 156 116 30 …185 184 191 192 209 195 198 168 102 49 22 15 11 …187 187 187 194 197 173 77 25 19 19 26 17 11 …190 187 190 191 158 37 15 14 20 20 15 12 11 …187 182 180 127 32 16 13 16 14 12 13 11 14 …184 186 172 100 20 13 15 18 13 18 14 15 10 …186 190 187 127 18 14 15 14 12 10 11 11 10 …189 192 192 148 16 15 11 10 10 9 9 9 9 …192 195 181 37 13 10 10 10 10 10 9 9 9 …189 194 54 14 11 10 10 10 9 8 9 8 9 …194 140 19 16 11 11 10 10 9 9 9 9 9 …192 88 12 11 11 10 10 10 9 9 9 9 9 …194 37 10 10 9 9 9 9 9 9 9 8 8 …… … … … … … … … … … … … … …

Image data types :: Intensity

256 gray values

Intensity (brightness)

Image data types :: Indexed

256 colors

colormap = 1.0000 1.0000 1.0000 11.0000 1.0000 0 21.0000 0 1.0000 31.0000 0 0 4

0 1.0000 1.0000 50 1.0000 0 60 0 1.0000 70 0 0 8

0.5176 0 0 90.4510 0 0 100.3882 0 0 110.5490 0.0627 0.0627 120.4824 0.0627 0.0627 130.4196 0.0627 0.0627 140.5490 0.0941 0.0941 15... ... ... ...

145 140 121 89 78 42 117 87 117 116 ...140 140 140 86 86 157 114 137 135 117 ...105 140 145 150 137 135 135 135 126 137 ...86 149 140 126 135 114 150 157 135 114 ...137 114 117 137 137 135 114 114 114 117 ...140 121 117 137 137 117 142 142 157 142 ...129 137 161 135 135 117 161 149 157 142 ...140 121 89 156 142 87 14 86 137 114 ...114 86 78 138 138 138 156 117 137 114 ...150 157 157 156 156 142 156 117 135 157 ...... ... ... ... ... ... ... ... ... ... ...

Indices

Image data types ::RGB (24 bit)

3 x 256 “gray values”: 3 x 8 bit = 24 bit image

8 bit gray (→ red)

8 bit gray (→ blue)8 bit gray (→ green)

3 channels RGB, 32 bit color

Definitions• Digital image

• Image resolution• Grayscale resolution• Spatial resolution

• Image data types• Binary (1 bit)• Intensity (8 bit, gray)• Indexed (8 bit, color)• RGB (24 bit, color)

• Color spaces

2/24/2012

6

The Lab color space isdesigned to approximatehuman vision. It aspiresto perceptual uniformity,and its L componentclosely matches humanperception of lightness.

The RGB color model isan additive color model inwhich red, green, and bluelight are added together invarious ways to reproducea broad array of colors.

HSL and HSV are the twomost common cylindrical-coordinate representations ofpoints in an RGB colormodel, which rearrange thegeometry of RGB in anattempt to be moreperceptually relevant thanthe Cartesian representation.

Color spaces

Image processing fundamentals-- Grayscale image processing basics

Grayscale image processing basics• Color lookup tables (LUT)

• Image contrast

• Image histogram

• Pixel statistics• Profile statistics• Region-of-interest statistics

• Fundamental spatial image processing tools• Neighbors and connections• (Matrix) transformations: scaling, rotation, other transforms• Interpolation

Grayscale image processing basics• Color lookup tables (LUT)

• Image contrast

• Image histogram

• Pixel statistics• Profile statistics• Region-of-interest statistics

• Fundamental spatial image processing tools• Neighbors and connections• (Matrix) transformations: scaling, rotation, other transforms• Interpolation

2/24/2012

7

Color lookup tables (LUT)• In biomedical imaging it is common to apply false colors while retaining

the original 8 bit scale.

Colored SEM image of soybean cyst nematode and egg. The color makes the image easier for non-specialists to view and understand the structures and surfaces revealed in micrographs.

• Color can also be used to convey a visual meaning to the values or measurements in the image.

Color lookup tables (LUT)

Heat map generated from DNA microarray data reflecting gene expression values in several conditions

Differences in blood and oxygen levels in the brain shown by fMRI maps.

Grayscale image processing basics• Color lookup tables (LUT)

• Image contrast

• Image histogram

• Pixel statistics• Profile statistics• Region-of-interest statistics

• Fundamental spatial image processing tools• Neighbors and connections• (Matrix) transformations: scaling, rotation, other transforms• Interpolation

Image contrast• Contrast is a measure of brightness difference, both globally (for the whole

image) and locally (in neighboring regions). One possible way to express it is:

C = ( IS – IB ) / IB

2/24/2012

8

Grayscale image processing basics• Color lookup tables (LUT)

• Image contrast

• Image histogram

• Pixel statistics• Profile statistics• Region-of-interest statistics

• Fundamental spatial image processing tools• Neighbors and connections• (Matrix) transformations: scaling, rotation, other transforms• Interpolation

Image histogram• The grayscale image histogram is a way of illustrating the distribution of

gray levels in an image: it shows how many pixels have particular gray values (or gray value ranges: “histogram bins”).

• The histogram is a very important tool for performing image enhancement.

Dynamic range ( 0 – 255 for 8 bit)

Image histogram• Color (RGB) images have three independent channels, displayed in Red,

Green and Blue, respectively.• Each channel has its own histogram.

Grayscale image processing basics• Color lookup tables (LUT)

• Image contrast

• Image histogram

• Pixel statistics• Profile statistics• Region-of-interest statistics

• Fundamental spatial image processing tools• Neighbors and connections• (Matrix) transformations: scaling, rotation, other transforms• Interpolation

2/24/2012

9

Pixel statistics ::Profile statistics (plots)

Profile

Pixel statistics :: ROI statistics

Region of interest (ROI)

SquareCircularFree-formManually drawn(Semi-)automatically extracted…

Grayscale image processing basics• Color lookup tables (LUT)

• Image contrast

• Image histogram

• Pixel statistics• Profile statistics• Region-of-interest statistics

• Fundamental spatial image processing tools• Neighbors and connections• (Matrix) transformations: scaling, rotation, other transforms• Interpolation

Common pixel neighborhoods

4-connected(2D)

6-connected(3D)

8-connected(2D)

18-connected(3D)

6-connected(2D)

26-connected(3D)

2/24/2012

10

Matrix transformations

scaling

rotationfree transform

Interpolation Interpolation is a method of constructing new data points within the

range of a discrete set of known data points.

Used e.g. when scaling or generally transforming an image. There are several interpolation methods: linear, polynomial, spline,…

92 72 54

92 72 5484 63Simplest linear interpolation.

Image processing fundamentals-- The five classes of image processing

The five classes of image processing• Image enhancement

• Image histogram operations

• Spatial domain filtering

• Frequency domain filtering

• Image restoration• Deconvolution

• Correction of geometrical distortions

• Correction of gray-level inhomogeneities

• Image analysis• Segmentation

• Classification

• Image compression

• Image synthesis• Registration

• Visualization

• 3D rendering

2/24/2012

11

The five classes of image processing-- Image enhancement

Image enhancement• Used to improve some aspects of the quality of an image:

• Increase contrast or and/or brightness

• Sharpen details

• Remove noise

• Image enhancement operations

• Contrast enhancement

• Spatial filtering

• Frequency filtering

• Often performed interactively

• Often performed as preprocessing step in an automated image analysis operation

• Result is often subjective, but quantitative measures of image quality do exist (i.e. contrast to noise ratio)

Image histogram operations :: Contrast enhancement (gray)

Low contrast image

Histogram

Datatype (8 bit) dynamic range: 0 – 255

Image histogram operations ::Contrast enhancement (gray)

Actual dynamic rangeLow contrast image

Histogram

Datatype (8 bit) dynamic range: 0 – 255

2/24/2012

12

Image histogram operations ::Contrast enhancement (gray)

Actual dynamic rangeLow contrast image

Histogram

Datatype (8 bit) dynamic range: 0 – 255

Histogram stretching (linear mapping)

Image histogram operations ::Contrast enhancement (gray)

Actual dynamic rangeLow contrast image

Histogram

Datatype (8 bit) dynamic range: 0 – 255

Histogram stretching (linear mapping)

Adaptive histogram equalization

(local)

Histogram stretching

(global)

Image histogram operations :: Contrast enhancement (gray)

Histogram equalization

(global)

Original

Image histogram operations ::Contrast enhancement (color)• Contrast enhancement of color images is typically done by transforming an

image to a color space that has image intensity as one of its components (e.g. Lab) and then work on the luminosity layer ‘L’ of the image.

• Manipulating luminosity affects the intensity of the pixels, while preserving the original colors.

2/24/2012

13

Image histogram operations ::Contrast enhancement (color)

Original Histogram stretching

Histogram equalization Adaptivehistogram equalization

(global)

(global) (local)

Spatial filtering operations ::Rank filters (nonlinear filters)• Spatial filters are operations applied to a pixel using information taken from

the neighborhood of that pixel.

• In rank (or ordered) filtering, the gray values of pixels within a defined neighborhood around the pixel of interest are arranged in a list in ascending order. The new value of for the pixel of interest is the value at the required rank position in the list.

• Common rank filters:• maximum ( rank = n )• minimum ( rank = 1 )• median ( rank = n/2 )• range ( maximum – minimum )

• The outcome of the filtering depends on the size and shape of the neighborhood.

Rank filters ::Median filter

Original image Added salt-and-pepper noise Median filtered image

Rank filters ::Median filter

Original image Added salt-and-pepper noise Median filtered image

The rank filter causes some (limited) blurriness in the image.

2/24/2012

14

Spatial filtering operations ::Convolution filters (linear filters)• Like rank filtering, convolution filter also replaces a pixel value with a new

value obtained from the pixel neighborhood. But there are a few more calculations involved.

• A square kernel is defined to represent the neighborhood around the pixel. Each location in the kernel is associated a numerical value, called weight.

a b c

d e f

g h i

• The kernel is moved across the image, and the pixel value under the center of the kernel is replaced by the weighted sum of the surrounding pixels.

Kernel size can vary a lot,depending on the application.This is a 3x3 kernel.

Spatial filtering operations

Convolution filters

a b c

d e f

g h i

A B C

F G H

K L M

D E

I J

N O

P Q R

U V W

S T

X Y

A B C

F cG bH

K fL eM

D EaI JdN O

P iQ hR

U V W

gS T

X Y

A B C

F aG bH

K dL eM

D EcI JfN O

P gQ hR

U V W

iS T

X Y

kernel image

convolution

correlation

Mirrored in case of convolution!

Used to find similarities between two images.

Spatial filtering operations• The convolution kernel is mirrored.

• For most applications, the kernel is symmetric, and mirroring does not change it. Many (lazy) convolution implementations indeed use correlation.

Convolution: M’ = iG + hH + gI + fL + eM + dN + cQ + bR + aS

Correlation: M’ = aG + bH + cI + dL + eM + fN + gQ + hR + iS

Spatial filtering operations

919191919191919191

h

3x3 average filter

0.07510.12380.07510.12380.20420.12380.07510.12380.0751

h

3x3 Gaussian filter

111181111

h

High-pass filter

Examples of convolution filters

2/24/2012

15

Spatial filtering operations

121/1...121/1121/1121/1...............121/1...121/1121/1121/1121/1...121/1121/1121/1121/1...121/1121/1121/1

h

11x11 average filter

101-101-101-

h

PrewittVertical edge

111000111

h

PrewittHorizontal edge

Examples of convolution filters

• Given the same kernel size, convolution filters introduce more blurring than rank filters.

Spatial filtering operations

Low-pass filterGaussian kernel

Band-pass filterDifference of Gaussians kernel

High-pass filter1 - Gaussian kernel

• Hybrid filters may include both rank and convolution steps, or involve other image processing operations.

• Adaptive filters perform a different operation depending on the image content in the region in which they are being applied.• For instance, an adaptive filter could be designed to blur only those regions of an

image that do not contain an edge (e.g. anisotropic filtering).

Spatial filtering operations :: Hybrid filters and adaptive filters

Frequency filtering has an effect equivalent to convolution filtering, but the approach is different.

Shapes in images are made up of changes in gray levels across the image, from dark to light and back to dark. The rate of this change is called spatial frequency.

A real image will be much more complex and contain many more frequencies. However, we can say that: high spatial frequencies correspond to fine detail, such as noise and edges

low spatial frequencies correspond to larger objects with fairly uniform gray values.

Frequency domain filtering ::Spatial and frequency domains

Increasing spatial frequency

2/24/2012

16

• A high-pass filter preserves high frequencies.

• A low-pass filter preserves low frequencies.

• A band-pass (or notch) filter preserves a particular band, or range, or frequencies.

Frequency domain filtering ::Types of frequency filters

Frequency domain filtering ::Approach to frequency filtering• Generate a representation to show which spatial frequencies are present in

an image:• i.e. generate a frequency spectrum using Fourier transformation.

• Remove selected frequencies from this representation.

• Reverse the process to get back to the image (spatial representation):• i.e. apply the inverse Fourier transformation.

Frequency domain filtering ::Advantages of frequency domain filtering• It is possible to perform operations that are difficult in the spatial domain,

such as removing or enhancing only specific frequencies in the image.

• Periodic patterns can be selectively removed or enhanced.

• For operations that require larger kernel sizes in the spatial domain (especially in 3D), frequency domain filtering is computationally faster.

Frequency domain filtering

Increasing frequency

Increasingfrequency

2/24/2012

17

Frequency domain filtering

Fourier transform

Frequency domain filtering ::Low-pass filtering

InverseFourier transform

Set high frequencies to 0.

RemarkIn practice, one does not create sharp cut-offs in frequency domain, since this creates ringing artifacts that appear as spurious signals near sharp transitions in a signal, i.e. they appear as "rings" near edges.

Frequency domain filtering ::High-pass filtering

InverseFourier transform

Set low frequencies to 0.

Frequency domain filtering :: Band-pass filtering

InverseFourier transform

Preserve frequencies only in this band.

2/24/2012

18

Frequency domain filtering

Fourier transform ?

Frequency domain filtering

Frequency domain filteringFrequency domain filtering ::Convolution theorem

Convolution in direct space=

multiplication in frequency (Fourier) space

Inverse filtering

Fourier transform

Inverse Fourier transform

2/24/2012

19

The five classes of image processing-- Image analysis

Image analysis• The goal of image analysis is to extract and assign identities to objects in

images, and then extract some form of (numerical) measurement from them.

• Operations:

• Image segmentation

• Object classification

• Image arithmetic

• Binary operations

• Examples of measurements:

• Length, area, volume, density,…

• Usually statistical analyses are then performed on the obtained measurements

Image analysis ::Segmentation• Image segmentation refers to the process of partitioning a digital image

into multiple sets of pixels with the goal of simplifying and/or changing the representation of an image into something that is more meaningful and easier to analyze.

• Image segmentation is typically used to locate objects and boundaries (lines, curves, etc.) in images.

• More precisely, image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain visual characteristics.

Image analysis ::Classification• Classification assigns objects obtained from segmentation into categories

based on quantitative information on one or more characteristics inherent in the items (referred to as traits, variables, characters, etc).

• There are many classification algorithms divided in several categories:• Linear classifiers (Fisher's linear discriminant, Logistic regression, Naive Bayes classifier, Perceptron,

Support vector machines)

• Quadratic classifiers• k-nearest neighbor• Boosting• Decision trees (Random forests)

• Neural networks• Bayesian networks• Hidden Markov models

• The separation between segmentation and classification is quite blurred in all but the simplest segmentation approaches!

2/24/2012

20

Segmentation• There are three approaches to segmentation:

• Manual

• Semiautomatic

• Automatic

Manual segmentation• Advantages:

• Simple, provided there are suitable tools

• Disadvantages:

• Very time-consuming

• Subject to human error

• Subjective

• Poor (intra-observer) reproducibility

Manual segmentation• Manual segmentation involves an expert observer outlining the object of

interest in the image.

• The computer simply provides tools to help with drawing.

Manual segmentation ::Simple thresholding• The user chooses an intensity threshold level in the histogram.

• The software shows a preview to facilitate the selection of the threshold.

2/24/2012

21

Manual segmentation :: Simple thresholding• After the threshold is set, all values below the threshold are set to 0.

• The values above the threshold are either left unchanged, or set to white (e.g. 255).

Semi-automatic segmentation• In semi-automatic segmentation, the observer helps the segmentation by

providing the software with a rough selection to refine.

• Region growing

• Active contours, snakes, and deformable surfaces

Semi-automatic segmentation ::Region growing• The operator defines a range of gray levels that represent the region, and

chooses a pixel (the seed point) that is known to be part of the region.

• All the pixels that are in the intensity range and are connected to the seed point are selected as being part of the region.

• Alternatively, the software returns the boundary of the region as an editable line.

Semi-automatic segmentation ::Region growing

Range = 10 gray values

2/24/2012

22

Semi-automatic segmentation ::Active contours, deformable surfaces• Active contour model, also called snakes ( = deformable splines), is a

framework for delineating an object outline from a possibly noisy 2D image.

• This framework attempts to minimize an energy associated to the current contour as a sum of an internal and external energy:

• The external energy is supposed to be minimal when the snake is at the object boundary position (e.g. low energy values for high gradient values);

• The internal energy is supposed to be minimal when the snake has a shape which is supposed to be relevant considering the shape of the sought object (e.g. high energy to elongated contours (elastic force) and to bended/high curvature contours (rigid force), considering the shape should be as regular and smooth as possible).

• The framework also allows for training.

Semi-automatic segmentation ::Active contours / snakes

Range = 10 gray values

Semi-automatic segmentation ::Deformable surfaces

Range = 10 gray values

Automatic segmentation • Automatic (computational) segmentation methods are areas of

active research.

• We will only look into a very simple automatic segmentation algorithm: Otsu’s method.

2/24/2012

23

Automatic segmentation ::Otsu’s method• Otsu's method is used to automatically perform histogram shape-based

image thresholding.

• The algorithm assumes that the image to be thresholded contains two classes of pixels (e.g. foreground and background) then calculates the optimum threshold separating those two classes so that their combined spread (intra-class variance) is minimal.

Range = 10 gray values

threshold

Ideal case

Automatic segmentation ::Otsu’s method

Range = 10 gray values

This image probably contains more than two classes of pixels.

There are extensions of Otsu’s algorithm that find multiple thresholds.

Image arithmetic Image arithmetic is the implementation of standard arithmetic operations,

such as addition, subtraction, multiplication, and division, on images.

Image arithmetic has many uses in image processing both as a preliminary step in more complex operations and by itself. Examples: Image subtraction can be used to detect differences between two or more images of the

same scene or object.

Image subtraction or division are used for shading corrections. Image multiplication is used for masking. Image addition is used for visualizing composite images (where each channel has a

color which is not exclusively either Red, Green, or Blue).

Binary and morphological operations Binary and morphological operations can be used to perform common

image processing tasks, such as contrast enhancement, noise removal, thinning, skeletonization, filling, and segmentation.

Morphology is a broad set of image processing operations that process images based on shapes.

Morphological operations apply a structuring element to an input image, creating an output image of the same size.

In a morphological operation, the value of each pixel in the output image is based on a comparison of the corresponding pixel in the input image with its neighbors.

By choosing the size and shape of the neighborhood, you can construct a morphological operation that is sensitive to specific shapes in the input image.

2/24/2012

24

Binary and morphological operations ::Example: Image dilation Dilation is one of the two basic operators in the area of morphology, the

other being erosion.

It is typically applied to binary images, but there is a version for grayscale images.

Dilation gradually enlarges the boundary of regions of foreground pixels (i.e. white pixels): thus area of foreground pixels grow in size, while holes within these regions become smaller.

The dilation operator takes two pieces of data as inputs. The first is the image which is to be dilated. The second is a (usually small) set of coordinate points known as a structuring element, for example:

1 1 1

1 1 1

1 1 1

0 1 0

1 1 1

0 1 0

Binary and morphological operations ::Example: Image dilation Mathematically, dilation is defined as:

This basically means, place the structuring element B on some position on the image A. If the pixel value of A under the center of B is 1, replace the n x n pixels in A with the structuring element B.

Example:

Binary and morphological operations

Problem: detecting (i.e. segmenting) a cell

Binary and morphological operations

A first rough segmentation is given by a threshold of gradient of the image, to give the strongest edges.

Binary gradient mask

2/24/2012

25

Binary and morphological operations

Image dilation: the binary gradient mask is dilated.

Dilated gradient mask

Binary and morphological operations

Image flood-fill: the dilated gradient mask shows the outline of the cell quite nicely, but there are still holes in the interior of the cell that we flood-fill.

Dilated gradient mask with filled holes

Binary and morphological operations

Here we got rid of the cell at the border. Admittedly, this is not strictly a morphological operation…

Cleared border image

Binary and morphological operations

Image erosion: finally, in order to make the segmented object look natural, we smoothen the object by eroding the image twice with a diamond structuring element.

Segmented image

2/24/2012

26

Binary and morphological operations

An alternate method for displaying the segmented object would be to place an outline around the segmented cell.

Segmented image with outline

The five classes of image processing-- Image restoration

Image restoration• Required if the acquisition method (e.g. the microscope) causes:

• geometric distortion

• blurring

• gray-level inhomogeneities

Geometric distortionAlthough distortion can be irregular or follow many patterns, the most commonly encountered distortions are radially symmetric, arising from the symmetry of a lens. The radial distortion can usually be classified as one of two main types: Barrel distortion and Pincushion distortion.

In barrel distortion, image magnification decreases with distance from the optical axis. The apparent effect is that of an image which has been mapped around a sphere (or barrel).

In pincushion distortion, image magnification increases with the distance from the optical axis. The visible effect is that lines that do not go through the centre of the image are bowed inwards, like a pincushion.

2/24/2012

27

Blurring• Any imaging system has a blurring effect.

• If a point object is imaged, the resulting image is not a perfect point, but a spread-out version of that point.

• Mathematically, this blurring effect is described using the convolution operation:

• Every imaging system has a characteristic blurring function called point spread function (PSF): h(x)

Point object, whose dimensions are below the resolution limit.

Point spread function, lateral view.

Axial view.

Removing blur Deconvolution• Knowledge of the PSF of a system is useful because of the potential for removing

blurring by applying the inverse of convolution, deconvolution.

• We saw that convolution can be performed in frequency space. Can we deconvolve by inverse filtering, then?

Point spread function, lateral view.

Axial view.

f

h

F

H

G g

f

F -1(G/H)?

F -1(G)

F (f)

F (h)

Deconvolution :: Cookie cutter

XZ

Missing frequencies

“Sub-resolution train“Noise-free convolution and deconvolution

Deconvolution :: Deconvolving trains

2/24/2012

28

“Sub-resolution train“Noise-free convolution and deconvolution

Confocal

Widefield

Deconvolution :: Deconvolving trains

Confocal

Widefield

Restored confocal

Restored widefield

“Sub-resolution train“Noise-free convolution and deconvolution

Deconvolution :: Deconvolving trains

Deconvolution :: MLE

Extreme noise amplification!

Inverse filtering will never allow us to recover the true object function f.

Artifacts!

H = 0 at many places!

• Future promising algorithms are based on wavelets.

Deconvolution is performed in practice with iterative algorithms like the maximum likelihood estimation (MLE) algorithm.

Correction of gray-level inhomogeneities(shading correction)

Background estimation

• “Rolling ball”• Morphological Opening

Subtraction

2/24/2012

29

Correction of gray-level inhomogeneities(shading correction)

A correction is important, for instance, for stitching applications to prevent border effects.

Correction of gray-level inhomogeneities(shading correction)

A correction is important, for instance, for stitching applications to prevent border effects.

The five classes of image processing-- Image compression

Image compression• Image data files are often very large in terms of number of bytes

• Large disk space usage

• Slow transfer over networks

• Compression algorithms exists

• Lossless

• Lossy

2/24/2012

30

Image compression :: Lossy compression

100%, 64kb

Image compression ::Lossy compression

Jpeg, 80%, 12kb

Image compression ::Lossy compression

Jpeg, 50%, 7kb

Image compression ::Lossy compression

Jpeg, 10%, 3kb

2/24/2012

31

Image compression ::Lossy compression

Jpeg, 1%, 1kb

Image compression ::Lossless compression

100%, 64kb

Image compression ::Lossless compression

PNG, best compression, 48kb

The five classes of image processing-- Image synthesis

2/24/2012

32

Image synthesis• Image synthesis is the general term for bringing together information from

more than one image.

• Synthesis can be separated into two parts:

• Image registration, covering processes required to bring images into spatial alignment

• Visualization, which allows information (from the aligned datasets) to be viewed

• Reasons for performing image synthesis include:

• Assessment of disease progression or growth using series of images

• Combination of structural and functional information from different imaging modalities

• Comparison of corresponding regions in different individuals by matching both to a standard coordinate system

• Generation and analysis of atlases or templates representing the typical appearance in health or disease

• Arithmetical or statistical operations which require registered images

Image registration• How to map one onto the other?

Image 1

Image 2

Smaller, rotated version of image 1.

Image registration• The four steps of image registration:

• Feature extraction

• One needs to identify features that appear in both images. This can be done manually, by placing markers on the images, or automatically. All pixels intensities could also be used.

• Pairing (also called: “identifying correspondences”)

• Once features have been identified in both images, they need to be analyzed to determine which feature in one image matches which feature in the other image.

• The features can be fiducial landmarks or pixel intensities (or related values).

• The term similarity metric is the general way of describing a measure that is used in the process of pairing points with similar properties. Examples: correlation coefficient, mutual information, entropy

• Calculation of transformation

• The transformation is the mathematical operation that will give the best alignment (mapping) of all pairs of features. There arefour different types of transformations:

• Rigid: in a rigid transformation all parts of the object are assumed to move as a whole, e.g. translation, rotation

• Affine: in affine transformations, straight lines remain straight and parallel lines remain parallel: e.g. scaling, shearing

• Projective: projective transformation include further deformation. Straight lines remain straight, but parallel lines do not remain parallel.

• Curved or elastic (non-rigid): this is the most general case of transformation. Straight lines need not remain straight. It should be used with care.

• Application of transformation

• This step is often called matrix transformation or matrix operation.

• Applying the transformation will very often require interpolation.

Image registrationExample: rigid transformation

1. The user chooses some fiduciary marks by clicking on them2. The positions are refined by cross-correlation3. The (rigid) transformation matrix is obtained by Least-Square fitting4. Image 1 is transformed (mapped) onto image 2 by matrix transformation and interpolation.

11

22

33

4455

12

3

4

5

111222

443

334

555

2/24/2012

33

Image registration• End result of registration

Image registration example ::Atlas generation

Mouse hippocampus

A user draws the outline of the hippocampus overmany aligned slices to reconstruct the 3-D shape.

The same operation is performed on 3 mice, and the resulting 3D objects are registered on top of each other to generate an atlas (i.e. a model of the hippocampus).

1)2)

3)

Image registration example ::Correlative microscopy

Biocytin labelled cell, Z projection of 80 confocal planes. (200nm X-Y pixel size)

Same cell imaged in SBFSEM (98x98x30 microns)

Visualization• Visualization is the display of image data.

• Visualization in biomedical imaging needs specialized approaches:

• Visualized objects can have a huge range of scales, from single molecules and cells to body parts

• Different attributes of these objects can be visualized, including biophysical, biomechanical and physiological properties.

• The result of visualization can be used for diagnosis, treatment planning, rehearsal, assessment, and intra-operative guidance.

• There are several approaches to visualization:

• Slice mode

• Projection mode

• Volume mode

• Surface mode

2/24/2012

34

Visualization ::Slice mode

Fluorescence image of the mouse retina. Red: collagen IV, Green: GFAP (source Imaris demos)

Shown: one slice (focal plane) in the middle of the 3D acquisition.

Visualization ::Slice mode

Shown: all slices in one view.

Visualization ::Maximum intensity projection

Visualization ::Volume rendering

2/24/2012

35

Visualization ::Surface rendering

Visualization ::Surface rendering (fancy)

Last remarks

Last remarks• Every image processing action taken must make sense:

• Do not apply any filter just because like this the image looks better.

• Remember that applying the wrong filter can destroy real features in the image and also introduce spurious ones that actually DO NOT EXIST.

• Keep a complete protocol of the actions taken:

• The result (i.e. the figure in a paper) must be reproducible by you and by others.

• Make sure you know the characteristics of your acquisition system (e.g. a microscope), because this will define:

• The parameters for acquiring the images.

• The parameters for processing the images.

• Image processing was born to process different kinds of pictures than biomedical ones (which are diffraction-limited, with extremely low SNR, 3D or more, …)

• There is a lot of work going on extending standard image processing techniques for these new fields.

• Image processing of microscopy data requires a lot of statistical modeling, learning algorithms, and the like (which we haven’t touched in this introductory course).