Image Image ProcessingProcessing

on GPUon GPU

Image ProcessingImage Processing• Define new images from existing image

for different purposes• Used for texture preprocessing for 3D

graphics and visualization• Simple processing:

– Transform each pixel independently• Pixel to pixel operations• Example: RGB image to grey scale image

– Move pixels inside without modifying intensities

• Complex processing– Many advanced technologies for different

applications

Noise ReductionNoise Reduction• An important application for image

processing• Important in many areas• Sources:

–Digital camera devices–Scanned documents–Satellite images–Medical imaging–…

Example Noise TypesExample Noise Types• Salt and pepper:

random white and black pixels

• Impulse: random white pixels

• Gaussian noise: variation in intensity from Gaussian distribution

• …• We need to “SMOOTH”

the image to reduce noises

Linear FiltersLinear Filters• Construct a new but same-size image

from original one• Fill new pixel with a weighted sum of

old pixels -- filtering• Using the same weights for each pixel• Linear filtering

–Output for the sum of tow images is the same as the sum of the outputs obtained for the images separately

ConvolutionConvolution• Filter kernel: the pattern of weights• Convolution: the process of applying

the filter on an image• Mathematically:

SmoothingSmoothing• Reduce noise• Replace each pixel with a weighted

average of its neighbors• Local blur effects to remove noise• Low frequency filter to remove high

frequency noise

Averaging SmoothAveraging Smooth• (2k+1)*(2k+1) region

• Poor blurring• Too simple

∑ ∑+

−=

+

−=+=

ki

kiu

kj

kjvuvij F

kR 2)12(

1

Gaussian SmoothingGaussian Smoothing• Symmetric Gaussian kernel

– Weights distributed with Gaussian function– Sigma: standard deviation

– In applications, discrete kernel used:

)2

)(exp(2

1),( 2

22

2 σπσσyxyxG +

−=

)2

))1()1((exp(2

12

22

2 σπσ−−+−−

−=kjkiHij

Gaussian Lookup TableGaussian Lookup Table–A lookup table can be pre-generated to

accelerate the computation, such as

Gaussian KernelGaussian Kernel• If standard deviation is very small

(smaller than one pixel), the smoothing have very little effect

• For a larger one, the average will be strongly biased toward a consensus of the neighbors. Noise will largely disappear, at the cost of some blurring

• Finally, if it is very large, much of detail disappear along with noise

Why use GaussianWhy use Gaussian• Not the only smoothing filter• Convenient to use:

– Efficiency– Separable

– Performed separately for x, y, z dimensions

22 2121 σσσσ +=∗ GGG

Boundary EffectsBoundary Effects• In practical system, image has

boundaries• When computing convolution, need

to handle boundaries where the neighbors do not exist– Ignore these locations-image may

shrink–Pad the image with constant values–Pad with other methods: periodical …

ExampleExample

• Robert J. Woodham, UBC

Edge DetectionEdge Detection• Edges are important information on

images–Object detection …–Large gradient– Intensity derivatives

• Edge detection: estimate derivative and an image represented by a discrete set of pixels

What is an edgeWhat is an edge

• Easy for human• Difficult for computer

Estimate DerivativeEstimate Derivative• Finite difference

• The similar convolution kernel is

• Gradient vector

jiji hhxh

,1,1 −+ −≈∂∂

⎪⎭

⎪⎬

⎫

⎪⎩

⎪⎨

⎧−=000101

000G

⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

∂∂

=∇yf

xfyxf ,),(

SobelSobel KernelKernel• A popular gradient magnitude

computation

Second DerivativesSecond Derivatives• Sobel operator can produce thick edges• Ideally, we want thin boundaries• Alternative approach:

– Looking for local extrema in first derivative– Where the change in gradient is highest

2D 2D LaplacianLaplacian• An equivalent measure of second

derivative

Edge detection cg codeEdge detection cg code• Laplacian edge detection

static const char *edgeFragSource = {"uniform sampler2D texUnit;"

"void main(void)""{"" const float offset = 1.0 / 512.0;"" vec2 texCoord = gl_TexCoord[0].xy;"" vec4 c = texture2D(texUnit, texCoord);"" vec4 bl = texture2D(texUnit, texCoord + vec2(-offset, -offset));"" vec4 l = texture2D(texUnit, texCoord + vec2(-offset, 0.0));"" vec4 tl = texture2D(texUnit, texCoord + vec2(-offset, offset));"" vec4 t = texture2D(texUnit, texCoord + vec2( 0.0, offset));"" vec4 ur = texture2D(texUnit, texCoord + vec2( offset, offset));"" vec4 r = texture2D(texUnit, texCoord + vec2( offset, 0.0));"" vec4 br = texture2D(texUnit, texCoord + vec2( offset, offset));"" vec4 b = texture2D(texUnit, texCoord + vec2( 0.0, -offset));"" gl_FragColor = 8.0 * (c + -0.125 * (bl + l + tl + t + ur + r + br + b));""}"

};

Results not satisfiedResults not satisfied• Noise are emphasized

How to improve?How to improve?• Add smoothing filter• Smoothing a differentiated image

tends to get good result we want• Smooth the edge detected due to

noises

Improved ResultsImproved Results

Derivative of Gaussian FilterDerivative of Gaussian Filter• The improvement is implemented as:

Gaussian smoothing first and then use differential filter

• This can be replaced by convolution with the derivative of Gaussian

xIG

∂∗∂ )( σ

Ix

Gx

IG∗

∂∂

=∂∗∂ σσ )(

Steps in edge detectionSteps in edge detection• Typical steps:

–Filtering: cut down on noise–Enhancement: amplify the difference

between edges and non-edges–Detection: use threshold–More: estimate geometry and other

operations

Image Processing on GPUImage Processing on GPU• Processing an image is fairly simple and

straightforward. • The filter renders a screen-aligned quad

into an invisible pixel buffer.– The screen-aligned quad has the input image

bound as a texture. – For each rendered pixel, a Cg fragment

program is executed, which does the actual image processing in a local neighborhood.

– Because the quad is rendered using the input image as a texture, the Cg program can access the pixels in the input image though simple texture reads.

Convolution on GPUConvolution on GPU

Typical GPU FrameworkTypical GPU Framework• Load Sources• Apply filters as cg

programs• Render image on

screen• Must also provide

operators for saving images

An Example: An Example: ScotopicScotopic FilterFilter• Vision under reduced illumination,

– lose most of our ability to see color– Scenes appear blurred and shaded in blue

• Blur use Gaussian filter• Blue shaded use night-vision filter• Then edge detection• This procedure mimic the human night

vision system

• GPU implementation enable immediate processing, useful for many real applications, such as military

ExampleExample

More Techniques and ExamplesMore Techniques and Examples

Image GlowImage Glow

Add Glow on GPUAdd Glow on GPU• (a) The scene is rendered normally.• (b) A rendering of glow sources is blurred to

create • (c) a glow texture, which is added to the ordinary

scene to produce • (d) the final glow effect.

Glow on GPUGlow on GPU• The alpha channel of an object's

texture can be used to specify glow sources and the glow source brightness.

Color ControlColor Control• Color correction is part of almost all print

and film imaging applications. – move color images from one color space to

another (say, from Adobe RGB to sRGB); – to stylize images (creating illusions such as

faded film, cross-processing, or other stylistic variations);

– to combine art elements from different sources (such as matching color palettes between different video sources, or models made by different artists);

– to give broad coherence and mood to entire parts of a game or scene without changing the underlying models and textures.

General IdeaGeneral Idea• Change color of individual pixels

– Per channel: alter red, green, and blue components individually

– color-mixing: each output channel may be an operation based on the red, green, and blue components simultaneously

• The mathematics of color corrections can be compactly and easily described in a GPU shader.

• Just as important, they can be controlled effectively with common tools used widely by computer artists and programmers. – E.g.Relying on Adobe Photoshop to let artists

create control resources, which can then be applied in real time via pixel shaders.

Gamma AdjustmentGamma Adjustment• Adjusting Image Gamma and Overall Dynamic

Range in photoshop• Cg program implement: • outPixel = (pow(((inPixel * 255.0) - inBlack) / (inWhite - inBlack), inGamma) *

(outWhite - outBlack) + outBlack) / 255.0;

Curved Level AdjustmentCurved Level Adjustment• Curves tool in Photoshop provides arbitrary remapping of

the color channels

Curved Level AdjustmentCurved Level Adjustment• A 1x256 texture map should be defined,

although smaller maps can often be used effectively

• Before creating the map, the artist must first define the color transformation, typically by applying adjustment layers to existing still images using the Curves tool in Photoshop.

• Create a ramp texture from an original white-black ramp according to the color transformation

Curved Level AdjustmentCurved Level Adjustment• Cg code• float3 InColor = tex2D(inSampler,

IN.UV).xyz; • float3 OutColor; • OutColor.r = tex1D(ColorCorrMap,

InColor.r).r; • OutColor.g = tex1D(ColorCorrMap,

InColor.g).g; • OutColor.b = tex1D(ColorCorrMap,

InColor.b).b;

• Channel-by-Channel Results of the Red, Green, and Blue Remappings

PDE and Image ProcessingPDE and Image Processing• We introduce simple local pixel

filtering for image processing• Some more complex image

processing techniques involves PDEs• Thus, GPU-based PDE solver is very

important– Image diffusion– Image editing–…

Repeated Smoothing: DiffusionRepeated Smoothing: Diffusion• Usually we need to repeat smoothing

operation• Such operation actually is a solution to

the diffusion equation

Φ∇=∂Φ∂

+∂Φ∂

=∂Φ∂ 2

2

2

2

2

yxt

Anisotropic DiffusionAnisotropic Diffusion• Edge preserving smoothing• Use a function g to control diffusion• Smoothing noise while conserve edges

)),,(( Φ∇⋅∇=∂Φ∂ σyxct

Diffusion FunctionDiffusion Function• c() is the diffusion function • A monotonically decreasing function

of the image gradient magnitude • Generally g() can be implemented

Gaussian Diffusion FunctionGaussian Diffusion Function• g() can be implemented by the

function of derivative of Gaussian distribution

• Smoothing noise while preserving edges

DemosDemos• Copyright by Joachim Weickert

How to solve the equation?How to solve the equation?• Typical Partial Differential equation• Basic finite difference operations to

discretize the equation• Thus, a linear system is achieved

–Show how?

• How to use GPU solving linear system?–Linear algebra on GPU