non-realistic computer graphicsstefans/slides/03-pixelmanipulation.p… · - digital halftoning -...

__ University of Magdeburg

Pixel Manipulation of Images 1

Non-PhotorealisticComputer Graphics

Pixel Manipulation of Images



Contents

1. Basic Terms and Definitions

2. Halftoning

- Traditional Halftoning

- Digital Halftoning

- Applications to NPR

- Screening

- Screening with Texts

- Embedding Morphed Contours

3. Image Mosaics



1. Basic Terms and Definitions

• first approach: work on the level of the image’s pixels

• input: gray level image in our algorithms usually S(x,y)

• output: black/white or gray level imagein our algorithms usually O(x,y)

• also applicable to color imagesThe application of the discussed algorithms to color images can be derived relatively easy and should not be discussed in detail here.

• application of the algorithms from this chapter: as post-processing to a photorealistic renderer

apply algorithms to scanned images

• introduce artifacts

• goal in photorealistic CG: reduce or remove such artifacts



• pixel intensity

- monotonously between 0.0 and 1.0

- 1.0 black; 0.0 white

This notion is not uniformly used throughout the literature, some authors use it the other way around!

- intensity represents “blackness”

intensity i=1.0 intensity i=0.0intensity ramp



2. Halftoning

• Halftoning is the process of rendering an image on a display device with fewer colors than are in the image.

Halftoning is often also referred to as “Dithering” or “Color Reduction”

• well known image reproduction technique(s)

- traditional halftoning

- digital halftoning



2.1. Traditional Halftoning

• photographic process

• often used in former times for newspaper production

• requires the original photograph, a “halftoning screen” and a camera

• time consuming and cost intensive

• halftone screens:

- two thin glas plates with parallel lines cemented together such that the lines cross at right angles

- contact sheets made of film

• output image composed of a set of variably sized circles (dots)

• newspapers usually 60 to 80 variably sized and variably shaped areas per inch

• magazines and books: 100 to 200



Project a strong light onto the image. The lighter areas, or highlights, of theimage reflect more light, while the darkerareas, or shadows, absorb the light (andreflect less)

When the light is reflected through the camera, it is filtered through a halftone screen, which breaks up the image into a pattern of dots making a halftoned negative on the film. Areas of the original that reflect more light, create larger dots; areas that reflect less light create smaller dots.



Different number of lines per inch on the screen

Different angle between the lines on the two glass plates

33 lpi 53 lpi 75 lpi

0° 20° 45°



2.2. Digital Halftoning

• replace film and halftoning screen by the computer

• algorithmically reduce color depth

• many methods:

Threshold Quantization

Ordered Dithering

Error Diffusion

Screening



2.2.1. Threshold Quantization

• reduce gray levels to 2 by comparing with a given threshold value

input signal

outp

ut sig

nal

0

1

1

threshold



• simple algorithm: iterate over all pixels and do the comparison

• single parameter: threshold

• no control over induced artifacts

1 for x := 0 to width(S)-1 do

2 for y := 0 to height(S)-1 do

3 if S[x,y] > threshold

4 then O[x,y] := 1.0

5 else O[x,y] := 0.0

6 od

7 od



Threshold quantization



2.2.2. Ordered Dithering

• Replace areas of pixels with patterns of the same size as the area which correspond to the medium intensity of the area.

• Introduces artifacts from the patterns that are used.

• applications

- display devices

- printing



• dither patterns



• dither matrices

- different representation for dither patterns

- To achieve intensity i, turn on all pixels whose values are less than i.

01

2

3

4

5

6

7

8



• more dither matrices



1 for x := 0 to width//n step n do

2 for y := 0 to height//n step n do

3

4

5

6

7 od

8 od

R := get n×n region of the image

medium intensity = 63%

i := mediumIntensity(R)

P := pattern[i]

replace R with P



Ordered Dithering



2.2.3. Error Diffusion

• ordered dithering introduces artifacts

• When replacing a pixel of intensity i (0.0 i 1.0) by a pixel of intensity i being either 0.0 or 1.0, an error is introduced.

• basic idea: distribute this error to the neighboring pixels

• “smoothes” the image reduces artifacts



1 for y := height(S) to 1 step -1 do


3 K := approximate(S[x,y])

4 O[x,y] := K

5 error := S[x,y] - K

6 S[x+1,y] := S[x+1,y] + 7/16 * error

7 S[x-1,y-1] := S[x-1,y-1] + 3/16 * error

8 S[x,y-1] := S[x,y-1] + 5/16 * error

9 S[x+1,y-1] := S[x+1,y-1] + 1/16 * error

10 od

11 od

Floyd/Steinberg error diffusion algorithm;portions of the error term ar distributed

among neighboring pixels.



Floyd/Steinbergerror diffusion



2.2.4. Concluding Examples



2.3. Applications to NPR

• algorithms so far: reducing the number of intensity levels in an image while keeping the introduced artifacts at a minimum

This goal has been set in “photorealistic” image synthesis as well as in applications for printing and display devices

• our goal now: use the algorithms to introduce artifactsuse properties of some algorithms to achieve the opposite goal as stated above



2.3.1. Non-photorealistic Dither Matrices

• primary goals for the design of dither matrices:

- distribute pixels as evenly as possible

- prevent patterns

• interesting effects if pixels are grouped so that they form visible patterns



• build matrix starting from the bottom left element and moving diagonally up to the top right element

triangular patterns

• large matrices needed to achieve a visible effect

6101315

371114

14812

0259

4D



715110

38134

51492

110126

4D

• build matrix starting in the middle and moving outwards

linear artifacts

• large matrices yield better results

• in general: artifacts increase with matrix size



2.3.2. Halftoning Using Lines

• Error diffusion algorithms result in a proper distribution of “dots” (pixels).

• instead of setting single pixels draw lines

• if a line is drawn, images get much darker compensate with new error term

• variations in line slope, width, length …



1 for y := height(S) to 1 step -1 do


3 K := approximate(S[x,y])

8 O[x,y] := K

9 S[x+1,y] := S[x+1,y] + 7/16*error

10 S[x-1,y-1] := S[x-1,y-1] + 3/16*error

11 S[x,y-1] := S[x,y-1] + 5/16*error

12 S[x+1,y-1] := S[x+1,y-1] + 1/16*error

13 od

14 od The rest of the algorithm stays the same.

4 if (k=1) then

5 drawLine(O,x,y,m)

If we would set a pixel in the original, we draw a line which is m pixels long.

6 error := S[x,y]-K-(m-1)Instead of placing just 1 pixel, there are m pixels set now. These are (m-1) pixels to many the error term has to be reduced.

7 else error := S[x,y]-K

Nothing changes if there is no pixel set (line drawn).

• Example: Floyd/ Steinberg



2.3.3. Summary

• Artifacts from the dithering algorithms remain visible.

• parameters for lines (or dither patterns):

- random variations

Not a function of the image nor the portrayed geometry

- use attributes from the underlying geometry

So called “G-Buffers” have already been discussed

- ask user for additional input



2.4. Screening

• Ordered Dithering/Error Diffusion: almost no control over the introduced patterns

• unimaginative and uninformative patterns

• Screening:

combination of two images

viewed from a distance: texture in the original image

Viewed from close up: information of the second image



2.4.1. Basic Method

• given:

input gray-level image S

nm gray-level threshold matrix M

• Algorithm:

compare each pixel in S with a pixel in M

Use values in M as threshold, i.e. if value in S is larger than in M set the output pixel

• also possible:

repeat M over the area of S to get a “dither screen” D

do the comparison with pixels in D

This avoids the modulo operations!





3 if S[x,y] < M[x mod n,y mod m]

4 then O[x,y]:=1

5 else O[x,y]:=0

6 fi

7 od

8 od

S

M

O



1 construct D by repeating M over the image



4 if S[x,y] < D[x,y]

5 then O[x,y]:=1

6 else O[x,y]:=0

7 fi

8 od

9 od



Basic Screening



• texture determined by the threshold matrix

• arbitrary images can be used

• properties of the threshold matrix determine quality of the result

uniform distribution of threshold values

homogeneous spatial distribution of threshold values

• design matrices to meet these requirements or tune them afterwards



1. uniform distribution of threshold values

- dither screen should contain an equal number p of pixels of each possible intensity value (0 i imax)

- enables the uniform reproduction of the maximum range of gray tones

2. Homogeneous spatial distribution of threshold values

- pixels of like threshold value should be spread uniformly throughout the dither screen

- approximation of the same gray tone in different regions of the image in the same fashion

- automatically met when matrices with a uniform threshold distribution are replicated in a screen



2.4.2. Histograms and Their Use

• histogram:

- map of an image’s grayscale luminance values

- comes from statistics to describe (frequency) distributions

- aka: Probability Mass Function – PMF

describes the probability with which a pixel in the image has a certain intensity value

• histogram computation:

- count all pixels that have a certain intensity value i

- list the numbers in a “table”



1 for i:=0 to 255 do H[i]:=0



4 H[S[x,y]] := H[S[x,y]] + 1

5 od

6 od

7 for i:=0 to 255 do

8 H[i]:=H[i]/(height(S)*width(S))

9 od

2.4.2.1. Histogram Computation

• count all pixels that have a certain intensity value i

• list the numbers in a “table”



Example image and histogram



2.4.2.2. Histogram Equalization

• need to achieve a uniform distribution of gray values

• apply pixel level transformation techniques that change the intensity distribution

• most promising: Histogram Equalization

• compute cumulative sums of the histogram values:

• Cumulative Mass Function (CMF)The CMF is a statistical measure of the probability that a pixel is of a given intensity or less.

• alter the PMF using the CMF as a lookup table

• compute for each pixel a new gray value g´ by g´=255·C[g]

2550][][0

ikHiCi

k



1 H:=histogram(S)

2 C[0]:=H[0]

3 for i:=1 to 255 do C[i]:=C[i-1]+H[i] od

4 for x:=0 to width(S)-1 do

5 for y:=0 to height(S)-1 do

6 O[x,y]:=255*C[S[x]]

7 od

8 od

• Algorithm for histogram equalization



• Results of Histogram Equalization:

- histogram is spread out more uniformly

- gray value distribution is more uniform

• many HE algorithms possible

• Here: non-adaptive uniform histogram equalization (NAHE)

- works uniformly on the whole image

- transformation of one pixel independent from transformation of neighboring pixels

- drawback: enhances low-contrast detail, thus also increases contrast of noise in the image



2.4.2.3. Other options…

• … for achieving the two qualities required for dither screens

spatial homogeneity still not achieved

• subdivide the screen into small blocks and apply NAHE to each of these blocks

pixels of each block are transformed using PMF and CMF of this block only

good approximation of input gray levels

strong blocking artifacts

dither screen



• avoid these problems

- … by taking neighboring blocks into account

- Adaptive Histogram Equalization (AHE)

1 split input image into small blocks

2 foreach block b do

3 hb := CMF(b)

4 hb* := linear interpolation of the hb

of all neighboring blocks

5 Transform intensity values of b

based on hb*

6 od



2.4.3. Procedural Screening

• introducing texture in an image in photorealistic CG procedural textures

• Here: procedural dither screens

- created directly without the need for a second input image or image processing techniques

- can be produced at any scale

- can produce non-repeated textures of any size

• procedural dither screens

- have to satisfy the conditions for dither screens

- Spatial homogeneity can be achieved by periodically repeating the screen over the whole image.

concentrate on the uniform distribution of gray values



• Let (s,t) be a function over [0,1][0,1] with a uniform distribution of values.

• Let all values of be between 0 and 1:: [0,1][0,1][0,1]

• Define a,b to be a set of points such that a (s,t) b where a,b [0,1] and a < b

• |a,b| counts the number of all points in such a set

• A uniform distribution of values is achieved, when for any value of n (n>1) the value of is a constant for any value of i with 0i<n.

• Such functions are called dither kernels.

|| 1,

n

i

n

i



2.4.3.1. Dither Kernels

• compute an intensity level for each pair of input values s

and t

• for screening: map pixel positions to input values for such a dither kernel

• mapping function M to transform (x,y) into (s,t)

• Example:

describes a double sided ramp

most simple mapping: apply kernel to a small image block and repeat over the image

otherwise22

5.0if2),(

s

ssts

m

my

n

nxyxM

mod,

mod),(



1 define a dither kernel function

2 define a mapping function M

3 for x:=0 to width(S)-1 do

4 for y:=0 to height(S)-1 do

5 (s,t):= M(x,y)

6 i := (s,t)

7 if S[x,y]<i then O[x,y]:=1

else O[x,y]:=0

8 od

9 od

(s,t)

Input image Output imageMapping

Texture



• choose between different kernels and mappings

• dither kernel controls texture shape

• mapping function controls size and orientation of the texture

• example for a 2D dither kernel:

Any areas with an intensity below I are hatched using a linear ramp, in all other areas crosshatching is achieved

otherwise)1(

if),(

IsI

IsItts



modulo mappingrotation by 20°

modulo mappingrotation by 20° (I=0.7)

modulo mappingrotation by 20° (I=0.3)


modulo mapping




2.4.3.2. Texture variation

• adding a displacement function D T

Displacement functions take both variables, thus dependencies between s and t can be modelled. Also keep in mind that the requirements for dither screens have to be fulfilled

• image based control of texture

C(s,t) is an auxiliary image that contains a map of a special feature that has to be encoded in the resultant image

)),(),,((),( tsDttsDstsT ts

)),(),(),,(),((),( tsDtsCttsDtsCstsT ts



Veryovka and Buchanan: „Comprehensive Halftoning of 3D Scenes“. In: Proceedings of EUROGRAPHICS’99. Computer Graphics forum, vol. 18, no. 3, pp. C13-C21



2.4.3.3. Screening … and then …

• shape of introduced artifacts is controlled by a second image (or function)

• Designing dither screens manually …

select a pattern for the artifacts

observe the requirements for gray value distribution

• Is there any way to do it better?

screening with text and contours



2.5. Screening with Texts

• history:

- using an old printer (typewheel) and produce dark regions by overprinting several characters

- chose small characters (period, colon) for bright regions

- microletters on money bills



Pnueli, Y. and Bruckstein, A. M.“Gridless Halftoning: A

Reincarnation of the Old Method”Graphical Models and Image

Processing: GMIP, 58(1):38–64.



V. Ostromoukhov, R.D. Hersch: “Artistic Screening”

In: Proceedings of SIGGRAPH’95, pp. 219-228.



microletters on a german 10 DM bill



2.5.1. Preprocessing: Dither Rows

• render characters for different intensities in a block of given size

• increasing character size/width makes the block darker

• start with tiny and huge characters and interpolate

dither rows



V. Ostromoukhov, R.D. Hersch: “Artistic Screening” In: Proceedings of SIGGRAPH’95, pp. 219-228.



2.5.2. The Algorithm

1 foreach row of blocks r in S do

2 foreach block b in row r of S do

3 i := average intensity if block b

4 place letter L[i,b mod t] at the

next available position in O

5 od

6 od



2.5.3. An Example



2.6. Embedding Morphed Contours

• Text yields nice result but is not very “general”

• want to produce images like this and similar artwork by M.C. Escher

more scientific approach

- distinguish between

Screen Element Definition Space (SEDS) and

Screen Element Rendition Space (SERS)

- i.e. distinguish between

definition of a screen element

rendering of a screen element

- What is a Screen Element?

M.C. Escher Sky and Water



2.6.1. Screen Element

• any shape defined in its own coordinate system by its contour

• for example, one of our characters (letters)

• fixed contours are associated with fixed intensity levels

• shapes may be complex or simple – preferred defined via Bézier curves



2.6.2. The Algorithm

• four main steps:

1. define the shapes of the screen elements for key intensity levels

2. scale these shapes to achieve the desired intensity

3. interpolate to get shapes for all intensity levels

4. perform the actual screening

• screening process may be accompanied by an additional non-linear mapping to achieve distortions



2.6.2.1. Defining Screen Elements

• use a drawing program (e.g., Adobe Illustrator)

• Make sure that each contour is defined using the same number of control points.

• Screen Element Definition Space is the coordinate space in which the Screen Elemnts are drawn. Use a convenient coordinate system (e.g., 0 to 1).

• assign shapes to key intensity levels



2.6.2.2. Achieve Desired Intensity

• so far only shapes are defined

• scale these shapes such that they cover exactly a fraction of size i of the “block”

• i=0 is an infinitely small dot



2.6.2.3. Shapes for All Intensity Levels

• screen elements only defined for a fixed number of intensities interpolation to get shapes for all intensity levels

• usually linear interpolation

• interpolation speed can be controlled by parameters



• control interpolation speed by parameters i(z) varying between 0 and 1

• coordinates of a contour control point P at intensity z interpolated between two fixed contour points Pi and Pi+1

given by:

• Pi and Pi+1 belong to intensities zi and zi+1

• i(z) are mapped to the intensity range [zi,zi+1]

• i(z) can be defined to be any function

))(()))((1()( 0100 PPzPPzPzP iiii



V. Ostromoukhov, R.D. Hersch: “Artistic Screening”. In: Proceedings of SIGGRAPH’95, pp. 219-228.



2.6.2.4. The Screening Process

• so far everything in SEDS

• transfer screen elements into the output bitmap SERS

• enables screen elements to be defined independent from size or resolution of the output bitmap

• coordinate transformation from SEDS to SERS

• generate a discrete screen element for each of the 256 intensity levels

divide intensity interval between z=0 and z=1 by 255

Generate screen dot contours at levelsz=0, z=1/255, z=2/255, … z=1

rasterize screen dots by applying well known shape rasterization techniques

fill the contours appropriately

• select screen element based on the medium intensity of the target region (block)



2.6.2.5. Non-linear Mapping

• normally: A rectangular area in SEDS is mapped to a rectangular area in SERS (block of pixels).

• mapping can be defined as a non-linear function

• interesting effects but rasterization more complicatedconformal mapping: rectangular gridgrid following electro-magnetic field lines

fisheye lens mapping

V. Ostromoukhov, R.D. Hersch: “Artistic Screening” In: Proceedings of SIGGRAPH’95, pp. 219-228.



conformal mapping: rectangular gridgrid following electro-magnetic field lines





View of the Ibn Tulun Mosque V. Ostromoukhov, R.D. Hersch: “Artistic Screening”




View of the Ibn Tulun Mosque





2.6.3. Summary

• also known as “Artistic Screening”

• preprocessing time required for computing in SEDS

• non-linear mapping yields nice images

• microlettering

• Non-repetitive screens can not be photocopied and scanned without producing Moiré patterns.



V. Ostromoukhov, R.D. Hersch: “Artistic Screening”. In: Proceedings of SIGGRAPH’95, pp. 219-228.



3. Image Mosaics

• mosaic in arts: surface decoration of small colored components (stone, mineral, glass, tile, shell) closely set to an adhesive ground

• such mosaics can also be simulated

• image mosaics: small components are images themselves, together they portray a larger subject



© Adam Finkelstein, Princeton University -- http://www.cs.princeton.edu/gfx/proj/mosaic/blair.html

This is a picture of Blair Arch at Princeton University composed of 250 smaller pictures from around the Princeton campus. The picture was created in celebration of Princeton's 250th anniversary.



This image was created by dividing up a picture of Marilyn into small tiles, and then arranging those tiles in a grid. The arrangement was based on an algorithm which attempted to optimally suggest the big picture of JFK with the grid of tiles. Then the grid of tiles was used as a printer's screen to „halftone“' the JFK image. We ran the algorithm twice (using two different size images of Marilyn) and then cut-and-pasted between the two resulting images to select the best parts from each.

© Adam Finkelstein, Princeton Universityhttp://www.cs.princeton.edu/~af/cool/jfk-mm.html



• Four steps for producing an image mosaic

1. choose images which are to be used as tile images

2. choose a tiling grid

3. find an arrangement of the mosaic tiles within the grid

4. possibly perform a color correction on the tiles to match the tone of the target image



3.1. Choose Tile Images

• rather artistic choice

• tiles should match the subject of the target image (e.g., JFK and Marylin)

• small version of the target image itself, then color correction is necessary

• selection from an image database



3.2. Choosing a Tiling Pattern

• Question: How to arrange the tiles spatially?

• Also: How to overlay the target image with a grid where in each cell fits exactly one tile?

• several possibilities:

scattered layout

regular tiling grid using different tile shapes

multiresolution tiling



3.2.1. Scattered Layout

• no tiling grid at all

• random placement of the tiles

• tiles may or may not be rotated

• tiles may overlap

• looks like a huge pile of photographs

• for later: always compare the tile with the original image and not with the possible tile being already at a target position



Tom McKenna and Gonzalo R. Arce: „New Image Mosaic Structures“. Technical Report, University of Delaware, 1998.

Mosaic with Scattered Layout



T. McKenna and G. R. Arce:„New Image Mosaic Structures“. Technical Report, University of Delaware, 1998.

Mosaic with Scattered andRotated Layout



3.2.2. Regular Tiling Grid

• tiles placed regularily next to each other

• tiles do not overlap

• several tile shapes possible




Mosaic with RectangularGrid Tile Layout



Courtesy of Ken Chidlaw.

Mosaic with HexagonalGrid Tile Layout



3.2.3. Multiresolution Mosaics

• basic principle: tiles are mosaics themselves

• more and smaller tiles placed in regions with high detail

• fewer and larger tiles in regions with low detail

• need a method to break up the target image into the desired regions

• principle idea: use a quadtree, subdivisions based on the level of detail in the image

• measuring level of detail: identification of the frequency content of the image region

using a Fourier or wavelet transform. Regions with high frequent content can be considered very detailed.

Computing the average value of an edge detection algorithm. Regions with many edges can be considered very detailed.

• subdivision criterion for the quadtree is level of detail



T. McKenna and G. R. Arce:„New Image Mosaic Structures“.

Technical Report, University of Delaware, 1998.

Multi-ResolutionImage Mosaic




Combinationof Quad Tree Placementand Rotated Tiles



• more simple way to produce multiresolution mosaics:

- create a mosaic using just a few tiles

- use these mosaics as new tiles for the second step

- … and so on

• works best when using just one image (i.e., smaller versions of the target image as initial tiles)

• color correction necessary



The image of Mona is made up of a bunch of little images of Mona, which in turn are made up of even littler images of Mona, which you probably cannot see in this image.

© Adam Finkelstein, Princeton Universityhttp://www.cs.princeton.edu/~af/cool/mona.html



3.3. Arrange Tiles Within the Grid

• Given a target region, which tile should go there?

- use the same tile image everywhere – color correction necessary

- don’t care, just take any random tile – color correction necessary

- arrange tiles manually by hand – You don’t want to do this …

- select tiles by matching features of the tile and the target region



• match the average color of the target region and the average volor of the tile image

compute the mean squared error between the RGB channels of both regions (images)

select the tile with the smalles J(I,T)

• match different properties of the tartget region and the tile

match edges, texture, …

high computational effort



3.4. Color Correction

• if tiles not chosen in a way that the tile color matches the target region color

• following description for gray levels, can be extended to color by working with different color channels in a color model

RGB

YIQ

HSV

LAB

…



3.4.1. The Problem

• given an image tile T

average color of the tile: at

• given a target region R

average color of the region: ar

• Change the color of the tile’s pixels so that after the adjustment the tile’s average color is ar.

• Color Correction Rules:

- input: image tile, desired average color

- generate a color correction function F: R1 R1



3.4.2. Color Correction Functions

• constant function F(x)=a each pixel set to a

original colors completely ignored

mosaic with uniformly colored tiles

• color scaling function assume at larger than desired average color a

scaling factor for each pixel then a/at F(x) = (a/at)x

• color shifting function shift color of the tile’s pixel by difference between a and at

F(x) = x + (a - at)

may shift out of the allowed range

• more sophisticated functions other means of color correction

gamma correction etc.



3.4.2.1. Scaling and Shifting



3.4.2.2. Combination Scaling and Shifting

• sufficient results for image mosaics

• relatively easy to compute

• basic idea:

- shift colors when not leaving the allowed color range

- otherwise shift as far as possible ans scale the result until the desired average color a is achieved

Shifting and scaling colors to darken an image tile. The distributionis shown as a histogram. Dashed line indicates average.



3.4.2.3. An Example

• assume desired average color is less than the average color in the tile a < at

• If minimum color in tile mt is greater than the difference at-a then shift using F(x)=x+(at-a) without leaving allowed color range

• otherwise (if mt(at-a)) combine shift and scaling using F(x)=a(x-mt)/(at-mt)

• similar formula for symmetric cases



© Adam Finkelstein,Princeton University

Original gray ramp,dithered usingbasic screeningand different mosaicingtechniques

non-realistic computer graphicsstefans/slides/03-pixelmanipulation.p… · - digital halftoning -...

Documents