The Color Of Water

John Breneman IV, Henryk Blasinski, Joyce Farrell

CASIS Workshop, LLNL

5/21/2014

U S I N G U N D E R WAT E R P H O T O G R AP H Y T O E S T I M AT E WAT E R Q U AL I T Y

Underwater color is a challenge

Surface color Manual color

Standard color Model-based color

This talk

1. Motivation

2. Extending the image formation model

a. Spectral basis functions

b. Medium scatter factors

3. Model evaluation

a. Prediction of real world dive data

b. Water content charts

4. Effect on color correction techniques

a. 3x3 vs 3x4 color corrections

Why a physical model for underwater lighting?

1. Better understanding of oceanic composition

A. What’s the “viz” at a popular dive site? Murky or Clear?

B. Is there a plankton bloom?

2. Insights for correcting underwater color images

A. Why are underwater dive filters so popular?

B. What form should our color correction take?

C. How can we identify the best color transform for a picture?

Roesler et al, 1989

Standard Linear Image Formation Model

m j = g jDl s j,lrlIll=1

N

å

Sunlight

Camera

Scene

Lossy Medium

m j = g jDl s j,lrlI0,le-Dpal

l=1

N

å

Camera

Scene

Medium

pD

eI

,0

Beer-Lambert

Law

Depth = Dp meters

Sunlight

Lossy Medium With Absorption Basis

m j = g jDl s j,lrlI0,le-Dpal

l=1

N

å

,,,, NAPCDOMw

,,,, NAPNAPCDOMCDOMw ccc

Lossy Medium With Absorption Basis

Camera

Scene

Medium

m j = g jDl s j,lrlI0,le-Dpal

l=1

N

å

Sunlight

Multilayer Lossy Medium

m j = g jDl s j,lrlI0,le- al ,lDdll=1

p

å

l=1

N

å

Medium 1

Medium 2

…

Medium p Camera

Scene

Sunlight

Multilayer Lossy Medium and Scatter

m j = g jDl s j,lrlI0,le- al ,lDdll=1

p

å

l=1

N

å +bs j

Medium 1

Medium 2

…

Medium p Camera

Scene

Scatter

Direct

Sunlight 6 unknowns: c{NAP,CDOM,Φ},bsj

Model Estimation

First term quasi-convex, for γ≥1

• m is smooth, convex

Use iterative first-order Taylor expansions for m → affine

Solve via cvx convex optimization toolbox

)(min ,

,,

,,,,

^

,,

lk

qtj

qtjqtjbsc

cRmmjlk

0, lkkcB

0jb

lk ,

j

qj

N d

tjjqtj bseIrsgm

p

l

ll

,

1

,0,,,,1

,

1,,

1

,)(

1)(

lklk

k l ll

lk ccdd

cR

For:

Test The Model With An Underwater Color Rig

! !(! ! ! )

= argmin ! ! ,! − !! ! ,! ! !! − ∇! ! ,! ! !

! ! !! ! ! − ! !

! − ! ! ,!! ,!

subject!to ! ! ! ! ,!! ! !

≥ 0 ! ! " #! ! " #,!! ! !

≥ 0 ! ! " #$ ! ! " #$ ,!! ! !

≥ 0

(11)

where

! ! ,! ! !!

= ! ! ! ! ! ! ,! ! ! ,!

!

! ! !

!! ,! !! ! ! !

(! )! ! !

!! !

(12)

∇! ! ,! ! !!

= ! ! ! ! ! ! ,! ! ! ,!

!

! ! !

!! ,! ! ! !! ! ! !

(! )! ! !

!! !

(13)

· Repeat the second step until all ! depth levels have been analyzed.

Note that ! ! ,! represent the measured sensor values of the k-th target in the j-th channel at the p-th depth level. The

subscript ! is ignored for convenience. The inequality constraint introduced in the optimization problem follows from

the non-negativity of the absorption coefficient values.

The convergence of this algorithm is guaranteed by the properties of convex functions. To confirm this, a numerical

stability was empirically verified using data simulated according to this model and solving for the unknown parameters.

Estimation error levels were comparable to the numerical precision.

4. DATA COLLECTION

To test our model for underwater illumination we constructed a color rig using a commercial point-and-shoot camera, a

Canon SX260HS, in an underwater housing and an XRITE color target. The XRITE color target was chosen primarily

because it is widely used and easily obtainable. Although the XRITE color patches themselves are not linearly

independent in the spectral space, they easily cover the spectral extent of the camera’s color filter sensitivities and

provide a sufficient basis for estimating the relative weights of the four absorption basis functions. The color target was

enclosed within a polycarbonate case to protect against the harsh underwater environment.

Figure 2. Underwater rig for color photography. The

color calibration target was an XRITE ColorChecker

(formally known at the Macbeth ColorChecker). The

calibration target was enclosed in a waterproof

polycarbonate casing. A depth gauge was visible in

all camera images in order to record the depth at

which each image was captured. The rig was

designed to minimize specular highlights and

shadows.

4.1 Camera

An alternative firmware for the camera, CHDK[11] was loaded to allow access to the raw image sensor data and to

program the camera to take a series of photos at a predefined interval. The camera was configured to operate in aperture-

priority mode while the exposure duration and ISO-sensitivity were selected automatically using the camera’s light

meter.

Test The Model With An Underwater Color Rig

! !(! ! ! )

= argmin ! ! ,! − !! ! ,! ! !! − ∇! ! ,! ! !

! ! !! ! ! − ! !

! − ! ! ,!! ,!

subject!to ! ! ! ! ,!! ! !

≥ 0 ! ! " #! ! " #,!! ! !

≥ 0 ! ! " #$ ! ! " #$ ,!! ! !

≥ 0

(11)

where

! ! ,! ! !!

= ! ! ! ! ! ! ,! ! ! ,!

!

! ! !

!! ,! !! ! ! !

(! )! ! !

!! !

(12)

∇! ! ,! ! !!

= ! ! ! ! ! ! ,! ! ! ,!

!

! ! !

!! ,! ! ! !! ! ! !

(! )! ! !

!! !

(13)

· Repeat the second step until all ! depth levels have been analyzed.

Note that ! ! ,! represent the measured sensor values of the k-th target in the j-th channel at the p-th depth level. The

subscript ! is ignored for convenience. The inequality constraint introduced in the optimization problem follows from

the non-negativity of the absorption coefficient values.

The convergence of this algorithm is guaranteed by the properties of convex functions. To confirm this, a numerical

stability was empirically verified using data simulated according to this model and solving for the unknown parameters.

Estimation error levels were comparable to the numerical precision.

4. DATA COLLECTION

To test our model for underwater illumination we constructed a color rig using a commercial point-and-shoot camera, a

Canon SX260HS, in an underwater housing and an XRITE color target. The XRITE color target was chosen primarily

because it is widely used and easily obtainable. Although the XRITE color patches themselves are not linearly

independent in the spectral space, they easily cover the spectral extent of the camera’s color filter sensitivities and

provide a sufficient basis for estimating the relative weights of the four absorption basis functions. The color target was

enclosed within a polycarbonate case to protect against the harsh underwater environment.

Figure 2. Underwater rig for color photography. The

color calibration target was an XRITE ColorChecker

(formally known at the Macbeth ColorChecker). The

calibration target was enclosed in a waterproof

polycarbonate casing. A depth gauge was visible in

all camera images in order to record the depth at

which each image was captured. The rig was

designed to minimize specular highlights and

shadows.

4.1 Camera

An alternative firmware for the camera, CHDK[11] was loaded to allow access to the raw image sensor data and to

program the camera to take a series of photos at a predefined interval. The camera was configured to operate in aperture-

priority mode while the exposure duration and ISO-sensitivity were selected automatically using the camera’s light

meter.

Image Comparison at 10m: Clear and Murky Water

Measured Predicted

Murk

y

Cle

ar

Water Content: Clear and Murky Water Murky Water Clear Water

10m Effective Absorption

Color Correction Underwater

Traditional color transform using 3x3 matrix:

Offset color transform using 3x4 matrix:

j

N d

jjj bseIrsgm

p

l

ll

1

,0,1

,

c

c

c

d

d

d

B

G

R

ccc

ccc

ccc

B

G

R

333231

232221

131211

134333231

24232221

14131211

c

c

c

d

d

d

B

G

R

cccc

cccc

cccc

B

G

R

15% - 50%

Color Correction Underwater – Murky Dive Example

Traditional color transform using 3x3 matrix:

Offset color transform using 3x4 matrix:

j

N d

jjj bseIrsgm

p

l

ll

1

,0,1

,

Reference 5 Meters 20 Meters

Reference 5 Meters 20 Meters

Summary

Changing depths changes natural illumination

• We can describe this with a physics-based model

Physics model is quasi-convex

• Solve regularized optimization iteratively

Particulate scatter is not handled well by 3x3 color correction schemes

• Build affine color correction transforms

Questions?

Acknowledgements:

Colin Roesler, Annick Bricaud ,Tetsuichi Fujiki, Sumit Chawla

Clear and Murky Water 20m Example

Corrected Measured

Murky

Clear

Corrected Measured

Why does clear water produce poor color?

Illuminant

Illuminant

Water Quality-Independent Color Solutions

1. Bring artificial light

• Expensive hardware

2. Apply an optical filter

• Tuned to depth, water content

3. Exposure Bracketing

• Long exposure Red

• Short exposure Green, Blue

• Apply 3x4 color transform

Backup – Bi-convex model fitting