on the perception of bandlimited phase distortion in natural scenes

1

On the perception of Bandlimited Phase Distortion in Natural scenes

Kedar Vilankar, Logesh Vasu and Damon ChandlerComputational Perception and Image Quality

LaboratorySchool of Electrical and Computer Engineering

Oklahoma State University

Importance of Phase Oppenheim and Lim (1981) demonstrated the

importance of phase in signals.

Phase spectrum contributes more to the image’s visual appearance.

Magnitude Spectrum

Phase Spectrum

2 of 51

Cells Compute Local Information Primates V1 is dominated by complex cells. V1 complex cells are insensitive to phase V1 complex cells encode the magnitude

information only. V1 complex cells are localized to receptive

fields.

3 of 51

Local Magnitude is all we need Phase information is implicitly encoded in

local magnitude.

Morrone and Burr demonstrated computation of location of lines and edges (Phase congruency) from local magnitude (Complex cell response).

Also, other researchers have shown only local magnitude is required for scene recognition and categorization.

4 of 51

Local Magnitude is all we need Morgan et. al demonstrated local phase is of

lesser importance than local magnitude.

HVS uses local magnitude information to determine global (image-wide) phase information.

Magnitude Spectrum

Phase Spectrum

5 of 51

Does HVS use Local Phase ? Signal processing perspective, local phase is

important.

If HVS uses only local magnitude, then we should not see any impact of distortion in local phase.

Original ImageLocal Phase Distorted Image

6 of 51

Does HVS use Local Phase ? Complex cells compute local magnitude by

combining the responses of two simple cells.

May also exist a visual mechanism to compute local phase using simple cells.

7 of 51

Does HVS use Local Phase ? Complex cells compute local magnitude by

combining the responses of two simple cells.

May also exist a visual mechanism to compute local phase using simple cells.

If HVS has mechanism to compute local phase, then do we infer global phase from

1. only local magnitude or 2. both local magnitude and local phase

8 of 51

Summary Global phase is most important for image

appearance. (Oppenheim & Lim)

Local magnitude can implicitly encode global phase. (Morrone et al. and Shams et al.)

Local phase is of lesser importance for image appearance

However, local phase distortion has substantial impact on image quality.

9 of 51

Outline Experiment (Local Phase Contribution)

Results (Interesting and surprising)

Discussion (Open questions. We need help)

Algorithm (Local magnitude and phase Distortion Rater)

Conclusion (Our belief)10 of 51

Experiment Measure the relative contribution of local

magnitude and local phase towards image appearance.

Stimuli were created by forming hybrid in complex wavelet subbands.

Each subbands local magnitude and local phase was taken from 2- 4 original images.

11 of 51

Experiment

Frequency (cyc/deg)Low Frequency

HighFrequency

LocalMagnitude

LocalPhase

Original Images12 of 51

Experiment

Frequency (cyc/deg)Low Frequency

HighFrequency

LocalMagnitude

LocalPhase

Original Images13 of 51

Experiment Based on permutations to make hybrid images 14

combination were created.

For each combination 12 stimuli were created.

Five subjects were asked to rate how much each original image contribute to the appearance of the stimulus.

Viewing distance : 60cm.

Five choices : 5%, 10%, 25%, 50% and 75%14 of 51

Result : Combination 1

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Original Images15 of 51

Result : Combination 1

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

16 of 51

Result : Combination 1

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

59%

27%8%

6%

17 of 51

18

Result : Combination 1

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

59%

27%8%

6%

When across frequencies, local magnitude and phase have non cooperative information, then HVS relies mostly on high frequency local magnitude information.

Result : Combination 3

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Original Images19 of 51

Result : Combination 3

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

20 of 51

Result : Combination 3

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

39%

55%

6%

21 of 51

22

Result : Combination 3

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

39%

55%

6%

When local phase across entire frequency channels cooperate, then local phase information dominates local magnitude information.

Result : Combination 12

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Original Images23 of 51

Result : Combination 12

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

24 of 51

Result : Combination 12

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

48%

52%

25 of 51

Result : Combination 12

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

48%

52%

26 of 51

Local phase and local magnitude have equal importance for the appearance of an image

Result : Combination 13

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Original Images27 of 51

Result : Combination 13

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

28 of 51

Result : Combination 13

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

24%

76%

29 of 51

Result : Combination 13

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

24%

76%

30 of 51

High frequency information is more important than low frequency for image appearance.

Discussion Do we infer global phase from local phase?

31 of 51

Result : Combination 12

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

48%

52%

32 of 51

Discussion Do we infer global phase from local phase? Is there visual summation of local phase

across frequency channels?

33 of 51

Result : Combination 3

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

39%

55%

6%

34 of 51

Discussion Do we infer global phase from local phase? Is there visual summation of local phase

across frequency channels? Why HVS needs local phase information?

35 of 51

Discussion Do we infer global phase from local phase? Is there visual summation of local phase

across frequency channels? Why HVS needs local phase information? How local phase is computed in HVS?

36 of 51

Discussion Do we infer global phase from local phase? Is there visual summation of local phase

across frequency channels? Why HVS needs local phase information? How local phase is computed in HVS? What is the neural basis for this computation?

37 of 51

Discussion What is more important low frequency or high

frequency?

38 of 51

Result : Combination 13

Original Images

Low Frequency

Frequency (cyc/deg)HighFrequency

LocalMagnitude

LocalPhase

Stimulus

24%

76%

39 of 51

Discussion What is more important low frequency or high

frequency? Previous research:

Low and high convey independent information about image structure

For categorization task1. Between class – Low frequency2. Within class – High frequency

Information content in low and high frequency. Is this task dependent?

40 of 51

Algorithm LMPDLocal Magnitude and Phase Distortion Rater Algorithm was developed to rate the quality of

local phase distorted images using experimental results.

Algorithm computes local phase as well as local magnitude distortions in an image.

41 of 51

Algorithm LMPDLocal Magnitude and Phase Distortion Rater Local Magnitude distortion

1. Decompose original and distorted images in five scale and ten orientation log-Gabor subbands.

2. For each scalea. Compute local energy maps of the original and

distorted images.b. Compute local MSE map between local energy maps of

the original and distorted images. Use block size of 16 × 16 for local MSE.

c. Collapse local MSE map via the L2 − norm into a single scalar value.

d. Compute correlation between local energy maps of original and distorted images.

e. Compute local magnitude distortion Si(where, i is the index for current scale) by multiplying scalar values obtained in step (c) and (d).42 of 51

Algorithm LMPDLocal Magnitude and Phase Distortion Rater

3. Using Equation combine the local magnitude distortion obtained for each scale in step (e) to compute local magnitude distortion rating in the distorted image.

= () + () + () + () + ()

1, 2, 3, 4, 5 are power coefficients with values of 4, 4, 2, 1.5 and 0.143 respectively

43 of 51

Algorithm LMPDLocal Magnitude and Phase Distortion Rater Local phase distortion

1. Decompose original and distorted images in four levels and four orientation complex wavelet subbands.

2. For each level of the complex wavelet subband

a. Extract local phase information of the original and distorted image.

b. Compute local phase distortion Ei (where, i is the index for current level) by computing MSE between local phase of the original and distorted image obtained in step (a).

44 of 51

Algorithm LMPDLocal Magnitude and Phase Distortion Rater

3. Using Equation combine the local Phase distortion obtained for each scale in step (b) to compute local phase distortion rating in the distorted image.

= () + () + () + ()

1, 2, 3, 4are power coefficients with values of 2.1, 2.4, 2.3, and 2.2 respectively

45 of 51

Algorithm LMPDLocal Magnitude and Phase Distortion Rater Final quality rating of distorted image

α = 0.6

46 of 51

Algorithm LMPDLocal Magnitude and Phase Distortion Rater Database of 48 phase distorted images. Five subjects rated the distorted images. Performance of LMPD compared with other

image quality assessment algorithms.

47 of 51

Algorithm LMPDLocal Magnitude and Phase Distortion Rater Database of 48 phase distorted images. Five subjects rated the distorted images. Performance of LMPD compared with other

image quality assessment algorithms.

48 of 51

PSNR SSIM CWSSIM NQM VIF MAD LMPD

Correlation 0.387 0.531 0.458 0.424 0.580 0.281 0.708

Conclusion Local magnitude is most important

information for image appearance. Local phase also has a substantial,

sometimes dominating contribution. Local phase distortion of the images also

has a substantial effect on image quality.

49 of 51

Conclusion Local magnitude is most important

information for image appearance. Local phase also has a substantial,

sometimes dominating contribution. Local phase distortion of the images also

has a substantial effect on image quality.

We believe that an explicit mechanism does exist in visual cortex for the computation of local phase information.50 of 51

Thank You Questions?

51 of 51

For more stimuli examples please visit

http://vision.okstate.edu/localphase/