11

CLUTTER MITIGATION DECISION (CMD)CLUTTER MITIGATION DECISION (CMD)THEORY AND PROBLEM DIAGNOSISTHEORY AND PROBLEM DIAGNOSIS

RADAR MONITORING WORKSHOP

ERAD 2010SIBIU ROMANIA

Mike DixonNational Center for Atmospheric Research

Boulder, Colorado

22

Why do we need to know where clutter is in order Why do we need to know where clutter is in order to filter it effectively?to filter it effectively?

We need to filter We need to filter normal-propagationnormal-propagation (NP) clutter and (NP) clutter and anomalous-anomalous-

propagationpropagation (AP) clutter (AP) clutter

The filters used remove weather power in some circumstances:The filters used remove weather power in some circumstances:

– Velocity close to 0Velocity close to 0

– Spectrum width close to 0Spectrum width close to 0

These conditions occur both in:These conditions occur both in:

– ClutterClutter

– Stratiform precipitationStratiform precipitation

We therefore need a technique which identifies likely locations of clutterWe therefore need a technique which identifies likely locations of clutter

The filter is applied only at those gates with a The filter is applied only at those gates with a high probability of clutterhigh probability of clutter..

33

Problem - weather and clutter combinedProblem - weather and clutter combinedReflectivity plotReflectivity plot

44

Problem - weather and clutter combinedProblem - weather and clutter combinedVelocity plotVelocity plot

55

What happens if we filter everywhere?What happens if we filter everywhere?

Filtered reflectivity – applying clutter filter everywhere

Weather power removed

66

And if we use CMD?And if we use CMD?

Filtered reflectivity using CMD

77

Combined clutter and weather.Combined clutter and weather.Weather and clutter are distinct.Weather and clutter are distinct.

88

Combined clutter and weather spectrum.Combined clutter and weather spectrum. Weather and clutter overlap somewhat. Weather and clutter overlap somewhat.

99

Combined clutter/weather spectrum.Combined clutter/weather spectrum.Weather and clutter overlap completely. Weather and clutter overlap completely.

1010

Notch filter – removes weather informationNotch filter – removes weather information

1111

Adaptive filter – does not remove weatherAdaptive filter – does not remove weather

1212

Adaptive filter has a difficult timeAdaptive filter has a difficult timewhen clutter and weather overlap when clutter and weather overlap

1313

Doppler RadarDoppler Radar CMD CMD

1414

MotivationMotivation

Adaptive spectral clutter filters show great promise for Adaptive spectral clutter filters show great promise for intelligently filtering clutter power while leaving weather intelligently filtering clutter power while leaving weather power largely unaffected.power largely unaffected.

However, these filters still remove weather power under the However, these filters still remove weather power under the following circumstances:following circumstances:– the weather return has a the weather return has a velocity close to zerovelocity close to zero;;– the weather return has a the weather return has a narrow spectrum widthnarrow spectrum width..

This tends to occur with This tends to occur with stratiformstratiform weather in the region of weather in the region of the the zero isodopzero isodop..

In order to mitigate the problem, information other than In order to mitigate the problem, information other than that used by the filters must be used to determine whether that used by the filters must be used to determine whether clutter exists at a gate.clutter exists at a gate.

1515

Principal feature fields for CMDPrincipal feature fields for CMD

In order to identify gates with clutter, we use a number of In order to identify gates with clutter, we use a number of so-called feature fields. These contain information which is so-called feature fields. These contain information which is independent of that used by the clutter filter.independent of that used by the clutter filter.

The feature fields used in the latest CMD version are:The feature fields used in the latest CMD version are:

– The The TEXTURETEXTURE of the reflectivity field – of the reflectivity field – TDBZTDBZ..

– The The SPINSPIN of the reflectivity field. This is a measure of of the reflectivity field. This is a measure of how often the reflectivity gradient changes sign.how often the reflectivity gradient changes sign.

– The The Clutter Phase AlignmentClutter Phase Alignment or or CPACPA, which is a measure , which is a measure of the pulse-to-pulse stability of the returned signal.of the pulse-to-pulse stability of the returned signal.

1616

TEXTURE of reflectivity - TDBZTEXTURE of reflectivity - TDBZ

TDBZ is computed as the TDBZ is computed as the mean of the squared mean of the squared reflectivity differencereflectivity differencebetween adjacent gates.between adjacent gates.

TDBZ is computed at each TDBZ is computed at each gate along the radial, with the gate along the radial, with the computation centered on the computation centered on the gate of interest.gate of interest.

TDBZ at a gate is computed TDBZ at a gate is computed using the dBZ values for the 4 using the dBZ values for the 4 gates on either side of the gates on either side of the gate of interest.gate of interest.

Computed for this gate

Using data from these gates

1717

TDBZ feature fieldTDBZ feature field

DBZ TDBZ

Example of TDBZ – Denver Front Range NEXRAD - KFTG

1818

Reflectivity SPINReflectivity SPIN

For a point at which a gradient sign change occurs, let x be the reflectivitychange from the previous gate and y be the reflectivity change to the next gate.

Then

SPIN CHANGE = (|x| + |y|) / 2

xy

x y

x y

1919

SPIN feature fieldSPIN feature field

DBZ SPIN

Example of SPIN – Denver Front Range NEXRAD – KFTG(SPIN is noisy in low SNR regions)

2020

Clutter Phase Alignment - CPAClutter Phase Alignment - CPA

In In clutterclutter, the phase of each pulse in the time series for a , the phase of each pulse in the time series for a particular gate is particular gate is almost constantalmost constant since the clutter does not since the clutter does not move much and is at a constant distance from the radar.move much and is at a constant distance from the radar.

In In noisenoise, the phase from pulse to pulse is , the phase from pulse to pulse is randomrandom..

In In weatherweather, the phase from pulse to pulse , the phase from pulse to pulse will varywill vary depending on the velocity of the targets within the depending on the velocity of the targets within the illumination volume. illumination volume.

2121

I,Q data is a complex numberI,Q data is a complex number

I

Q

phase

power

2222

CPA – theoretical phasor diagrams CPA – theoretical phasor diagrams

2323

CPA feature fieldCPA feature field

CPA is computed as the length of the cumulative phasor vector, CPA is computed as the length of the cumulative phasor vector, divided by the sum of the power for each pulse.divided by the sum of the power for each pulse.

CPA is computed at a single gate.CPA is computed at a single gate.

It is a normalized value, ranging from 0 to 1.It is a normalized value, ranging from 0 to 1.

In clutter, CPA is typically above 0.9In clutter, CPA is typically above 0.9..

In weather, CPA is often close to 0In weather, CPA is often close to 0, but increases in weather with a , but increases in weather with a velocity close to 0 and a narrow spectrum width.velocity close to 0 and a narrow spectrum width.

In noise, CPA is typically less than 0.05In noise, CPA is typically less than 0.05..

CPA was originally developed as a quality control field for clutter CPA was originally developed as a quality control field for clutter targets used for refractivity measurements.targets used for refractivity measurements.

2424

CPA feature fieldCPA feature field

DBZ CPA

Example of CPA – Denver Front Range NEXRAD - KFTG

2525

Combining TDBZ, SPIN and CPACombining TDBZ, SPIN and CPA

The individual The individual featurefeature fields, TDBZ, SPIN and CPA, are fields, TDBZ, SPIN and CPA, are combined into a single combined into a single interestinterest field using fuzzy logic. field using fuzzy logic.

First, each First, each featurefeature field is converted into an field is converted into an interestinterest field, field, using a membership transfer function.using a membership transfer function.

Interest fields have a range from 0.0 to 1.0.Interest fields have a range from 0.0 to 1.0.

The The interestinterest fields are assigned a fields are assigned a weightweight..

The The combined interestcombined interest field is computed as a field is computed as a weighted weighted meanmean of the individual interest fields. of the individual interest fields.

2626

Steps in computing the single-pol CMDSteps in computing the single-pol CMD

Step 1:Step 1:

– Compute TDBZ and TDBZ-interestCompute TDBZ and TDBZ-interest

– Compute SPIN and SPIN-interestCompute SPIN and SPIN-interest

– Compute CPA and CPA-interestCompute CPA and CPA-interest

Step 2:Step 2:

– Compute Texture-interest = maximum of (TDBZ-interest, SPIN-interest)Compute Texture-interest = maximum of (TDBZ-interest, SPIN-interest)

Step 3:Step 3:

– Compute CMD valueCompute CMD value

= fuzzy combination of CPA-interest and Texture-interest= fuzzy combination of CPA-interest and Texture-interest

Compute CMD flag: true if CMD >= 0.5, false if CMD < 0.5Compute CMD flag: true if CMD >= 0.5, false if CMD < 0.5

2727

Membership functions for single-pol CMDMembership functions for single-pol CMD

-> (15,0)

-> (30,1)

2828

Membership function combinationMembership function combination as used in single-pol CMD as used in single-pol CMD

-> (15,0)

-> (30,1)

Weight=1.01

Weight=1.0

2929

Creating combined interest field - CMDCreating combined interest field - CMD

TDBZ SPIN

CPA CMD

3030

Logic for setting the clutter flagLogic for setting the clutter flag

1.SNR> 3dB?

3.CMD> 0.5?

3.SetFlag.

3131

EXAMPLE OF APPLYING CMDEXAMPLE OF APPLYING CMD

3232

KFTG 2006/10/26, 1200 UTCKFTG 2006/10/26, 1200 UTC

Reflectivity

3333

VELVELOCITYOCITY, WIDTH, WIDTH

Radial velocity Spectrum width

3434

TDBZ, SPINTDBZ, SPIN

TDBZ SPIN

3535

CPA, CMDCPA, CMD

CPA CMD

3636

Clutter flagClutter flag

CMD flag

3737

Unfiltered reflectivityUnfiltered reflectivity

3838

Filtered reflectivityFiltered reflectivity

3939

DIAGNOSING ERRORSDIAGNOSING ERRORS

4040

THERE ARE 3 MAIN ERROR TYPESTHERE ARE 3 MAIN ERROR TYPES

1.1. FALSE DETECTIONSFALSE DETECTIONS: the algorithm detects clutter incorrectly, so : the algorithm detects clutter incorrectly, so

that the filter is applied excessively. This is particularly that the filter is applied excessively. This is particularly

problematic when it occurs in the region of 0 velocity, since the problematic when it occurs in the region of 0 velocity, since the

filter cannot distinguish between clutter and weather.filter cannot distinguish between clutter and weather.

2.2. MISSED DETECTIONSMISSED DETECTIONS: the algorithm fails to identify clutter, and : the algorithm fails to identify clutter, and

the filter is therefore not applied where it should be.the filter is therefore not applied where it should be.

3.3. FILTER FAILUREFILTER FAILURE:: the CMD algorithm works correctly, but the the CMD algorithm works correctly, but the

filter fails to work effectively.filter fails to work effectively.

4141

ERROR TYPE 1:ERROR TYPE 1:

FALSE DETECTIONSFALSE DETECTIONS

This is the most common error type.This is the most common error type.

4242

Example 1 - Example 1 - Filtered DBZFiltered DBZ

Filtered reflectivity using CMD Filtered everywhere

4343

Example 1 – Example 1 – VELVELOCITY and OCITY and WIDTHWIDTH

Radial velocity Spectrum width

4444

Example 2 : Filtered DBZExample 2 : Filtered DBZ

Filtered using CMD Filtered everywhere

Weather powerremoved

4545

Example 2 : VEL, filtered VELExample 2 : VEL, filtered VEL

Velocity Filtered velocity

4646

Example 3 : KFTG 2006/10/10, 1000 UTCExample 3 : KFTG 2006/10/10, 1000 UTC

Filtered using CMD Filtered everywhere

Weather powerremoved

4747

Example 3 : VEL, filtered VELExample 3 : VEL, filtered VEL

Velocity Filtered velocity

5050

Example Example 44 : : Operational NEXRADOperational NEXRAD

Filtered reflectivity Filtered velocity

5151

ERROR TYPE 2:ERROR TYPE 2:

MISSED DETECTIONSMISSED DETECTIONS

It was found that in some clutter regions, CPA It was found that in some clutter regions, CPA values can be lower than expected.values can be lower than expected.

5252

KEMX reflectivity 2009/04/22 21:22 UTCKEMX reflectivity 2009/04/22 21:22 UTCAlso showing counties, interstates and US highwaysAlso showing counties, interstates and US highways

We will investigate the region indicated by the ellipse

5353

UUnfiltered reflectivitynfiltered reflectivity

5454

FFiltered reflectivityiltered reflectivity

Ellipses highlightmissed detections

5656

Region 1 – CPARegion 1 – CPA

This region has CPA values with considerable variability, with some values in the clutter region being as low as 0.2.

5757

Characteristics of the CPA fieldCharacteristics of the CPA field

It was noted that in some of the clutter regions, there are a It was noted that in some of the clutter regions, there are a

considerable number of gates with low CPA values. The CPA considerable number of gates with low CPA values. The CPA

values can vary from high to low in adjacent gates.values can vary from high to low in adjacent gates.

Examining the phase time series for these gates shows that Examining the phase time series for these gates shows that

the phase can change significantly over a small part of the the phase can change significantly over a small part of the

time series.time series.

It is probable that 2 separate targets are illuminated during It is probable that 2 separate targets are illuminated during

the dwell.the dwell.

This phase change reduces CPA.This phase change reduces CPA.

However, for the remainder of the time series the phase is However, for the remainder of the time series the phase is

stable.stable.

5858

Example - spectrum of clutter point with low CPAExample - spectrum of clutter point with low CPACPA = 0.19CPA = 0.19

A-scopeX-axis: range (km)

Red – unfiltered spectrumX-axis: samples

Pink – filtered spectrum

Power time seriesX-axis: time

Phase time seriesX-axis: time

Pulse-to-pulse phaseDifference time seriesX-axis: time

Magenta line shows range

Change in phase for partof the time series leads to low CPA value

5959

ERROR TYPE 3:ERROR TYPE 3:FILTER DOES NOT WORK EFFECTIVELYFILTER DOES NOT WORK EFFECTIVELY

The clutter filter can have problems dealing with The clutter filter can have problems dealing with multi-mode spectra.multi-mode spectra.

6060

Traffic clutter spectraTraffic clutter spectra

It appears that some of the missed detections and poor It appears that some of the missed detections and poor

clutter filter performance are caused by traffic echoes.clutter filter performance are caused by traffic echoes.

The following slides show spectra from normal clutter The following slides show spectra from normal clutter

echoes and suspected traffic echoes.echoes and suspected traffic echoes.

The traffic echoes exhibit multi-modal spectra. This makes The traffic echoes exhibit multi-modal spectra. This makes

them both difficult to detect as clutter, and difficult to filter them both difficult to detect as clutter, and difficult to filter

with the current adaptive filters.with the current adaptive filters.

6161

Unfiltered reflectivityUnfiltered reflectivity

Note interstate 10 – shown in

bold – traversing this

area,and smaller

roads

6262

FFiltered reflectivity showing gates atiltered reflectivity showing gates atwhich CMD and the clutter filter failedwhich CMD and the clutter filter failed

Ellipse high-lightsgates for whichCMD and/or theclutter filterfailed

6363

Normal-propagation clutter signatureNormal-propagation clutter signatureCPA = 0.88CPA = 0.88

A-scopeX-axis: range (km)

Red – unfiltered spectrumX-axis: samples

Pink – filtered spectrum

Power time seriesX-axis: time

Phase time seriesX-axis: time

Pulse-to-pulse phasedifference time seriesX-axis: time

Magenta line shows range

6464

Spectrum of suspected traffic targetsSpectrum of suspected traffic targetsCPA = 0.17CPA = 0.17

A-scopeX-axis: range (km)

Red – unfiltered spectrumX-axis: samples

Pink – filtered spectrum

Power time seriesX-axis: time

Phase time seriesX-axis: time

Pulse-to-pulse phasedifference time seriesX-axis: time

Magenta line shows range

Multi-modal spectrum

6565

Spectrum of suspected traffic targetsSpectrum of suspected traffic targetsCPA = 0.30CPA = 0.30

A-scopeX-axis: range (km)

Red – unfiltered spectrumX-axis: samples

Pink – filtered spectrum

Power time seriesX-axis: time

Phase time seriesX-axis: time

Pulse-to-pulse phasedifference time seriesX-axis: time

Magenta line shows range

Multi-modal spectrum

6666

THANK YOUTHANK YOU