detection of anthropogenic signals that are below thermal noise power

Detection of Anthropogenic Signals that are Below Thermal Noise Power

Roger De Roo

IEEE SEM Fall Conference

2012 Nov 14

Outline

• Motivation– Physics of passive remote sensing

– Monitoring the soil moisture with microwave radiometry

• Radio Frequency Interference: a major problem– Summary of RFI detection approaches

– Kurtosis algorithm development

• Conclusions

• Thanks to: Chris Ruf, U-M; Joel Johnson, OSU; Jeff Piepmeier, NASA Goddard; Sid Misra, JPL

Soil Moisture: who cares?• Soil Moisture

regulates plant transpiration

• Transpiration determines humidity

• Humidity gives rise to clouds

• No widespread measurements of soil moisture currently

What’s so great about Microwave Remote Sensing?

Radar (“Active”)

Radiometry (“Passive”)

Tx

Rx

Rx

Long wavelengths (3mm to 30cm) don’t scatter off of objects the size ofcloud droplets -- microwaves see through clouds

•Very high spatial resolution•Power hungry: expensive•Sensitive to geometry of water: eg. Movement of trees causes big signal changes

•Poor spatial resolution•Low power requirements•Insensitive to geometry of water

Planck Blackbody Radiation

1 GHz 1 THz 1 PHz frequency wavelength 0.3 m 0.3 mm 0.3 um

3K outer space

30K

300K room temp

3000K red hot

6000K white hot the Sun

Microwave Characteristics of the Atmosphere

from LeVine, Wilheit,Murphy and Swift, 1989

Products by frequency

from LeVine, Wilheit,Murphy and Swift, 1989

Also: -Sea surface salinity at 1.4GHz

-Vegetation moisture content at 1.4 and 6 GHz

-Vegetation temperature at 18 – 90 GHz

Microwave Brightness and Moisture• Water molecules have large electric dipole, unlike rest of nature

H- O + H

Liquid water molecules will orient itself with passing electromagnetic waves, slowing the wave downThe molecule can keep up with the wave until 9 GHz(index of refraction: n = 9 at 1GHz, but n = 2 at 100 GHz)

ε' = n2

Microwave Brightness and Moisture

• An interface w/ high contrast of index of refraction leads to reflection

• Dry soils appear warm, while wet soils appear cold, at the same temp.

Space 2.7K

Dry Soil~300K

Sensor Space 2.7K

Wet Soil~300K

Sensor

TransparentAtmosphere

TransparentAtmosphere

Low Contrast at Interface High Contrast at Interface

Example Brightness Image from Space

NASDA

Sensitivity of Radiobrightness to Soil Moisture Under a Vegetation Canopy

0.0

1.0

2.0

3.0

4.0

0 5 10 15 20 25Wavelength (cm)

Sen

siti

vity

T

B /

V

SM

(K

/%)

BARE

VEGETATIONWATER CONTENT (kg/m2)

1

2

4

0

SSM/I AMSR SMAP

Courtesy of P. O’Neill19 GHz 6.9 GHz 1.4 GHz

University of Michigan Radiometers

L-band1.4 GHz = 21 cm

satellites:SMOS Nov ‘09Aquarius Jun ‘11SMAP ‘14

19 GHz 37 GHz = 1.6 cm = 0.8 cm satellites: SSM/I etc. ‘87 to present

C-band6.7 GHz = 4.5 cm

satellites:AMSR-E ’02-’11

Antenna sizeis proportionalto wavelength

The Tundra Landscape

Diurnal Brightness Measurements

Brightness of Tundra and Shrubs

Trouble with the 1.4GHz Radiometer

Both of these ranges appear plausible

Potentially Interfering Radars:Cobra Dane

Peak Transmit Power 16.8 MWTransmit Frequency 1.215-1.375GHz

Raytheon

Surrounded by Interfering Radars?

FPS-108Cobra Dane

FPS-117

FPS-124

Observation site:Toolik Lake

ITT, ‘05

AMSR-E Interference at 6.9GHz

Li et al., ’04

If it is not purple, we cannot use the data from that locationIf it is purple, the data from that location might be OK…or not

Traditional Radiometer Technology

• Use square law detector for signal power

k1 v2 v

21

Effect of Finite Samples: RFI-free signals

• Variance of voltage waveform contains brightness power:

PIF = kBTSYSB G = <v2(t)>/Z

TSYS = TB + TREC

• Finite number of samples results in a measurement variance:

σ = NEΔT = TSYS / √N N=Bτ

• RFI always biases measurements of TB upwards

– Averaging preserves the bias – thus, not a solution

– We’d like to see RFI at near the NEΔT power level

Approaches to Detecting RFI

1. Time domain – look for pulses

2. Frequency domain – look for carrier frequencies

3. Amplitude domain – look for non-thermal distribution

Gaussian pdf Non-Gaussian pdfSinusoidal waveformThermal waveform

Digital Radiometry

Digital radiometers use fast analog-to-digital converters to measure the voltage waveform

Power is determined by finding the variance (2nd moment) of the quantized data

Processing capability allows for implementation of one or more RFI mitigation strategies

Literature Search

• What has already been done on this problem, or related problems?

• For RFI mitigation, nothing in the amplitude domain. Some in time-domain and some in frequency domain.

• However, testing for normality of a distribution does have a rich literature. Lotsa ways to do it, and it is known how well they work.

Is it Normal?

Statistical moments:

• 0th… event count

• 1st… Mean

• 2nd …Variance

• 3rd … Skew

• 4th … Kurtosis

dvvpvnn )(

Skew

• Measures how asymmetric a distribution is

• Normal distribution has zero skew

• So does RFI

Sources of Skew

• No skew for o pulsed sinusoid (ps), o Amplitude Modulation (AM), or o Frequency Modulation (FM)

• Skew possible, but unlikely, with Phase Modulation (PM)

AM FM PM

Kurtosis

• Kurtosis measure “peakedness” of a distribution

• Normal distribution has kurtosis = 3

• RFI can have any kurtosis

29

Definition of Kurtosis

• Desired radiometric (science) signals generated by thermal noise– Gaussian (bell-curve) probability distribution function (PDF)

• RFI is man-made – PDFs will be non-Gaussian in general

• Underlying Statistics– all higher-order moments of a Gaussian are uniquely determined by its

lowest two moments

– for example, the kurtosis equals 3 for a Gaussian v(t)

where v(t) is the zero-mean pre-detected radiometer output voltage

22

4

)(

)(or

tv

tvR

Technology Approach • Digitized IF waveforms lend themselves to moment

estimation• Use moment ratios to test for presence of RFI• 1st moment, 1, is a DC offset• 2nd central moment, m2, is power – the measurement

objective• Odd central moments are all zero• The lowest moment for RFI detection is the 4th

central moment m4

1 1

Nsv i

i1

N s

m2 1

Nsv i 1 2

i1

N s

m4 1

Nsv i 1 4

i1

N s

ADC vi FPGA

Alternative Technology Approach(explored by Jeff Piepmeier of NASA)

• Use square law detector for signal power

– (traditional radiometer architecture)

• Use a second square law detector for the 4th moment

k1 v2

k2 v4

v

32

v

HeA

kdevp k

k

k

v

2

2

12

2

2!

11

2

1)(

2

2

Expected PDF of thermal noise with variance σ2 with pulsed sinusoidal RFI of amplitude A and duty cycle d: (extension of Rice, 1948)

Probability Densities of signals w/ & w/o RFI

Noise w/ constant power, σ2 RFI w/ Constant Amplitude, A Varying duty cycle, d

=

33

All curves have the same variance:

A radiometer will report all of these signals as the same brightness

Pulsed sinusoid to noise ratio:S = dA2/2σ2

34

Effect of Finite Samples: RFI-free signals

• Exact kurtosis pdf unknown

• Kurtosis pdf is skewed– less so as N∞

– kurtosis pdf is essentially Gaussian N>50k

• Mean of R=3(N-1)/(N+1)

• Variance of R 24/N as N∞

35

Detection Concepts

Rth,a

FAR

PD

36

False Alarm Rate and Probability of Detection of Pulsed Sinusoidal RFI

• For RFI power level at brightness temperature equivalent to 2NET, detection threshold can be set to give:– 90% probability of

detection– 3% false alarm rate

• 0.1% duty cycle case corresponds to ARSR-1 operating mode

• Higher duty cycle reduces detectability

d is radar duty cycle

PD=1-FAR

z is a FAR parameter

z=3 FAR=0.25%

z=2 FAR= 5%

z=1 FAR=30%

For large N,

min detectable RFI

TPS ~ N-¼

NEDT ~ N-½

37

Minimum detectable RFI

“Blind Spot” at 50% duty cycle, and solution

CFAR R4 (kurtosis)

CFAR R6

13%blindspot

61%blindspot

50%blindspot

R6=+0.085R6= -0.085

R6=+0.085

R 4=+0.0155

R4=-0.0155

0155.01

0849.01

10024

100720

k

k

N=100kSaThreshold at 1σ (30% FAR)

Laboratory Experiments

• Check assumptions about radiometer operation

• RFI is prescribed to conform to our theory’s assumptions

VariableAttenuator

ArbitraryWaveformGenerator

DetMit Rcvr

CNCS ControlADD Data AcqADD

CNCS RF head

Laboratory Experiment Results

• Kurtosis R= m4/m22

• In the absence of RFI, R=3• For CW RFI (eg. Carriers) R<3• For short duty cycle RFI (eg. RADAR), R>3• But R=3 also for 50% duty cycle

41

Kurtosis of RFI free signals• Kurtosis False Alarm

Rate confirmed with

simulations

• PALS-ADD data:

a minute of apparently

RFI free data

• RFI free

assumption

supported

by kurtosis FAR

predictions

z = |Rth-3| / σR0

Theoretical FAR=1-erf( z /√2)

RFI flags from clean PALS-ADD data

42

Kurtosis of the RFI-free: digitization effects• Effects considered (and are very small):

– Kurtosis pdf itself is Gaussian for 50k+ independent samples

– Clipping of signals by finite Analog to Digital Converter (ADC) dynamic range: 4 bits is enough; 3 bits, maybe

– Digitization (ADC bin size) effects are negligible.

– ADC null offsets can be corrected with 1st and 3rd moments in addition to 2nd and 4th moments needed for kurtosis algorithm.

• Effects not yet considered:

– Integral Nonlinearity and Differential Nonlinearity of Analog to Digital Converters: likely is small effect because Flash ADC typically have small INL and DNL

– Correlated data: we are still applying the theoretical tools to analyze the effects of sampling above the Nyquist rate on the kurtosis calculation.

43

Kurtosis of RFI-free: effects of digitization

• Digitization reduces kurtosis• Bin size effects decrease as noise amplitude increases• Threshold locations not critical for s>3/4• Saturation of ADC at high noise amplitude distorts kurtosis

Number of ADC bins

s = 3/4

Field Experiments

• A lot of fun to do!

• Takes lots of people ($$$) to do.

45

Example of RFI detection with Kurtosis (1)

• 1 minute of data from ADD back-end attached to PALS front-end at JPL

• Antenna looking to sky

• Kurtosis thresholds set to trigger 1 false alarm per minute

• Flagged observations: some obvious RFI, some not

seconds

TSY

S (

coun

ts2 )

RFI flags

46

Example of RFI detection with Kurtosis (2)

• Another PALS minute of data; same kurtosis thresholds

• Antenna looking to sky; absorber placed over antenna

• Changes in brightness do not get flagged

seconds

TSY

S (

coun

ts2 )

absorbersky sky

RFI flags

Airborne Campaign Results

Soil Moisture: Active and Passive (SMAP)

NASA environmental satellite

Currently in planning stages

Launch Nov 2014

Kurtosis is the main RFI detection algorithm

49

Conclusions

• Theoretical behavior of the kurtosis statistic as a detector of pulsed sinusoidal RFI has been explored.

– Kurtosis has a blind spot at 50% duty cycle sinusoids

– CW RFI lowers kurtosis below 3

– Low duty cycle sinusoidal pulses raise the kurtosis above 3

– Kurtosis is very sensitive to low duty cycle sinusoid pulses

– Kurtosis is minimally affected by digital receiver properties

– False Alarm Rate of kurtosis algorithm is confirmed

– Minimum detectable RFI is comparable to NEDT in realistic circumstances, may be less than NEDT

– Kurtosis false alarms do not bias the estimate of the brightness

– The kurtosis does not flag gradual changes in brightness.

Thank You!

Backup Slides

ADC offset and non-central moments

• offset in ADC “ground” requires 4 moments • 3rd moment of questionable value• elimination of 3rd moment can

• relax back-end data rate requirements,• allow more subbands, and/or • permit shorter integration periods

s

s

s

s

N

ii

s

N

ii

s

N

ii

s

N

ii

s

vN

vN

vN

vN

1

44

1

33

1

22

11

1

1

1

1

vi=+1

vi= 0

vi= -1

vi= -2

Tanana River Breakup at Nenana

Guess the moment of breakup!Tickets cost $2.50 eachTypical Jackpot: $300,000+

www.nenanaakiceclassic.com

Spring Breakup

105

110

115

120

125

130

135

140

145

1900 1920 1940 1960 1980 2000 2020

Year

Da

y o

f Y

ea

r

Observed Global Temperature Trends

IPCC ‘01

Projected Global Temperature Trends

IPCC ‘01

2071-2100 temperatures relative to 1961-1990.Special Report on Emissions Scenarios Storyline B2 (middle of the road warming).

Carbon Stocks by Biome

IPCC ‘01

Atmospheric stock is about 750PgC

050

100150200250300350400450500

Tropic

al F

ores

ts

Tempe

rate

For

ests

Borea

l For

ests

Tropic

al S

avan

nas

Tempe

rate

Gra

ssla

nds

Deser

ts

Tundr

a

Cropla

nds

Wet

land

s

To

tal C

arb

on

(P

gC

)

plants

soils

Permafrost extent

Global Terrestrial Network for Permafrost

20m Borehole Temperature Trends in AK

Hinzman et al 2005

Permafrost structure

NSIDC

Active Layer Depth Trends

Circumpolar Active Layer Monitoring Network

Year

Max

imum

Act

ive

Lay

er D

epth

(cm

)

Strategy for Estimating Temperature and Moisture Profiles

Atmospheric Model

Weather & downwelling radiance

SVAT Model

Temperature & Moisture Profiles

Radiobrightness Model

Satellite L-band Radiometer

Tb (model)

Tb(observed)

Assimilate Tb(observed) - Tb(model)

320

310

300

290

280Tem

pera

tu

re (

K)

198196194192190188186184182

SGP'97 (TIR) LSP/R (Canopy)

LSP-SGP = -0.28 K

Variance = 3.28 K2

320

310

300

290

280Tem

pera

tu

re (

K)

198196194192190188186184182

SGP'97 LSP/R

Depth = 3cm

Mean Diff = 0.27 K

Variance = 2.41 K2

320

310

300

290

280Tem

pera

tu

re (

K)

198196194192190188186184182

Julian Day from Jan. 1(CST)

SGP'97 LSP/R

Depth = 10cm

(July 1) (July 17)

Mean Diff = 0.03 K

Variance = 0.75 K2

Calibrated LSP/R model of Prairie Grassland

Judge et al. 1999

Correlated Noise Calibration System

Ruf and Li, ‘03

Low Noise Amplifier (LNA) input is a matched source of sub-ambient noise… it is an electronic device which, at RF, looks like it is at LN2 temperatures

CNCS concept:Onto this very low noise background, couple in some much stronger noise. This much stronger noise can be generated in a COTS Arbitrary Waveform Generator

CNCS extension:This same concept can be used to create known weak RFI

From AWG To Radiometer

Detection and Mitigation Testbed

C-band RFI Detection and Mitigation TestbedVariable

Bandpass

Ambient andSub-ambient Calibration

(from CNCS Design)

CNCS Design:Artificial RFI Generator

AWG SpectrumAnalyzer

Digital Scope

PersonalComputer

VariableCtr Freq

FlashMemory

TMRS-3 Design:Digital Radiometer

Vin

GND

Vref

B1

B8

Sign

ENB

A/D Converter

FPGA

uC

ConclusionsMicrowave Radiometry has been demonstrated to have high sensitivity to surface soil moisture.

Hydrologic models can use this measurement to constrain the evolution of profiles of temperature and moisture.

This technique should work well for the low vegetation content of the Arctic.

Understanding the evolution of the active layer will help us understand the threat of carbon release from Arctic soils in response to climate change.

Microwave observations are very susceptible to interference.

RFI mitigation for microwave radiometry is an emerging research area at Michigan

Microwave Brightness and Moisture• Water molecules have large electric dipole, unlike rest of nature

• An interface w/ high contrast of index of refraction leads to reflection

• Dry soils appear warm, while wet soils appear cold, at the same temp.

H- O + H

Space 2.7K

Dry Soil~300K

Sensor Space 2.7K

Wet Soil~300K

Sensor

Liquid water molecules will orient itself with passing electromagnetic waves, slowing the wave downThe molecule can keep up with the wave until 9 GHz(index of refraction: n = 9 at 1GHz, but n = 2 at 100 GHz)

TransparentAtmosphere

TransparentAtmosphere

Low Contrast at Interface High Contrast at Interface