friedman's supersmoother : special notes on span

NNational ational TTransfer ransfer AAccountsccounts

Friedman's Friedman's SuperSmoother: Special SuperSmoother: Special

Notes on SpanNotes on SpanIVAN MEJIA GUEVARAIVAN MEJIA GUEVARA

CONSEJO NACIONAL DE POBLACIONCONSEJO NACIONAL DE POBLACIONCEPAL/IDRC PROJECTCEPAL/IDRC PROJECT

Honolulu, June 5 2008Honolulu, June 5 2008

National Transfer Accounts

Seoul’s Meeting: SmoothingSeoul’s Meeting: Smoothing►Some problems identified in the workshop:Some problems identified in the workshop:

Young ages.Young ages. Many profiles are inherently not Many profiles are inherently not smooth at early ages smooth at early ages

Old ages.Old ages. The age profile estimates at old The age profile estimates at old ages are often based on relatively few ages are often based on relatively few observationsobservations

Over smoothing.Over smoothing. There is a natural tendency There is a natural tendency to over-smooth the data. The danger is that to over-smooth the data. The danger is that real and important fluctuations may be real and important fluctuations may be concealed. Smoothing too little is better than concealed. Smoothing too little is better than smoothing out real changes. smoothing out real changes.

National Transfer Accounts

Seoul’s Meeting: SmoothingSeoul’s Meeting: Smoothing►Some problems identified in the workshop:Some problems identified in the workshop:

National Transfer Accounts

Seoul’s MeetingSeoul’s Meeting► For most, but not all purposes, For most, but not all purposes,

SUPSMU with a relatively narrow SUPSMU with a relatively narrow bandwidth (0.05 or less) appears to be bandwidth (0.05 or less) appears to be a relatively reliable methoda relatively reliable method

National Transfer Accounts

Alternative Method using Alternative Method using SUPSMUSUPSMU► This approach attempts to deal with the This approach attempts to deal with the

problem of selecting the most problem of selecting the most appropriate bandwidth (span)appropriate bandwidth (span)

► The method might also be effective to The method might also be effective to deal with the problem of over smoothingdeal with the problem of over smoothing

►The method proposed relies on:The method proposed relies on: The use of statistical criteriaThe use of statistical criteria Semi-automatic selection of bandwidthSemi-automatic selection of bandwidth

National Transfer Accounts

Method DescriptionMethod Description1.1. Definition of a metric must be defined Definition of a metric must be defined

in order to measure the difference in order to measure the difference between actual (unsmoothed) values between actual (unsmoothed) values ((AAtt) and output (smoothed) values () and output (smoothed) values (FFtt))

2.2. Specification of span optionsSpecification of span options3.3. Compute the metric(s) defined in 1Compute the metric(s) defined in 14.4. Apply the Majority Voting Scheme Apply the Majority Voting Scheme

(MVS) as a selection criterion(MVS) as a selection criterion

National Transfer Accounts

1. Definition of a metric1. Definition of a metric

n

ttt FA

nMSEErrorSquareMean

1

21)(

n

ttt FA

nMAEErrorAbsoluteMean

1

1)(

n

t t

tt

AFA

nMAPEErrorPercentageAverageMean

1

1)(

National Transfer Accounts

2. Specification of span 2. Specification of span OptionsOptions Consider several span options from a Consider several span options from a

list previously specified by the user. list previously specified by the user. For example, the list {0.035, 0.04, For example, the list {0.035, 0.04, 0.045, 0.05, 0.055, 0.06, 0.065, 0.07, 0.045, 0.05, 0.055, 0.06, 0.065, 0.07, 0.075, 0.08, 0.09, 0.1, ‘cv’}0.075, 0.08, 0.09, 0.1, ‘cv’}

Users are free to choose the range Users are free to choose the range that they consider more appropriate that they consider more appropriate for their particular situationfor their particular situation

National Transfer Accounts

3. Compute the metric(s) 3. Compute the metric(s) defined in 1.defined in 1.► Compute the MSE, MAE and MAPE for Compute the MSE, MAE and MAPE for

every span in the list specified in 2.every span in the list specified in 2.

National Transfer Accounts

4. Use of MVS as a selection 4. Use of MVS as a selection criterioncriterion

National Transfer Accounts

ImplementationImplementationR code: R code: ******SUPSMU*************SUPSMU*******aloha<-read.csv("earnings.csv", header=T)aloha<-read.csv("earnings.csv", header=T)t1<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.035")t1<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.035")t2<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.04")t2<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.04")t3<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.045")t3<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.045")t4<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.05")t4<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.05")t5<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.055")t5<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.055")t6<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.06")t6<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.06")t7<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.065")t7<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.065")t8<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.07")t8<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.07")t9<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.075")t9<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.075")t10<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.08")t10<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.08")t11<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.09")t11<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.09")t12<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.1")t12<-supsmu(aloha$age, aloha$income, aloha$sample, span = "0.1")t13<-supsmu(aloha$age, aloha$income, aloha$sample, span = "cv")t13<-supsmu(aloha$age, aloha$income, aloha$sample, span = "cv")write.csv(c(t1,t2,t3,t4,t5,t6,t7,t8, t9,t10,t11,t12,t13), "smoothed_earnings")write.csv(c(t1,t2,t3,t4,t5,t6,t7,t8, t9,t10,t11,t12,t13), "smoothed_earnings")

National Transfer Accounts

ImplementationImplementationExcel Template: Excel Template:

National Transfer Accounts

DiscussionDiscussion► Ad hoc is used to break ties where MVS Ad hoc is used to break ties where MVS

cannot be appliedcannot be applied► This method could be effective in practice This method could be effective in practice

for some profilesfor some profiles► A lot of time is saved by (semi) A lot of time is saved by (semi)

automatically selecting the span in this wayautomatically selecting the span in this way► This method might be no appropriate when This method might be no appropriate when

the researcher cares more about some parts the researcher cares more about some parts of the age profile than others or when of the age profile than others or when profiles present a lot of sharp elbows profiles present a lot of sharp elbows

National Transfer Accounts

Example 1Example 1

National Transfer Accounts

Example 2Example 2

National Transfer Accounts

WikiWiki http://www.schemearts.com/proj/nta/web/nta/show/Documents/Smoothing/Friedman's%20SuperSmoother/Special%20Notes%20on%20Span