Experience you can trust.

NTG Estimation for California IOUs’ 2004-2005 Upstream Lighting Program

CALMAC MeetingPacific Energy CenterJuly 18, 2007Tami Rasmussen, KEMA Inc.

2

Organization of Presentation

Program characteristics NTG assessment objectives Methods considered

– Overview of each method– Caveats– Whether method was used/if not, under what

contexts method may be useful– Keys to success for method used

3

Program Characteristics

Upstream program– lighting retailers and manufacturers receive the

incentives (point-of-sale or buydown)– Consumers may or may not be aware that the price

of the product they are buying has been reduced by the program

Prior California IOU lighting programs have created broad and cumulative market effects

Prior and current programs have influenced the market beyond the specific products rebated

4

NTG Assessment Objectives

Prior Upstream Lighting program evaluations focused on characterizing the market and establishing linkages between program interventions and market change– Attribution was assessed qualitatively

2004-2005 study required an explicit estimate of NTG– Estimate must include only free-ridership – spillover

may be estimated separately but must be excluded from final NTG estimate

5

NTG Methods Considered – Net Effects

Overview of method– Net effects = Difference between baseline sales and total

CFL sales– Baseline sales = sales that would have occurred in

absence of the program (typically estimated on a per capita basis for a similar state or region that does not have a program)

– Total CFL sales – often difficult to estimate– NTGR = ratio of program sales to net effects– This method has been used recently in the Northeast and

Wisconsin

6

Net Effects Method

Caveats:– Difficult if not impossible to reliably estimate total CFL sales for

California– Baseline sales estimates for CA will be overstated (net

effects/NTGR will be understated) because any region or state without a program will have been influenced by CA’s programs

– Method will implicitly capture cumulative effects of prior CA programs (reflected in total CFL sales) leading to overstatement of net effects/NTGR

– Results include both free-ridership and spillover

7

Net Effects Method

Method may be useful:– In regions where programs are small-scale, such that

the national market is not effected (so baseline estimates are not biased)

– Where total CFL sales may be reliably estimated– During the first year or two of a program, where

cumulative effects are not a significant issue– In contexts where it is acceptable to use a NTGR

that includes both free-ridership and spillover

8

NTG Methods Considered – Regional Regressions Overview of method:

– Model CFL sales (or market share/penetration) for various states/regions across the U.S. including California

– Use the parameter estimates to estimate CFL sales in California in absence of the program

– Used in the Northeast to estimate program attribution of ENERGY STAR appliance rebates

– Incorporates both free-ridership and spillover

9

Regional Regressions

Caveats:– Significant CFL data quality and availability issues

across the U.S.– California programs have influenced CFL

sales/market shares across the U.S. leading to an understatement of the program’s influence in California (like the net effects method)

– Not enough observations (obs=states/regions) since few states/regions collect and/or estimate CFL sales

10

Regional Regressions

Method may be useful:– For CFLs if CFL sales data become more available

across the U.S.– For other products

11

NTG Methods Considered – Participant Self-Report Overview of method:

– Surveying participants (retailers and manufacturers in this case) regarding the influence of the program on their behavior

– May address free-ridership and spillover separately– Used for NYSERDA Upstream Motors program

12

Participant Self-Report

Caveats:– A handful of lighting suppliers handle the majority of

CFL sales in the state– Key decision-makers are often hard to contact and

may be unwilling to provide information they consider proprietary

13

Participant Self-Report – Selected Method Algorithm used for 1 of 3 CF product categories

A B C D E F G

Retailer category

% of Total Program Low-Wattage CFLs

Lighting Market Actor

# Respondents

% of Program Low-Wattage CFL

Incentives Paid to Respondents

Free Ridership Estimate

Combined Sales-Weighted Free

Ridership Estimate

Manufacturers 1 55% aRetailers 1 73% b

Manufacturers 3 72% cRetailers 1 10% d

Manufacturers 9 70% eRetailers 7 34% f

Manufacturers 3 70% gRetailers 1 <1% hManufacturers 5 94% iRetailers 0 na

Manufacturers 3 27% jRetailers 9 13% k

Total 100% 69%avg of col G

(weighted by col B)

avg of a and b (weighted by col E)

avg of c and d (weighted by col E)

avg of e and f (weighted by col E)

avg of g and h (weighted by col E)

i

avg of j and k (weighted by col E)

General Merchandise/Big Box 21%

Large Home Improvement 9%

Grocery 43%

Drug 15%

Discount 6%

Small Hardware 4%

14

Participant Self-Report – Selected Method

Algorithm used to combine results

A B C D

CF Product Category

% of Total Program CF

Products

% of Program CF Product

Incentives Paid to Respondents

Sales-Weighted Free Ridership

EstimateLow-Wattage CFLs 97% 69% aSpecialty CFLs 1% 41% bCF Fixtures and Torchieres 2% 27% c

Total 100% 68%avg of col D

(weighted by col B)

15

Participant Self-Report

Keys to success:– To set up analysis framework:

Anticipate how program influence will likely vary by CF product and market actor type

Obtain detailed program tracking data– To ensure representative sample:

Obtain current contact information for participating suppliers Have program managers encourage participants to respond

to evaluators Identify and focus on “must-have” interviews (based on % of

rebates)

16

Participant Self-Report

Keys to success, continued:– To address possible bias:

Sought FR estimates from both manufacturers and retailers (mostly buydown – retailers did not receive incentives)

Assessed consistency of FR estimates across and within various categories

Experience you can trust.

End sheet

Thank you for your attention.