measurement and verification report of opower energy ... · pdf filemeasurement and...

Connexus Energy Ramsey, MN

Measurement and Verification Report of

OPower Energy Efficiency Pilot Program

Madison, WI

. Minneapolis, MN

. Marietta, OH

. Indianapolis, IN

. Sioux Falls, SD

Contact: Chris Ivanov 1532 W. Broadway Madison, WI 53713

Direct: 608-268-3516 Fax: 608-222-9378

Email: [email protected] Web Site: www.powersystem.org

July 28, 2010

Confidential, Copyrighted, and Proprietary

This document contains information confidential to Connexus Energy (Connexus or

Cooperative) and Power System Engineering, Inc. (PSE). Unauthorized reproduction or

dissemination of this confidential information is strictly prohibited.

Copyright 2010 Power System Engineering, Inc.

This document includes methods, designs, and specifications that are proprietary to Power

System Engineering, Inc. Reproduction or use of any proprietary methods, designs, or

specifications in whole or in part is strictly prohibited without the prior written approval of

Power System Engineering, Inc.

NEITHER POWER SYSTEM ENGINEERING, INC. NOR CONNEXUS ENERGY

SHALL BE RESPONSIBLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, OR

CONSEQUENTIAL DAMAGES (INCLUDING LEGAL FEES AND COURT COSTS)

ARISING OUT OF OR CONNECTED IN ANY WAY TO THE UNAUTHORIZED USE,

MODIFICATION, OR APPLICATION OF THIS DOCUMENT OR THE

PROPRIETARY INFORMATION, METHODS, AND SPECIFICATIONS SET FORTH

IN THIS DOCUMENT, WHETHER IN WHOLE OR IN PART.

TABLE OF CONTENTS

1.0 Executive Summary ............................................................................................................... 1

2.0 Background of the Program ................................................................................................. 4

3.0 Impact Evaluation Approaches and Results ....................................................................... 6

3.1 True Impact Test Approach and Results ............................................................................ 6

3.2 OLS Model Approach and Results .................................................................................... 8

3.3 Fixed Effects Model Approach and Results .................................................................... 10

4.0 Conclusion ............................................................................................................................ 13

5.0 Appendix ............................................................................................................................... 14

5.1 Thoughts on Further Research ......................................................................................... 14

5.2 Model Results .................................................................................................................. 15

5.3 PSE Background .............................................................................................................. 16

5.4 Authors ............................................................................................................................. 16

1.0 Executive Summary

1.0

Connexus OPower Energy Measurement-Verification Report 1 MN0480806 8/9/10

1.0 Executive Summary

In response to Minnesota’s new state-wide conservation goals, Connexus Energy (Connexus)

partnered with OPower, formerly Positive Energy, to launch a new energy efficiency program.

Power System Engineering, Inc. (PSE) was hired as an independent third-party evaluator for this

pilot program. PSE’s role was to validate the methods and estimates of energy savings attributed

to the program in the first year.

Data was provided to PSE by OPower staff; in particular, Tyler Curtis. This included monthly

billing data, program-specific characteristics, customer demographics, and climatic data. The

time span of the data stretched from January 2007 to January 2010. This time period

encompassed over two years of data before the pilot began (January 2007 through February

2009) and 11 months of data during the pilot (March 2009 through January 2010). The dataset

included over 2.5 million observations encompassing nearly 80,000 member accounts.

PSE examined the sample selection, program design, and data outputs. In evaluating the energy

savings of the program, three measurements were calculated. These included the True Impact

Test, an Ordinary Least Squares econometric model, and a Fixed Effects econometric model.

These evaluation techniques are standard methods of measuring energy efficiency impacts.

The True Impact Test is a non-parametric analysis which examines the change in the differences

of the control and treatment groups from pre-pilot to post-pilot time periods. The advantage of

this method is that it is relatively simple to understand and is a powerful tool in evaluating

program impacts. In algebraic terms, the True Impact Test can be described by the following

equation:

The other two evaluation methods use econometric analysis to estimate a specified model. Each

model calculates a parameter estimate of the impacts of the pilot program on energy usage. The

first model uses an Ordinary Least Squares (OLS) estimation procedure. The second model

1.0


employs a Fixed Effect estimation procedure, which is the same approach that OPower uses in its

models.

The advantage of the OLS model is that it allows the researcher to include customer-specific

information (e.g., income, housing structure, age) into the regression. This enables estimation of

demographic impacts on electricity usage. The advantage of the Fixed Effects model is that it

automatically adjusts for all participant differences in estimating the program impacts. The

downside of the Fixed Effect model is that it cannot include time invariant customer information.

For example, many demographic variables such as sex or race do not vary over time and cannot

be included in the Fixed Effects Model.

The three estimates are presented in the table below. There is stability in results across the three

estimation procedures. The True Impact Test estimates a daily kilowatt hour (kWh) reduction of

0.621 for each participating member of the pilot program during the post-pilot time period.

Similarly, reductions of 0.622 and 0.637 are seen for the OLS and Fixed Effects models,

respectively. Reduction percentages range from 2.05 to 2.10 percent.

Given the stability of the results derived from separate estimation procedures, the large sample

size, and the randomness of sample selection, PSE concludes there are tangible energy savings

resulting from this program. Results are robust with a high degree of confidence attached to

them.

Percentage Reduction

Daily Annual

True Impact Test 0.621 227 2.05%

OLS Model 0.622 227 2.05%

Fixed Effects Model 0.637 232 2.10%

Average: 0.626 229 2.07%

Annual KWH Savings per customer

Estimated Per Customer Savings of OPower Program

1.0


OPower’s fixed effects model estimated a program impact of 2.27%. This is slightly higher than

PSE’s Fixed-Effect model which estimated a reduction of 2.10%. The small disparity in results is

most likely due to the OPower process of continually updating their results as new data becomes

available and using a rolling program start date commencing after the second Home Energy

Report mailing was received by a particular household.

Further research is required to better quantify the long-run impacts of this program. Research

items, such as estimating the impacts of multiple years of treatment and the possible legacy

impacts of treatment after the program is discontinued, would have important ramifications on

cost benefit tests and on optimal program design and cycling. Research on customer

demographics and their impact on program energy savings would also add value. Additionally,

interaction effects with other energy efficiency programs would be of interest. Along these same

lines, determining the ways the pilot participants lowered energy usage would be important

information (e.g., conservation efforts, appliance replacement).

2.0 Background of the Program

2.0


2.0 Background of the Program

In response to Minnesota’s new state-wide conservation goals, Connexus Energy (Connexus)

partnered with OPower to launch a new energy efficiency program. Power System Engineering,

Inc. (PSE) was hired as an independent third-party evaluator for this pilot program. PSE’s role

was to validate the methods and estimates of energy savings attributed to this program.

The program provides residential participants with a Home Energy Report designed to motivate

and educate recipients to improve the energy efficiency of their homes. The program is designed

as a large scale behavioral experiment. Control and treatment groups were randomly selected

out of a total of 80,000 households. Each group consisted of approximately 40,000 households.

Households had to meet the following criteria to be included in one of the groups:

1. Exactly one active electric account per household.

2. Account history dating to January 2007.

3. Valid meter read cycle.

4. Not on a medical rate plan.

5. At least 50 “neighbors.”

The Home Energy Report relies on providing residences with a comparison of energy usage to

their “neighbors,” thus the definition of what constitutes a neighbor is important. OPower

defines the neighbor benchmark by the following characteristics.

Size of home: Neighbors are selected based on similarity in home size. Homes

greater or less than 25 percent of the participant’s home are excluded from the

comparison.

Distance: A radius of 0.5 miles of the targeted household is searched in order to find

100 similar households. At least 50 households must be available within this radius

for it to be included in the pilot.

Heating Fuel: Households are compared with other homes that have the same

heating fuel.

3.0 Impact Evaluation Approaches and

Results

3.0


3.0 Impact Evaluation Approaches and Results

Section 3.0 provides a more detailed look at the three approaches used to quantify electrical

savings resulting from the pilot program. Research results and findings are provided for each

approach. These three approaches do not depend on each other. Thus, they are independent

estimations with only the data as a common element.

The three approaches all reveal similar findings. The pilot program did induce consumers to

lower energy use, provoking annual savings of about 229 kWh per participant. This amounts to

around 9,160 MWh annually saved by the 40,000 participants. The True Impact Test estimated

savings of about 2.05 percent, the OLS Model estimated savings at 2.05 percent, and the Fixed

Effects Model estimated savings of 2.10 percent. Results for the OLS and Fixed Effects Models

were statistically significant at a 99 percent confidence threshold.1

3.1 True Impact Test Approach and Results

The True Impact Test is the most straightforward method of evaluating energy efficiency

programs. The True Impact Test is a non-parametric analysis which examines the change in the

differences of the control and treatment groups from pre-pilot to post-pilot time periods. The

advantage of this method is that it is relatively simple to understand and examine the results and

is a powerful tool in evaluating program impacts. In algebraic terms, the True Impact Test can

be described by the following equation:

1 The True Impact Test is a non-parametric approach to estimating savings and thus is not able

to provide a confidence level based on statistical tests.

Percentage Reduction

Daily Annual

True Impact Test 0.621 227 2.05%

OLS Model 0.622 227 2.05%

Fixed Effects Model 0.637 232 2.10%

Average: 0.626 229 2.07%

Annual KWH Savings per customer

Estimated Per Customer Savings of OPower Program

3.0


To calculate the True Impact Test, PSE took the difference in the average daily use of the

treatment group during the pilot ( ) to the average daily use of the control group during the

pilot ( ) and then subtracted this difference from the difference in the average daily use of

the treatment group before the pilot started ( ) to the average daily use of the control group

before the pilot began ( ). This provides an estimate of the change in the post-pilot treatment

energy use that can be attributed to the pilot itself. Given the large sample size and design of the

program, this estimate is convincing. The graph below further illustrates the finding. The

difference between the treatment and control group significantly widens after the start of the

energy efficiency program.

In an attempt to separate the seasonal differences in the impacts, the seasons were divided into

three categories.

1. Winter (December, January, February, March, April).

2. Summer (July, August, September).

3. Shoulder (May, June, October, November).

30.288

28.622

30.222

27.935

26.500

27.000

27.500

28.000

28.500

29.000

29.500

30.000

30.500

Daily KWH Use per Customer of Test Groups

Control Group Treatment Group

Treatment PeriodPre-Treatment Period

3.0


The table below shows the average daily use per group on an annual, winter, summer, and

shoulder basis. Treatment energy savings in the summer are estimated at 0.647 kWh per day.

This is a reduction of 1.82 percent over the control group’s usage. The winter months show an

impact of 0.677 average kWh reductions per day. The shoulder months have an estimated

reduction in average daily use of 0.517 kWh.

3.2 OLS Model Approach and Results

To further substantiate the True Impact Test results, an Ordinary Least Squares regression was

estimated. This regression incorporated pre-pilot data (January 2007 to February 2009) and the

post-pilot data March 2009 to January 2010) and estimated variables to determine their impact

on average daily electricity use.

Six explanatory variables were included in the OLS Model.

1. Intercept Term (a): This variable measures the expected electricity usage with zero

values of the other variables considered. This would be the expected usage if there was

no pilot, zero heating degree days, and zero cooling degree days.

2. Test Indicator Binary Variable (T): This variable equals “1” if the customer was

selected to be in the treatment group and “0” if the customer was placed in the control

Annual Winter Summer Shoulder

Pre-Treatment Period (Jan. 2007-Feb. 2009)

Control Group 30.288 30.779 35.639 25.473

Treatment Group 30.222 30.760 35.497 25.392

Difference (T-C) -0.066 -0.019 -0.142 -0.081

Treatment Period (March 2009 - January 2010)

Control Group 28.622 29.557 31.184 25.697

Treatment Group 27.935 28.861 30.395 25.100

Difference (T-C) -0.687 -0.696 -0.789 -0.598

Treatment KWH Impact: -0.621 -0.677 -0.647 -0.517

Percent Reduction: -2.05% -2.20% -1.82% -2.03%

True Impact Test of OPower Program

(Daily KWH Use per Customer)

3.0


group. These values are constant across the pre-pilot and post-pilot time frames. This

variable allows the researcher to estimate the inherent differences between the two

groups.

3. Post-Pilot Binary Variable (P): This variable equals “1” if the time period is after the

start of the pilot program (March 2009). It equals “0” if the time period is before the

start of the pilot program (prior to March 2009). This variable allows an estimation of

the impact on average daily use of the different time periods.

4. Test Indicator multiplied by the Post-Pilot (P*T): This variable equals “1” if the

observation is in the treatment group and the month is after the start of the pilot. It

equals “0” if the observation is either in the control group or is in the treatment group

but before the pilot start date. This variable estimates the impact the pilot program

has on average daily use. PSE’s estimate of program savings is taken from the

estimated coefficient on this term.

5. Heating Degree Days (HDD): This variable measures the heating degree days for the

billing month. The parameter estimates the impact that heating degree days have on

average daily usage.

6. Cooling Degree Days (CDD): This variable measures the cooling degree days for the

billing month. The parameter estimates the impact that cooling degree days have on

average daily usage.

The estimated equation takes the following functional form:

Econometric analysis is used to estimate the values of a, b, c, d, e, f in order to minimize the

squared sum of the error term ( ). The parameter estimates (b,c,d,e,f) are interpreted as the

marginal energy use of the variable they are multiplied by. For example, parameter “f” is a

measure of how the average daily use will increase when cooling degree days increase by one.

In the context of this report, the most interesting parameter is “d.” This measures the marginal

impact on average daily use of being in the treatment group during the pilot period. The

parameter estimate for “d” is -0.622, meaning that average daily use is estimated to be reduced

by over half a kWh if the participant is currently being sent Home Energy Reports.

3.0


Estimated Impacts

OLS Model

Daily kWh -0.622

Percent Reduction -2.05%

While the econometric method is more complicated and less easy to comprehend relative to the

True Impact Test, it does have the advantage of being able to test how much confidence policy

makers can have on the estimated impact. One way to do this is to calculate the t-statistic on the

parameter estimate “d.” PSE estimates a t-statistic of 12.40. For context, a t-statistic of 1.645

indicates a 90 percent level of confidence and a t-statistic of 2.326 indicates a 99 percent degree

of confidence. A t-statistic of 12.40 is, therefore, highly significant and offers a great deal of

confidence in the result.

3.3 Fixed Effects Model Approach and Results

To further substantiate the results of the pilot, PSE estimated a Fixed Effects Model. A Fixed

Effects Model is an econometric method used to capture all household-specific effects on energy

consumption. It is quite unlikely that household characteristics would significantly influence the

estimated impact of the program, given the large sample size and random selection of the

treatment and control groups. However, a Fixed Effect Model can alleviate any potential

concerns.

The logic behind the approach is to estimate separate intercepts for each customer. These

intercepts incorporate the household-specific characteristics that influence energy consumption.

It is implausible to actually estimate 80,000 separate intercepts, as is the case for this dataset.

However, econometricians have discovered a computational trick which leads them to the same

estimation results. By subtracting all included variables by the average of the variable for each

individual household, the same parameter estimates are attained.

The Fixed Effects Model has three main disadvantages.

1. Degrees of freedom loss. By implicitly including 80,000 intercept terms, the

estimation loses 79,999 degrees of freedom. The loss of degrees of freedom makes the

3.0


estimation less precise. However, given that the dataset is so large and contains over

2.5 million observations, this loss is negligible.

2. Variables that do not vary with time. The transformation involved in the Fixed

Effects Model wipes out the possibility of including variables that do not vary with

time. This limits the researcher’s ability to identify variables which influence program

impacts to only those variables where data is available and varies over time for each

household. For example, an income variable that does not vary could not be included

in the Fixed Effects Model.

3. Ease at which the results can be interpreted by a non-econometrician. For this

reason, PSE included the True Impact Test and the OLS Model in this report. All three

tests show similar results.

PSE included all of the same variables in the Fixed Effects Model that were included in the OLS

model. Given the nature of the Fixed Effects Model, however, the intercept term and the

treatment indicator term do not vary over time by household and were thus unable to be

estimated. The equation used in the estimation is as follows:

The parameters have the same interpretations as they did in the OLS Model. Of interest to this

report is the parameter estimate of “d.” Again, this is the estimated impact on average daily use

for those households in the treatment group currently receiving Home Energy Reports.

The parameter estimate for “d” is equal to -0.637, revealing that those households who are being

treated are, on average, reducing average daily usage by 0.637 kWh. As was the case for the

OLS Model, the parameter estimate was highly significant with a t-statistic of 23.46.

Estimated Impacts

OLS Model

Daily kWh -0.637

Percent Reduction -2.10%

4.0 Conclusion

4.0


4.0 Conclusion

Given the stability of the results derived from three independent estimation procedures, the large

sample size, and the randomness of sample selection, PSE concludes there are tangible energy

savings resulting from this program. Results are robust with a high degree of confidence

attached to them.

The estimated savings range from 0.621 to 0.637 regarding kWh average daily use. In

percentage terms, this is a range of 2.05 to 2.10 percent reductions in electricity use for those

receiving the Home Energy Reports relative to those that are not. PSE also concludes there are

higher savings resulting from receiving monthly reports relative to quarterly reports.

Using a rolling start date OPower claims energy savings of 2.27 percent over the life of the pilot

program. PSE used a start date to March 2009 for the fixed effect model and it produced a

savings of 2.10 percent. PSE can verify a savings between 2.05 and 2.10 percent for the first

year.

5.0 Appendix

5.0


5.0 Appendix

The Appendix begins with a list of further research options that PSE believes would be

worthwhile investments in uncovering the fullest potential of this program. The second part

presents the models for the OLS Model and Fixed Effects Model discussed in this report.

5.1 Thoughts on Further Research

Listed below are research items which could significantly enhance the understanding and optimal

structuring of the program.

1. Diminishing returns and legacy impacts: Determining the incremental gains for each

additional year of treatment would be of great interest. At what point do energy

savings level off? Do savings actually decrease as customers become desensitized to

the reports over time? Another important question is: Do savings continue after a

household is discontinued from the program and how long do these legacy savings last?

Would cycling households on and off the program be a cost effective strategy?

2. Interactions with energy efficiency programs: How does this program interact with

other energy efficiency programs? Does it enhance or detract from them? Separating

out the interaction impacts would very useful in determining accurate estimates of

energy efficiency program savings.

3. Behavioral changes behind energy savings: An issue of this program is what is

causing the decline in energy use for residences involved in the program. Is it a

conservation effect, whereby households are more alert to turn off lights and adjust

thermostats? Or are they more inclined to purchase energy efficient appliances and

light bulbs?

4. Demographic relationships with energy savings: What types of households are more

or less inclined to change energy use behavior? Is program targeting a valuable option?

5. Differences in energy savings due to neighbor comparisons “Boomerang Effect”:

How does the comparison between the households use and the neighborhood energy

use impact energy savings? Do those households who are classified as energy efficient

relative to their neighbors actually increase energy use? What comparison percentages

drive the largest energy reductions?

5.0


5.2 Model Results

OLS Model

Dependent Variable: kwhdur

Independent Estimated Standard t-

Variable Coefficient Error Statistic

constant 20.92733 3.44746e-002 6.07036e+002

testind -6.55090e-002 2.79682e-002 -2.34227

post 0.40916 3.58424e-002 11.41544

posttst -0.62150 5.01023e-002 -12.40453

hdddur 0.23123 8.39772e-004 2.75350e+002

cdddur 2.60318 7.17852e-003 3.62635e+002

Number of Observations 2585514

R-squared 5.11016e-002

Corrected R-squared 5.10997e-002

Sum of Squared Residuals 8.99848e+008

Standard Error of the Regression 18.65570

Durbin-Watson Statistic 1.76353

Mean of Dependent Variable 29.64004

Fixed Effects Model

Dependent Variable: kwhdur3

Independent Estimated Standard t-

Variable Coefficient Error Statistic

post3 -2.73721e-002 1.94173e-002 -1.40968

posttst3 -0.63681 2.71456e-002 -23.45898

hdddur3 0.22990 4.52831e-004 5.07703e+002

cdddur3 2.62359 3.87046e-003 6.77849e+002

Number of Observations 2585514

R-squared 0.16062

Corrected R-squared 0.16062

Sum of Squared Residuals 2.61489e+008

Standard Error of the Regression 10.05665

Durbin-Watson Statistic 1.85488

Mean of Dependent Variable -2.69004e-009

5.0


5.3 PSE Background

Founded in 1974, PSE is a full-service consulting firm serving the utility industry with offices in

Wisconsin, Indiana, Minnesota, Ohio, and South Dakota. PSE has expertise in the areas of

demand response, energy efficiency, revenue decoupling, merger valuations, load forecasting,

cost and reliability performance benchmarking, T&D system planning and design, resource

planning, communication technologies, smart grid investments, rate design, alternative

regulation, and cost of service studies.

5.4 Authors

Chris Ivanov, Economist

Mr. Ivanov began his career at Wisconsin Public Power Inc. and is now a utility consultant.

While at WPPI, he prepared, evaluated, and managed electric load forecasts, weather

normalization, and small area forecasts for its 49 members. As a consultant, Mr. Ivanov

prepares, evaluates, and manages electric load forecasts, surveys, and economic analyses for a

wide range of clients including distribution and G&T cooperatives. His current focus is on

assisting utilities with DSM studies and load forecasts. He has a Masters in Applied Economics

from Marquette University and is currently finishing his MBA.

Steve Fenrick, Economist

Mr. Fenrick has nearly a decade of consulting experience in the utility industry. His work has

focused on cost and reliability performance benchmarking, incentive regulation, demand-side

management program designs and evaluation, revenue decoupling, and load forecasting. Mr.

Fenrick has worked with cooperatives, investor-owned utilities, regulatory commissions, and

international utilities. He earned a Bachelor of Science in Economics from the University of

Wisconsin-Madison and is finishing a Masters degree in Agricultural and Applied Economics

from the same university.

measurement and verification report of opower energy ... · pdf filemeasurement and...

Documents