evaluation of market transformation programsevaluation of market transformation programs | march 30,...

Evaluation of Market Transformation Programs | March 30, 2014 | pg. 1

Evaluation of Market Transformation Programs

Jane Peters, Ph.D., Research Into Action

Baltimore, MarylandMarch 30, 2014


Agenda

Evaluation Concepts

Key Market Transformation (MT) Evaluation Concepts

Planning MT Evaluation

MT Indicators Exercise

Conducting MT Evaluation

Wrap-up


Evaluation Concepts


Definition of Evaluation

The systematic acquisition and assessment of information to provide useful feedback

about something.

William Trochim, Ph.D. & Evaluation Rockstar (Cornell)


Evaluation Purposes

Program evaluations are often done for the following reasons

1. Document the stated program activities, outputs, impacts and effects

2. Provide feedback to program management

3. Determine if program is being implemented as planned

4. Assess actual program impacts and effects

5. Identify opportunities for program involvement

6. Meet a statutory or regulatory requirements


Approaches to Evaluation

Summative

– Assessment at the end of a period, looking backward, summing the results

• Impact evaluation is usually summative

• Market effect evaluation is usually summative but can be formative


Approaches to Evaluation

Formative

– Assessment during implementation, to improve the program and identify needed changes or reasons to stop the program if design seems unlikely to work

• Process evaluation is usually formative

• Market progress evaluation is usually formative


Evaluation and Program Cycle

Program Planning

Program Implementation

Program Design (Redesign) Evaluation


Key Market Transformation Evaluation Concepts


Definition of Market Transformation

The strategic process of intervening in a market to create lasting change in market behavior by removing identified barriers and/or exploiting opportunities to accelerate the adoption of all cost-effective energy efficiency as a matter of standard practice.


Market Transformation Evaluation Principles

MT evaluation usually emphasizes these principles:

1. Program theory driven evaluation

2. Baseline studies and periodic data collection to understand the nature and size of the pre-program market and changes in the market over time

3. Integrated program, market and customer data collection

4. Multiple lines of evidence to increase credibility, validity and reliability of evaluation findings


Market Progress Evaluation

Purpose– Trace progress of MT program

Timing– Usually annual

Models– Logic model review– Cost effective model assumption review– Participant feedback – Market characterization update– Market feedback

Outcome– Net incremental energy and demand savings


Market Effects Evaluation

Purpose– To estimate the magnitude of energy savings associated with a change

in the structure or function of a market or behavior that is caused by an EE program

Timing– Usually one to three years, after program launch

Methods– Data collection: surveys, interviews, market research,

baseline measurement

Outcome– Assessment of progress in addressing market barriers,

adjustments and decreased assumptions


Savings Estimates

Program Launch

Ener

gy S

avin

gs

Time Measurement

Savings we can attribute to Program

with Program

Natural Savings

Baseline(no Program)


Gross Program

Savings

Savings EstimatesEn

ergy

Sav

ings

Program Launch

Time Measurement

Direct/Indirect Program Spillover (Participant and Non-participant)

Direct Measured Savings (Participants)

Non-participants

Additional Savings

with Program



FRIm

pact Evaluation

Savings EstimatesEn

ergy

Sav

ings

Program Launch

Time Measurement

Direct/Indirect Program Spillover (Participant and Non-participant)

Savings Directly Attributed to Program

Non-participants

Market

Effects

Participant Freeriders

with Program



Research Designs

Research design refers to the structure of the research elements

Observations/measurements: O

Treatments/programs: X

Assignment to a group is represented by a letter at the beginning of each line with– Random: R

– Nonequivalent Group: N

Each treatment or comparison has its own line


Some Common Research Designs

Experimental Study

Participants are randomly assigned to a treatment or a control group, with pre- and post-period measurements

R: O X O

R: O X O

Quasi-experimental Study

Participants choose a treatment or comparison group, with pre- and post-period measurements

N: O X O

N: O X O

Observational Study

Post treatment, participants and non-participants are measured

N: O X O

N: O X O


Logic Model Assists Evaluation

Provides the program description that guides the evaluation process

Helps to– Match evaluation to the program– Know what and when to measure– Focus on key, important information

• What do we really need to know?• How will we know when change has occurred?


What is a Market Progress Indicator

A metric that gauges progress in overcoming identified market barriers or otherwise changing the market, per the initiative logic model

A way to define performance

Not necessarily scalar!


Marketing & outreach

Trade ally support and incentives CHEAs Consumer financial

incentivesQuality control

$ value and reach of marketing activities; web analytics

Co-op advertising incentives and materials

Number, type and $ value of incentives to contractors; program training provided

Number and results of CHEAs

Number, $ value and type of projects accessing incentives or financing

QC procedures and inspections

Homeowners aware of HPwES opportunity; prospective homeowners reached Participating contractors

increasingly employ diagnostic equipment, projects scoped appropriately; comprehensive

Qualified home upgrades occur; are accessible

Contractor performance improves

Installations follow best practice

Increased use of diagnostic equipment and application of building science principles.

Contractors value their affiliation with HPwES and their BPI certification; find participation profitable; obtain multiple certifications

Increases in HPwES project volume demonstrate viability; increased installation of HEMI-qualified measures

Increased consumer confidence in the value of HP upgrades

New York benefits from a stable home performance-contractor market

Participating contractors expand capacity & territory

Increased demand for home energy upgrades

Nonparticipating contractors offer advanced diagnostics and HP services

Diagnostic equipment & higher project quality become standard

Sustained energy savings & demand reduction, NY housing stock more efficient

Sufficient amount of qualified / engaged contractors for program demand

Unaccredited firms and noncertified techs experience pressure to compete

CBOs engaged; recruiting targeted populations

Incremental costs of service and measures decreases

Expected savings are confirmed and refined


Outcomes & Links to Home Performance


The Home Performance Logic Indicators

BPI Status Indicators of Market Pressure

Accredited Firm

If successful, accredited firms should:– Maintain their accredited status– Be more profitable– Be able to sell HP-quality services; other evidence of

differentiation– Expand in size or in services offered– Establish expectations for enhanced standard

practices on specific types of upgrades

Non-accredited Firm

The success of BPI-accredited firms will:– Create pressure to offer similar services– Create pressure to obtain BPI accreditation– Create interest in BPI certified technicians– Force consideration, and eventual adoption, of

practices aligned with accredited firms


The Home Performance Logic Indicators

BPI Status Indicators of Market Pressure

Certified Technicians

If successful, certified technicians should:– Maintain their certification status– Receive higher wages– Have supervisory or oversight responsibility over non-

certified technicians– Possess better skills and employ practices in line with

building science and home performance– Differentiate themselves from their peers

Non-certified Technicians

The success of certified technicians will:– Create financial pressure based on perceived wage

disparity– Create pressure to avoid oversight by peer within

accredited firm– Increase interest in BPI certification– Increase interest in and commitment to aligning

practice with BPI certified peers or the increased expectations of customers or employees


Exercise: Progress Indicators


Program Description

This Home Performance with ENERGY STAR® (HPwES) Program provides information, installation, and financing of energy efficiency measures for homeowners and renters of existing one-to-four-family units. The purpose of the program is to increase the energy efficiency of homes through increasing the capacity and expertise of the energy efficiency contractor base via training, certification, and accreditation. Through these activities, the Program Administrator (PA) fosters “one-stop shopping”for home efficiency and facilitates customers’ implementation of recommended measures.

The HPwES Program comprehensively addresses a building’s shell, systems, and appliances, and is fuel-neutral; it focuses on electric savings while also addressing fuel oil and natural gas or propane end uses when they are part of a package of cost-effective measures. HPwES’s objectives are to:

– Provide comprehensive one-stop-shop services through Home Assessments (HAs) that establish the scope and estimated costs of recommended energy improvements.

– Increase local private contractor capacity for delivering high-quality comprehensive services through training, certification of contractors, and accreditation of firms and community-based organizations through an independent accreditation agency.

– Facilitate implementation of measures through (1) incentives to qualified participating contractors to cover the cost of performance tests for households who install recommended improvements and (2) access to financing for qualified households, supported by the utility.

The HPwES Program is market-based. It builds upon the capabilities of local private contractors to deliver services by expanding their expertise (be it in HVAC systems or weatherization) through training in energy efficiency, environmental skills,and technology applications, and by providing a quality seal of approval by certifying individual contractors and through the accreditation of qualifying firms. Aggressive marketing of the program through print and TV ads helps build consumer demand and works with the capacity-building activities of the program.


Progress Indicator Exercise

Expected Outcome

ProgressIndicator

Evidence Needed to Confirm


Conducting Market Transformation Evaluation


Market Characteristics

Purpose– Market size– Market structure– Information on barriers– Potential intervention points

Results– Information on market

characteristics (potential and primary market research results)

– Collected data– Clear target market– Clear product


Baseline Assessment

Purpose– Pre-initiative condition to

measure impact into the future– To assess investment

prudency

Results– Methodology of baseline

construction– Data sources used– Secondary research citations– 20-year forecast


MT Evaluation Data Sources

Primary Data– Discussions

• Individual interviews• Group interviews

– Surveys• Mail• Telephone• Electronic

– On-site• Site visits• Field observations• Ride-alongs

– Informants• Stakeholders• Market experts• Participants and non-participants

Secondary Data– Program-specific

• Program collateral• Program planning• Program databases• Program records

– Information about other programs• Evaluations• Conference papers• Best practice studies• Internet search

– Market data• Trade press• Sales data


Data Collection Advantages/Disadvantages

Method Advantages Disadvantages

Discussions and Interviews

ImpressionisticQualitative

Depth

Potential for subjective bias

Surveys BreadthQuantitative Limited depth

Observation Site VisitsImpressionistic

QualitativeExperimental

Potential for subjective bias

Program Documents Qualitative Reference

Potential for screening bias

Program Databases QuantitativeReal time GIGO


Cost and Quality Dimensions

SmallSample

Breadth

In-depth Interviews

Dep

th

Cost

LargeSample

Unlimited Questions

Limited Questions

Focus Group

Mail Survey

Electronic Survey

Telephone Survey


Wrap-up


Distinguishing Features of MT Evaluation

Barriers assessment

Program logic model and program theory of change

Baseline studies – early

Market studies – often

Data and program activities –detailed chronology


References

Websites– www.calmac.org

– www.cee1.org

– www.iepec.org

– www.neea.org


References

Barakat & Chamberlin, (1992) DSM Evaluation: Six Steps for Assessing Programs (Palo Alto, CA. Electric Power Research Institute, CU-6999)

Herman, P., Feldman, S., Samiullah, S., Mounzih, K. (1997) Measuring Market Transformation: First You Need A Story. In Proceedings of the 1997 Energy Evaluation Conference. Chicago, IL. pps. 319-326

Sebold, F., A. Fields, L. Skumatz, S. Feldman, M. Goldberg, K. Keating, & J. Peters. (2001) A Framework for Planning and Assessing Publicly Funded Energy Efficiency. Pacific Gas & Electric. San Francisco, CA

Patton, M.Q. (1997) Utilization-Focused Evaluation. (3rd Edition) Thousand Oaks, CA. Sage Publications The TecMarket Works Team. (2006) California Energy Efficiency Evaluation Protocols: Technical,

Methodological, and Reporting Requirements for Evaluation professionals. California Public Utilities Commission. http://www.calmac.org/publications/EvaluatorsProtocols_Final_AdoptedviaRuling_06-19-2006.pdf

TecMarket Works. (2004) The California Evaluation Framework. California Public Utilities Commission. http://www.calmac.org/publications/California_Evaluation_Framework_June_2004.pdf

United Way of America. (1996) Measuring Program Outcomes: A Practical Approach. United Way Item # 0989. Fairfax, Virginia. (800)772-0008

Contact:

[email protected]