measuring and reporting energy savings for the energy ... · measuring and reporting energy savings...

Measuring and reporting energy savings for the Energy Services Directive – how it can be done Results and recommendations from the EMEEES project

Wuppertal Institute on behalf of the EMEEES Consortium Wuppertal 30 June 2009

The Project in brief

The objective of this project is to assist the European Commission in developing harmonised evaluation methods. It aims to design methods to evaluate the measures implemented to achieve the 9% energy savings target set out in the EU Directive (2006/32/EC) (ESD) on energy end-use efficiency and energy services. The assistance by the project and its partners is delivered through practical advice, technical support and results. It includes the development of concrete methods for the evaluation of single programmes, services and measures (mostly bottom-up), as well as schemes for monitoring the overall impact of all measures implemented in a Member State (combination of bottom-up and top-down).

Consortium

The project is co-ordinated by the Wuppertal Institute. The 21 project partners are:

Project Partner Country

Wuppertal Institute for Climate, Environment and Energy (WI) DE

Agence de l’Environnement et de la Maitrise de l’Energie (ADEME) FR

SenterNovem NL

Energy research Centre of the Netherlands (ECN) NL

Enerdata sas FR

Fraunhofer-Institut für System- und Innovationsforschung (FhG-ISI) DE

SRC International A/S (SRCI) DK

Politecnico di Milano, Dipartimento di Energetica, eERG IT

AGH University of Science and Technology (AGH-UST) PL

Österreichische Energieagentur – Austrian Energy Agency (A.E.A.) AT

Ekodoma LV

Istituto di Studi per l’Integrazione dei Sistemi (ISIS) IT

Swedish Energy Agency (STEM) SE

Association pour la Recherche et le Développement des Méthodes et Processus Industriels (ARMINES)

FR

Electricité de France (EdF) FR

Enova SF NO

Motiva Oy FI

Department for Environment, Food and Rural Affairs (DEFRA) UK

ISR – University of Coimbra (ISR-UC) PT

DONG Energy (DONG) DK

Centre for Renewable Energy Sources (CRES) EL

Contact

Dr. Stefan Thomas, Dr. Ralf Schüle

Wuppertal Institute

for Climate, Environment and Energy

Döppersberg 19

42103 Wuppertal, Germany

Tel.: +49 (0)202-2492-110

Fax.: +49 (0)202-2492-250

Email: [email protected]

URL: www.evaluate-energy-savings.eu

www.wupperinst.org

The sole responsibility for the content of this publication lies with the authors. It does not necessarily reflect the opinion of the European Communities. The European Commission is not responsible for any use that may be made of the information contained therein.

Measuring and reporting energy savings for the ESD – how it can be done

2 Wuppertal Institute on behalf of the EMEEES Consortium

Measuring and reporting energy savings

for the Energy Services Directive – how it can be done

Presented by the EMEEES project consortium; drafted by Stefan Thomas, Wuppertal Institute

Contents

Executive Summary ..........................................................................................5

1 Introduction.................................................................................................14

1.1 The ESD and the EMEEES project...........................................................14

1.2 Results of the project ................................................................................15

1.3 Structure and contents of the report .........................................................15

2 How to calculate energy savings for the ESD? .......................................17

2.1 What are ESD energy savings, and why do calculation methods matter?17

2.1.1 Requirements of the ESD for monitoring and evaluation of energy savings ..........................................................................................................172.1.2 What are energy savings in 2016? ........................................................182.1.3 Open issues and their potential impacts on the energy savings achieved by the ESD ........................................................................................................19

2.2 Principles and useful terminology for ESD calculation and harmonisation ... ..............................................................................................................20

2.2.1 Guiding principles for EMEEES in the development of methods ...........202.2.2 Can energy savings be measured at all?...............................................212.2.3 Addressing harmonisation issues ..........................................................222.2.4 What is the subject of an evaluation method? .......................................24

2.3 Bottom-up calculation methods.................................................................25

2.3.1 Four steps in the calculation process.....................................................262.3.2 Three levels of harmonisation................................................................282.3.3 Five general bottom-up methods ...........................................................29

2.4 EMEEES bottom-up case applications .....................................................31

2.5 Top-down calculation methods .................................................................32

2.6 EMEEES top-down cases.........................................................................35

2.6.1 Additional energy savings ......................................................................352.6.2 All energy savings..................................................................................372.6.3 Applicable top-down calculation methods..............................................37

2.7 Integration: selection and consistency of methods ...................................38

Stefan Thomas et al.

Wuppertal Institute on behalf of the EMEEES Consortium 3

2.7.1 Selection of bottom-up or top-down methods by end use or sector.......392.7.2 Selection of bottom-up or top-down methods by type of EEI measure..412.7.3 Consistency between bottom-up and top-down methods ......................41

2.8 Existing evaluation and monitoring methods in the EU.............................43

2.9 Need for further research and development .............................................46

2.10 Evaluation of costs and benefits .............................................................47

3 How to monitor and report to the European Commission? ...................51

3.1 The overall monitoring and reporting process...........................................51

3.2 Coverage of end uses and sectors as well as overlap between EMEEES bottom-up case applications and top-down cases.............................................53

3.3 Applicability of EMEEES bottom-up case applications and top-down cases by EU Member State .........................................................................................55

3.3.1 Application of top-down and bottom-up methods for countries..............553.3.2 Meeting ESD demands with EMEEES cases ........................................573.3.3 Comments on applicability from the national workshops and the Final Conference ........................................................................................................58

3.4 Feedback from the Pilot tests on applicability of methods ........................59

4 How can the European Commission judge the results? ........................63

4.1 The tool for NEEAP assessment ..............................................................63

4.2 Test assessments of the energy savings expected in selected NEEAPs .63

4.2.1 Germany ................................................................................................644.2.2 Italy ........................................................................................................654.2.3 Austria....................................................................................................654.2.4 Conclusions ...........................................................................................67

4.3 How to compare individual evaluation results?.........................................68

5 Conclusions................................................................................................70

6 References ..................................................................................................74

7 Ackowledgements......................................................................................76

8 Appendices .................................................................................................78

8.1 Appendix 1: List of EMEEES Reports and Bottom-up Case Applications 78

8.2 Appendix 2: Proposal for a reporting checklist for bottom-up evaluations 81

Appendix to the Bottom-up Reporting Checklist: Non-exhaustive list of energy efficiency improvement measures and mechanisms.........................................86

8.3 Appendix 3: Proposal for a reporting checklist for top-down evaluations .87

8.4 Appendix 4: why the harmonised system of evaluation methods needs to be able to calculate both additional and all energy savings ..............................90



8.5 Appendix 5: Types of evaluation methods vs types of EEI measures ......93



Executive Summary

The primary objective of the EU Directive on energy end-use efficiency and energy

services (2006/32/EC), also abbreviated as ‘Energy Services Directive’ or ESD, is to

stimulate activities by the Member States and the market actors to save energy in final

use.

Clearly, monitoring and evaluation of the energy savings is very relevant and important

for the impact the Energy Services Directive may finally have. The Directive requires

the Member States to adopt cumulative annual energy savings targets to be achieved

by 2016. Some have adopted the indicative 9 % target, some have even adopted

higher targets. The targets are calculated in relation to the average overall annual

energy consumption in a Member State in the base period 2001 to 2006, excluding

final energy consumption in the industrial undertakings subject to the EU emissions

trading scheme and for military purposes. The targets are expressed as an amount of

annual energy consumption saved through energy efficiency improvement measures.

The Member States have to prove to the European Commission that they have saved

enough energy to reach their targets.

Consequently, the methods and tools, by which the Member States calculate, evaluate,

and report their energy savings towards achieving the targets adopted for the Directive

are very important. The results from individual Member States must be comparable to

build confidence that all have taken comparable effort to stimulate the markets for

energy services and energy end-use efficiency in general. At the same time, the effort

for monitoring, evaluation, calculation, and reporting must be limited to a reasonable

level. However, it should be kept in mind that there is a trade-off between effort and

precision. The higher the precision, the fairer the comparison between the results

presented by Member States. Therefore, there is also a trade-off between effort and

fairness.

What are energy savings for the Energy Services Directive?

The potential for energy efficiency improvement measures that save more money than

they cost is large enough to achieve at least 9 % by 2016 in each Member State. The

EU Action Plan for Energy Efficiency of 2006 has estimated that from 2005 and 2020,

between 25 % and 30 % of energy savings are possible in the four major end-use

sectors (residential, tertiary, industry, and transport). Within the nine years period 2008

to 2016 covered by the Energy Services Directive, that would be equivalent to between

15 and 18 %. And those savings come in addition to savings due to ‚autonomous

improvement’, such as normal replacement of technology stock, as the EU Action Plan

states. Furthermore, the action plan estimates ‘it is still technically and economically

feasible to save at least 20% of total primary energy by 2020 on top of what would be

achieved by price effects and structural changes in the economy, natural replacement

of technology and measures already in place’. This cost-effective potential includes



savings both in end-use sectors and at the level of energy transformation but translates

to at least 12 % within nine years.

However, what exactly is the meaning of the 9 % or more of cumulative annual energy

savings? The Energy Services Directive does not mention that they shall be additional

to ‚autonomous improvement’.

Furthermore, the Directive contains a paragraph on ‚early action’ since 1995, which

can be interpreted in two ways. It could either mean to reassure that new energy

savings in the 2008 to 2016 period due to ‚early measures’ can be counted towards the

target; an example would be energy savings from an energy-efficient building refur-

bishment in 2010, which is economically attractive due to an energy tax enacted in

1999. Or this paragraph can be read as to allow inclusion of ‚early energy savings’,

e.g., from a building refurbishment in 2005 stimulated by the 1999 tax reform.

What could be the consequence of these two unclarities? The EU Action Plan for

Energy Efficiency has estimated the ‚autonomous improvement’ to be 0.85 % per year,

or 7.4 % in the period 2008 to 2016. And in the period 1995 to 2016, it would be

17.1 %1. In other words: if energy savings counting towards the targets are not required

to be additional to ‚autonomous improvement’ AND ‚early energy savings’ are allowed,

a Member State may not need to prove any new energy savings in addition to ‚au-

tonomous improvement’. But would this be a wise decision, given that the cost-

effective potential for new energy savings additional to ‚autonomous improvement’

within 2008 to 2016 is most likely even higher than 9 %?

The European Commission and the regulatory Committee created for the implementa-

tion of the Directive have not yet published a decision on how to deal exactly with these

two issues, and it is not in the competence of the EMEEES project to decide this. The

methods and case applications developed by EMEEES, therefore, enable Member

States to both calculate all energy savings (including those through ‘autonomous

improvement’) and the additional energy savings (excluding those through

‘autonomous improvement’). Furthermore, the methods and case applications enable

Member States to assess whether early energy savings achieved before 2008 still

exist in 2016. This does not prejudice the choice of calculating all or additional energy

savings, nor whether early energy savings should count towards the ESD target or not.

The EMEEES project

This is the environment, in which this project has worked to devise how energy savings

can be calculated. From November 2006 to April 2009, the IEE2 project “Evaluation

1 The result for 2016 must not be calculated by adding up 0.85% of the initial value 21 times for 21

years, but by multiplying 99.15 % (100% - 0.85%) 21 times and then comparing the result to 100 %. 2 Programme Intelligent Energy Europe of the European Commission:

http://ec.europa.eu/energy/intelligent/index_en.html



and Monitoring for the EU Directive on Energy End-Use Efficiency and Energy

Services” (EMEEES) worked on a set of calculation methods and case applications,

with 21 partners and co-ordinated by the Wuppertal Institute. The project partners were

able to bring strong experience in evaluation methodology and practice as well as

different perspectives to the consortium. They included energy agencies, a ministry,

two energy companies, and several research institutes and consultancies; they are

listed on the back of the front page and in the acknowledgements. The objective of this

project was to assist the European Commission in the elaboration of evaluation

methods through delivering practical advice, support, and results. This included the

development of concrete methods for the evaluation of single programmes, services

and measures, as well as with schemes for monitoring the overall impact of all

measures implemented in a Member State. Altogether, EMEEES was focused on the

energy savings impact evaluation of measures, not on process and cost/benefit

evaluation.

This report presents the EMEEES project consortium’s findings. The many detailed

reports, tools, and guidelines that the project prepared are listed in Appendix 1. They

are available for download at www.evaluate-energy-savings.eu.

Calculation of energy savings for the Energy Services Directive with the methods

developed by EMEEES

Within the limitations mentioned above, EMEEES has been able to prepare general

methods for bottom-up and top-down calculation methods plus guidelines for ensuring

consistency between the results of bottom-up and top-down calculations. Bottom-up

methods start from data at the level of a specific energy efficiency improvement

measure (e.g., energy savings per participant and number of participants) and then

aggregate results from all the measures. Top-down methods start from global data

(e.g., national statistics for energy consumption or equipment sales), then going down

to more disaggregated data when necessary (e.g., energy efficiency indicators already

corrected for some structural or weather effects).

The project furthermore developed 20 bottom-up case applications and 14 top-

down cases of these general methods, which together already cover the largest part

of potential ESD energy savings from the energy efficiency improvement measures the

Member States have pledged to implement in their national Energy Efficiency Action

Plans. For example, we estimate that with our total set of bottom-up case applications,

more than 90% coverage of the energy use subject to the ESD can be achieved.

Some of the developed methodologies were since tested through pilot cases.

Therefore, EMEEES has been able to confirm that evaluation of energy efficiency

improvement measures and calculation of ESD energy savings is possible. In

summary, we recommend to use the following methods:



• Top-down calculation methods can be used for electric appliances and

vehicles, for which there is a well-defined statistical indicator of the average

specific annual energy consumption per unit of appliance or per vehicle, and for

solar water heaters. In these cases, the indicator is well-suited to capture the

effects of the whole package of measures, including multiplier3 (market

transformation) effects.

Bottom-up calculations are possible for appliances and vehicles, too, but it is

often difficult to calculate multiplier (and free-rider4) effects with them.

For top-down calculation on appliances and vehicles, a reference trend can be

defined for these specific energy consumption indicators to calculate additional

energy savings; this reference trend should either be an EU harmonised trend

or a national trend based on an EU harmonised coefficient (e.g., for elasticity to

increases in market energy prices5). Furthermore, the base year value may be

assumed to be a proxy for the correct reference trend for calculating all energy

savings for these indicators.

• Top-down methods are the way to calculate the effects of energy taxation6

and add them to the effects of bottom-up calculations for a sector, but only if

these bottom-up calculations exclude free-rider effects. The energy savings due

to taxation must not be added to results of top-down calculations on sectors or

end-use equipment, if the latter already include an analysis to calculate the

effects of energy taxation.

• It is the best and often the only possible way to use bottom-up calculation

methods for all other end-use sectors, end-uses, and energy efficiency

improvement measures. This is particularly the case for buildings, for the

industry and tertiary sectors with their larger final consumers that are easier

to monitor, and for modal shifts and eco-driving in transport.

In these areas, structural effects can often not be corrected for in top-down

indicators, or it will need costly bottom-up modelling and gathering the

necessary data for that modelling to do the required corrections: Neither of the

two reference trends mentioned above for appliances and vehicles are usually

possible for top-down indicators measuring the energy consumption of a sector

3 The multiplier (or spill-over) effect enhances the initial effect of EEI measures. According to Annex IV-5

of the ESD the multiplier effect means that “the market will implement a measure automatically without any further involvement from the authorities or agencies referred to in Article 4-4 or any private-sector energy services provider”.

4 The free-rider effect regards market actors who make use of facilities or support, provided for by EEI programmes, policies, or energy services, but would have taken energy-saving actions anyway. The free-rider effect is not mentioned in the ESD. It must be corrected for, if the aim is to calculate additional energy savings.

5 We found that it was usually not statistically meaningful to calculate national values for the price elasticity of the different indicators. Therefore, we propose to use EU harmonised coefficients, although price elasticity is likely to differ between Member States.

6 This includes general taxes on energy, but not fiscal incentives targeting specific energy efficiency investments, such as tax deductions or rapid depriciation rules. These incentives are specific measures and should be covered by calculations specific to a sector, end use, or type of end-use action.



per unit of production or per employee. For indicators measuring the diffusion of

energy-efficient transport modes or combined heat and power in industry, the

situation will depend on the country. For most of these sectoral or diffusion

indicators, some Member States may see ‘apparent total’ savings when

comparing the current value of an indicator with its value in the base year, while

others may not. The reason for this is that either these countries really do not

have savings, or their savings are hidden by structural changes that cannot be

corrected for due to lack of data. Using ‘apparent total’ savings as a proxy for all

energy savings for these top-down indicators would, therefore, lead to

inconsistent and arbitrary measures of energy savings between Member States.

This will disable the use of top-down methods in such cases.

By contrast, bottom-up calculations are usually feasible.

This recommendation is based on our analysis of case applications for bottom-up and

top-down methods (cf. sections 2.3 to 2.6), as well as on practical experience in many

countries and our pilot tests (cf. section 3.4). They are based on the general trend of

findings from these sources.

Bottom-up calculation needs specific monitoring but can provide information on the

effectiveness and cost-effectiveness of measures, on potential improvements, and on

greenhouse gas emission reductions additional to baseline projections. However,

calculation of multiplier and free-rider effects with bottom-up methods can be costly,

particularly for appliances and vehicles, for which the multiplier effects are particularly

important. Furthermore, they work into the opposite direction, hence partly cancel out

each other. Calculation of both multiplier and free-rider effects could, therefore, be

restricted to measures either yielding at least 40 million kWh of annual electricity

savings or 100 million kWh of annual energy savings of other fuels, or at least 5

percent of a Member State’s ESD energy savings target.

Top-down calculation starts from using existing statistical data and can be easier to

apply, particularly in areas, for which many and overlapping energy efficiency

improvement measures exist. However, it is often difficult to define the reference trend

as stated above, or the indicator is not showing energy savings at all without costly

corrections.

Therefore, the quality of data available in a country will finally determine which

bottom-up or top-down methods are best to apply for evaluating the energy savings for

the ESD from a sector, an energy end use, an end-use action, or a measure.

How to ensure consistency between top-down and bottom-up calculations

The ESD monitoring system can include both bottom-up or top-down methods for

monitoring and evaluation with one Member State. In order for it to be a harmonised

system, the results of either bottom-up or top-down calculation must be consistent



and comparable with each other. This requires that the elements of calculation need

to be chosen in a consistent manner for both bottom-up and top-down calculations, and

for the two evaluation targets introduced above: additional energy savings and all

energy savings.

This section presents the elements that would ensure consistency, in the tables

ES1 and ES2 below. It must be noted that only the elements of bottom-up and top-

down calculations in either of the two columns of the tables: all energy savings and

additional energy savings, respectively, are consistent with each other. Using the

elements of bottom-up calculation from one and those of top-down from the other

column of the tables would be highly inconsistent.

Table ES1: Elements of bottom-up calculation for all or additional energy savings



Table ES2: Elements of top-down calculation for all or additional energy savings

The overall process of evaluating energy savings for the Energy Services

Directive

EU Member States can choose from the EMEEES bottom-up case applications and

top-down cases, use own existing or new methods, or develop their own case

applications of the general methods descibed by EMEEES when fulfilling the demands

of the ESD:

• proving that the 9% or higher savings target has been met for 2016 (or the

intermediate target for 2010)

• showing that bottom-up methods applied cover at least 20-30% of the energy use

covered by the ESD

• taking account of overlap in the scope of top-down cases and bottom-up case

applications focusing on the same targeted energy use, in order to avoid double

counting of energy savings.

Figure ES1 shows how, in an interactive five-step process, countries can choose a set

of methods that meets the ESD demands.



Figure ES1: The process of evaluating ESD energy savings

TD = top-down; BU = bottom-up

Towards a harmonised calculation model of energy savings for the Energy

Services Directive?

The ESD has required the European Commission to propose a harmonised

calculation model of bottom-up and top-down calculation methods for ESD energy

savings.

Certainly, the Commission and the Member States could decide to use as many

default or even harmonised values as possible. EMEEES has developed some

proposals in this area.

On the other hand, the accuracy will probably be higher if national level 2 and 3

calculations (bottom-up) and national reference trends (top-down) are used, but with

harmonised rules for a) definition of formulas, parameters, monitoring, and

calculation procedures, and b) harmonised reporting of results. This is certainly also

an area, in which more experience needs to be collected in the next round of national

Energy Efficiency Action Plans (NEEAPs) in 2011. These NEEAPs will include the first

ex-post calculations of energy savings. We strongly recommend that the European

Commission require harmonised reporting using at least a format such as the

reporting checklists developed by EMEEES and presented in Appendices 2 and 3.

Probably the most effective way to achieve harmonisation between Member States is

to encourage and facilitate sharing of experience. If Member States learn from each

other, this will lead to more harmonised practices. Harmonised reporting will be highly

important to facilitate such sharing of experience and mutual learning.



Harmonised reporting will also allow the Commission to better judge the plausibility and

comparability of savings (and hence efforts) between Member States and in many

cases also a verification of the reported energy savings using models such as the

assessment tool that is another product of EMEEES.

Finally, it should be noted that evaluation is not only possible, it is also necessary for

the sake of continuous development and refinement of energy efficiency policies

and other energy efficiency improvement measures. However, evaluation is an area

where a single best method cannot be devised. Methods must evolve and be adapted

to the measure and context at hand. The quality of evaluations will improve as

experience accrues through learning-by-doing. This will not only benefit the cost-

effective implementation of the Directive, but also provide a basis for future and

probably more ambitious energy efficiency policy efforts.



1 Introduction

1.1 The ESD and the EMEEES project

The Directive on energy end-use efficiency and energy services (2006/32/EC; for the

remainder of this report abbreviated as the ESD) has required Member States to adopt

an indicative target of 9 % of cumulative annual energy savings in 2016. It has also

raised concerns among the Member States about how they could evaluate the energy

savings from energy services and other energy efficiency improvement measures

implemented in order to achieve their energy savings targets. The constitution of a

regulatory Committee of the Member States and the European Commission (hereafter

named ESD Committee) has therefore been included in the Directive to assist the

European Commission in the task of elaborating common and harmonised methods for

the evaluation of energy savings. Due to the difficulties related to this task, the

Commission also needed support from independent experts.

From November 2006 to April 2009, the IEE7 project “Evaluation and Monitoring for the

EU Directive on Energy End-Use Efficiency and Energy Services” (EMEEES) worked

on a set of calculation methods and case applications, with 21 partners and co-

ordinated by the Wuppertal Institute. The project partners were able to bring strong

experience in evaluation methodology and practice as well as different perspectives to

the consortium. They included energy agencies, a ministry, two energy companies, and

several research institutes and consultancies; they are listed on the back of cover page

and in the acknowledgements. The objective of this project was to assist the

Commission in the elaboration of evaluation methods through delivering practical

advice, support, and results. This included the development of concrete methods for

the evaluation of single programmes, services and measures (mostly bottom-up), as

well as with schemes for monitoring the overall impact of all measures implemented in

a Member State (combination of bottom-up and top-down methods8). Altogether,

EMEEES was focused on the energy savings impact evaluation of measures, not on

process and cost/benefit evaluation. For process evaluation, the policy theory or

programme theory approach is very useful. It has been explored for energy efficiency

policies in EU Member States in the AID-EE project (AID-EE, 2007). As for cost/benefit

evaluation, we offer some information in chapter 2.9. Further information can be found,

e.g., in a report coordinated by SRCI (1996).

7 Programme Intelligent Energy Europe of the European Commission :

http://ec.europa.eu/energy/intelligent/index_en.html 8 Bottom-up methods start from data at the level of a specific energy efficiency improvement measure

(e.g., energy savings per participant and number of participants) and then aggregate results from all the measures. Top-down methods start from global data (e.g., national statistics for energy consumption or equipment sales), then going down to more disaggregated data when necessary (e.g., energy efficiency indicators already corrected for some structural or weather effects).



With this report, the EMEEES project consortium presents the essence of its findings.

The many detailed reports, tools, and guidelines that the project prepared are listed in

Appendix 1. They are available for download at www.evaluate-energy-savings.eu.

1.2 Results of the project

The direct results of EMEEES are

(1) a system of bottom-up and top-down methods and their integrated application for

the evaluation of around 20 types of energy efficiency technologies and/or energy

efficiency improvement measures, harmonised between Member States;

(2) a set of harmonised input data and default values for these evaluation methods;

(3) a template for Member States for the national Energy Efficiency Action Plans

(NEEAPs); and

(4) a method and tool for the European Commission to assess the plans.

With regard to the evaluation methods developed by EMEEES, the results include:

• Two summary reports on methods: bottom-up (Vreuls et al., 2009) and top-down

(Lapillonne et al., 2009)

• A bottom-up methodological report (Broc et al, 2009)

• 20 bottom-up case applications papers (cf. table 3 in section 2.4 for the list)

• Compilation of EMEEES formulae for unitary gross annual energy savings,

baselines, and default values as well as data to collect for bottom-up case

applications

• A compilation report on 14 top-down case studies (cf. table 4 in section 2.6 for the

list)

• A report on consistency and the integration of the savings from bottom-up and top-

down methods (Boonekamp and Thomas 2009)

• The EMEEES checklist for reporting the results of energy efficiency improvement

(EEI) measures (cf. Appendix 2 and 3).

A full list of EMEEES reports and papers can be found in Appendix 1.

1.3 Structure and contents of the report

After this introduction, three main chapters follow. Chapter 2 presents the most

important findings on how to measure and calculate energy savings for the ESD:

requirements by the Directive, general principles, bottom-up calculation methods and

case applications, top-down calculation methods and case applications, and how

bottom-up and top-down compare. The chapter further expands on methods already

applied in EU Member States and on appropriate choices of methods by type of EEI



measure. It also deals with what EMEEES could not resolve, and offers some

information on how to evaluate economic benefits and costs of EEI measures, although

this was beyond the scope of the EMEEES project.

Chapter 3 addresses the overall monitoring and reporting process: (1) what are the

steps in it, which end uses and sectors are covered by the EMEEES bottom-up and

top-down cases, (2) which of the cases can be applied in which EU Member State on

the basis of the EEI measures included in their national Energy Efficiency Action Plans,

and (3) what were the results of pilot tests EMEEES carried out on the applicability of

its methods and cases in a number of Member States.

Based on the Action Plans, the Commission must judge if Member States can achieve

their targets. Chapter 4 presents a tool that EMEEES developed for this task, and

results of its application for three Member States. It also proposes what information the

Commission should ask from the Member States to judge the plausibility and

comparability of the energy savings reported. Finally, a set of conclusions in chapter 5

rounds off the report.



2 How to calculate energy savings for the ESD?

2.1 What are ESD energy savings, and why do calculation methods

matter?

As the first step, we have to take a look at what is the quantity to be calculated. What

does the ESD say about it, and is it really clear what it means?

2.1.1 Requirements of the ESD for monitoring and evaluation of energy savings

The Directive requires the Member States to adopt cumulative annual energy savings

targets to be achieved by 2016. Some have adopted the indicative 9 % target, some

have even adopted higher targets. The targets are calculated in relation to the average

overall annual energy consumption in a Member State in the base period 2001 to

2005, excluding final energy consumption in the industrial undertakings subject to the

EU emissions trading scheme and for military purposes. The targets are expressed as

an amount of annual energy consumption saved through energy efficiency

improvement measures. The Member States have to prove to the European

Commission that they have saved enough energy to reach their targets.

This is why the methods and tools, by which the Member States calculate, evaluate,

and report their energy savings towards achieving the targets adopted for the Directive

are very important. The results must be comparable to build confidence that all

Member States have taken comparable effort to stimulate the markets for energy

services and energy end-use efficiency in general. At the same time, the effort for

monitoring, evaluation, calculation, and reporting must be limited to a reasonable level.

However, it should be kept in mind that there is a trade-off between effort and

accuracy. The higher the accuracy, the fairer the comparison between the results

presented by Member States. Therefore, there is also a trade-off between effort and

fairness.

The ESD is the first European Directive requiring Member States to report for energy

savings. Member States already have their own experiences and skills in this field, but

it should not be expected that all Member States will be able to use state-of-the-art

evaluation methods right away. And the Commission will also have to set up its own

evaluation system to comment the National Energy Efficiency Action Plans (NEEAPs)

reported by the Member States. Therefore, the first ESD interim implementation phase

(i.e., 2008-2010, to be reported in the second NEEAPs in 2011) should be used as a

formative process for all stakeholders to issues raised when evaluating energy savings.

One of the main challenges of the ESD methodology requirements for the work of the

EMEEES project is to enable the definition of harmonised methods. It has to take into

account that some countries have a longer history in monitoring and evaluating energy

efficiency activities than others. "Harmonised methods" does not mean that the



Member States already using their own evaluation systems will have to change them.

The EMEEES project developed bottom-up and top-down calculation methods that are

to be seen as general principles for the way to report the results and as guidelines

to perform evaluations, for Member States which may look for support.

Member States can use their own monitoring systems, based on their specific needs

and experience. But at the end, the principles of calculation and reporting they use for

their NEEAP should be harmonised for all Member States. In most of the cases, it will

be possible to use existing monitoring methods and schemes, but then the results

should be processed to fit ESD reporting needs. In some cases, also the monitoring

methods and schemes may have to be adapted to collect the data needed for

calculating energy savings according to the ESD.

2.1.2 What are energy savings in 2016?

There can be different interpretations of the term ‘cumulative annual energy savings’ in

the target year 2016. EMEEES has adopted the ‘vintage year’ interpretation: in each

year from 2008 to 2016, energy efficiency improvement measures are implemented in

a Member State and yield annual energy savings (e.g., expressed in kWh/year). These

annual energy savings are valid as long as their saving lifetime lasts. The annual

energy savings from each ‘vintage year’ that are still lasting in 2016 are then cumulated

to yield the ESD energy savings of the Member State. This approach is the only one

consistent with adopting a target of x% of annual energy savings in a target year; it is

also the only one consistent with top-down calculations of the annual energy savings

that accumulate over the years. Figure 1 presents this process.

Figure 1: Accumulation of annual energy savings

As can be seen, some energy savings terminate before 2016 and can then not be

counted towards the ESD energy savings. This may particularly be the case for



behavioural actions or for some technologies with shorter lifetimes, such as computers.

Concerns have been raised that Member States may be punished for measures early

in the 2008 to 2016 ESD period, and may wait until the last minute to implement

measures. However, there is always the possibility to check at regularly intervals,

whether energy-efficient behaviour is retained or markets have been permanently

changed, so that the lifetime of the respective energy savings can be extended.

Furthermore, this is one reason, why the intermediate target for 2010 was introduced in

the ESD.

2.1.3 Open issues and their potential impacts on the energy savings achieved by the ESD

What exactly is the meaning of the 9 % or more of cumulative annual energy savings?

The Energy Services Directive does not mention that they shall be additional to

‚autonomous improvement’. In October 2006, however, the European Commission

published the Action Plan for Energy Efficiency: Realising the Potential

(COM(2006)545 final). It stated that there is a cost-effective potential for energy

savings of more than 20 % compared to baseline projections by 2020. Based on this

Action Plan, the European Council on 8/9 March 2007 stressed “the need to increase

energy efficiency in the EU so as to achieve the objective of saving 20 % of the EU’s

energy consumption compared to projections by 2020, as estimated by the

Commission in its Green Paper on Energy Efficiency, and to make good use of their

National Energy Efficiency Action Plans for this purpose.” This is, therefore, a target for

additional energy savings. The reference to the National Energy Efficiency Action

Plans suggests that the European Council expects a significant contribution from the

ESD towards these additional energy savings.

Furthermore, the Directive contains a paragraph on ‚early action’ since 1995, which

can be interpreted in two ways. It could either mean to reassure that new energy

savings in the 2008 to 2016 period due to ‚early measures’ can be counted towards

the target; an example would be energy savings from an energy-efficient building

refurbishment in 2010, which is economically atractive due to an energy tax enacted in

1999. Or this paragraph can be read as to allow inclusion of ‚early energy savings’,

e.g., from a building refurbishment in 2005 stimulated by the 1999 tax reform.

What could be the consequence of these two unclarities? Appendix 4 presents a

thorough discussion, here we just wish to present the essence: The EU Action Plan for

Energy Efficiency has estimated the ‚autonomous energy savings’9 to be 0.85 % per

year, or 7.4 % in total in the period 2008 to 2016. And in the period 1995 to 2016, it

would be 17.1 %10. In other words: if energy savings counting towards the targets are

not required to be additional to ‚autonomous improvement’ AND ‚early energy savings’

9 “brought about by natural replacement, energy price changes, etc.” as stated in the EU Action Plan

(EC, 2006) 10 The result for 2016 must not be calculated by adding up 0.85% of the initial value 21 times for 21

years, but by multiplying 99.15 % (100% - 0.85%) 21 times and then comparing the result to 100 %.



are allowed, a Member State may not need to prove any new energy savings in

addition to ‚autonomous improvement’. But would this be a wise decision, given that

the cost-effective potential for new energy savings additional to ‚autonomous improve-

ment’ within 2008 to 2016 is higher than 9 %?


tion of the Directive have not yet published a decision on how to deal exactly with these

two issues, and it is not in the competence of the EMEEES project to decide this. The

methods and case applications developed by EMEEES, therefore, enable Member

States to both calculate all energy savings (including those through ‘autonomous

improvement’) and additional energy savings (excluding those through ‘autonomous

improvement’). Furthermore, the methods and case applications enable Member

States to assess whether early energy savings achieved before 2008 still exist in

2016. This does not prejudice the choice of calculating all or additional energy savings,

nor whether early energy savings should count towards the ESD target or not.

• Additional energy savings are understood as those that are additional to

autonomous energy savings (i.e., to savings that would occur without energy

efficiency programmes, energy services, and other energy efficiency policies such

as building codes or energy efficiency mechanisms). These additional energy

savings include additional energy savings due to existing policies, programmes,

and services that are ongoing or have a lasting effect.

• By contrast, all energy savings are those resulting from all technical, organisational,

or behavioural actions taken at the end-use level to improve energy efficiency,

whatever their driving factor (or cause) (energy services, policies, or market forces

and autonomous technical progress).

2.2 Principles and useful terminology for ESD calculation and

harmonisation

What are the consequences of the requirements and uncertainties about the

interpretation of the ESD for the calculation methods? What is the subject of an

evaluation method? And can energy savings be measured at all? These are questions

to be analysed in this section.

2.2.1 Guiding principles for EMEEES in the development of methods

As a consequence of all the considerations made in chapter 2.1, the EMEEES team

adopted the following guiding principles for our work:

• Be as thorough as possible in analysing the relevance of correction factors, and

the possibilities to evaluate them,

• but be as pragmatic as possible in the methods proposed as a result of the

analysis;



keeping in mind that the evaluation system has to be applicable (technically), not

costly (economically) and fair between Member States (ethically)

• with as many EU-level average values as possible

• enable avoiding double-counting

• enable estimating the multiplier effect, if possible

• enable the evaluation of all, additional, and early energy savings in a consistent

way (cf. section 2.7.3 on how to achieve such consistency)

• develop both bottom-up and top-down methods

The Member States will have to report energy savings based on harmonised

methods (ESD Annex IV(1.1)). This harmonisation obviously covers the following

issues:

• using the same accounting unit

• using common and consistent basic assumptions (e.g. baseline/reference trend;

correction factors); these must differentiate between the calculation of all and

additional energy savings, but be consistent between bottom-up and top-down

calculations, cf. section 2.7.3

• providing a minimum set of information for each type of calculation

• to the highest extent as possible, using a consistent level of evaluation efforts.

Member States have different experiences and starting points; but they should use

harmonised requirements for reporting their results.

EMEEES developed six bottom-up methods, 20 bottom-up case applications, and 14

top-down cases based on three types of indicators. These are not exhaustive, but a

starting point. General principles for calculation and reporting to ensure comparable

results are therefore provided, too, so that the results can be compared, no matter

whether EMEEES methods or own methods of the Member States are used.

2.2.2 Can energy savings be measured at all?

Unlike energy, energy savings can usually not directly be measured. However, they

can be indirectly measured or estimated in relation to a reference situation. Such a

reference situation will, therefore, always be needed to calculate energy savings.

ESD Annex IV states: “Energy savings shall be determined by measuring and/or

estimating consumption, before and after the implementation of the measure,...”

For bottom-up methods, the reference situation ‘before’ the measure is called the

baseline.



For top-down methods, energy savings are calculated from the difference between

the actual value of an indicator and the value of the indicator for the same year that

would have materialised in a reference trend.

We have chosen to differentiate the name of the reference situation in bottom-up and

top-down methods, since they are not the same thing. The following sections on

bottom-up and top-down methods will expand on how to define and calculate baselines

and reference trends.

2.2.3 Addressing harmonisation issues

A harmonised model of bottom-up and top-down calculation methods should be

developed and used for the ESD reporting (cf. ESD article 15). Harmonisation should

give a reasonable freedom for the Member States (following the principle of

subsidiarity), while the results reported can be compared. Therefore, the methods and

the 20 bottom-up and 14 top-down case applications developed by the EMEEES

project are a starting point, but these methods and applications are not intended to

exclude the use of own methods and further methods for other sectors, end uses, and

kinds of energy services and energy efficiency improvement measures by the Member

States. However, harmonisation should be ensured by key elements proposed by

EMEEES: a general structure both for the documentation of bottom-up and top-down

energy savings and for the calculation itself, with the selection of baseline and baseline

parameters, reference trends, as well as correction factors, and a dynamic approach to

ensure improvement over time. In bottom-up measurement, a three-level approach

has been proposed by EMEEES to facilitate such improvement over time (cf. chapter

2.3.2).

These EMEEES proposals were based on past experiences and existing literature (e.g.

CPUC 2006, SRCI et al. 2001, TecMarket Works et al. 2004, Vreuls et al. 2005), taking

account of the ESD specificities. Bottom-up and top-down methods can both be used

for calculating ESD energy savings. In order to avoid “adding up apples and oranges”,

the key elements for top-down and bottom-up should also be mutually consistent.

EMEEES findings on how to achieve such consistency will be presented in chapter 2.7.

The development of a harmonised model is a learning process, and the methods

should be improved in the future as more experiences from Member States become

available and lessons are learned.

In the ESD process, the EMEEES results are not to be directly compulsorily used by

the Member States. They are inputs to the work of the Commission and the ESD

Committee. According to the harmonisation level needed for the ESD implementation,

the decisions from the Commission and the ESD Committee may correspond to

different levels of requirements (“could, should or shall”). It is therefore necessary to

clarify what level of requirements the different EMEEES proposals correspond to. We

hereafter distinguish supporting resources, reporting check-list and general principles,

as described in the table below.



Table 1: Three main categories of methodological outcomes.

Supporting Resources Reporting Checklist General Principles

Concrete evaluation methods Member States COULD use when they are looking for technical support.

(example of provided information: examples of algorithms, formulae, or data commonly used to calculate a baseline for heating systems)

List of questions Member States SHOULD answer in their future NEEAP to provide a consistent set of information about how they assessed their energy savings results.

(e.g.: reporting what data were used to calculate the baseline values)

Harmonised rules Member States SHALL apply when evaluating their energy savings results.

(e.g.: update frequency for baselines)

To be available for all Member States (no need for decision)

To be discussed by the ESD Committee (but no need for decision unless its use should be compulsory)

To be decided by the European Commission and the ESD Committee

From specific issues… …To general issues

The supporting resources are made available by the Commission to Member States.

These materials are mainly developed by Intelligent Energy Europe projects, such as

EMEEES, for concrete evaluation methods and pilot tests. Data on average annual

energy consumption (for equipment stocks or markets) can also be found in

preparatory studies for implementing the EuP (Energy-using Products) Directive

(2005/32/EC). As these resources are not mandatory, they do not require a decision

(validation) from the ESD Committee.

The reporting checklist is to address issues that do not necessarily need to be

harmonised at an EU level, but that are relevant when evaluating energy savings. This

checklist is a quality assurance (on data, sources, etc.) that would enable the

Commission to compare data provided by the Member States on their achieved energy

savings. An example of such a checklist can be found in (Vine and Sathaye, 1999).

The checklist specific to ESD proposed by the EMEEES project will have to be

validated by the European Commission and is included in Appendix 2 and 3.

The checklist does not require Member States to apply a given method nor to include

all possible issues in their evaluations. But they are asked to report whether they

address the listed issues, and how. By pinpointing the main evaluation issues, the aim

is to induce better evaluation designs. And by structuring the evaluation reporting, the

checklist will also facilitate the collection and analysis of experience to share between

Member States.

General principles correspond to the major priority issues, for which harmonisation is

required in order to achieve a harmonised evaluation system for all Member States.

Their application will be mandatory, so they require a consensual decision from the

ESD Committee and the Commission.



These principles are proposed, e.g., by the ESD Working Groups11. The EMEEES work

provided analyses about possible options that might be considered in these decisions.

We hope that the EMEEES proposal to distinguish these three levels of requirements

(“could, should or shall”) will be useful, as it focuses the debates in the Committee on

the highest level (i.e. general principles) and therefore limits the discussions to the

main issues. At the same time, national representatives are reassured to see that for

lower requirement levels they retain freedom on how to manage ESD implementation

in their country.

2.2.4 What is the subject of an evaluation method?

From the definitions provided by the ESD, it is not directly clear what an "energy

efficiency improvement (EEI) measure" is, as this is presented in Article 3, Definition

(h) as “all actions that normally lead to verifiable and measurable or estimable energy

efficiency improvement”. This can be very broad, as Annex III of the ESD, Indicative list

of examples of eligible energy efficiency improvement measures, starts with “This

Annex provides examples of areas in which energy efficiency improvement

programmes and other energy efficiency improvement measures may be developed

and implemented in the context of Article 4”. Many of these examples of areas are

technical, organisational, or behavioural action taken at an end-user’s site (or building,

equipment, etc.) that improve the energy efficiency of that end-user’s facilities or

equipment, but some of the examples given are also types of energy services, EEI

programmes, or other policy instruments (as Article 4 and Annex I 1(d) state, the

national indicative energy savings target shall: “be reached by way of energy services

and other energy efficiency improvement measures”).

We therefore make an analytical clarification on terms for more precisely presenting

the subject of evaluation in the EMEEES project:

• An end-use energy efficiency improvement action (end-use (EEI) action) is a

technical, organisational or behavioural action taken at an end-user’s site (or

building, equipment, etc.) that improves the energy efficiency of that end-user’s

facilities or equipment, and thereby saves energy. It can be a result of a facilitating

measure.

• An energy efficiency improvement facilitating measure ((EEI) facilitating measure)

is an action by an actor that is not the final consumer him-/herself, which supports

the final consumer in implementing an end-use action, or implements it for the final

consumer. Examples are energy efficiency programmes, energy services or EEI

mechanisms.

Figure 2 shows how some evaluation methods can focus on one type of end use or

end-use action subject to several facilitating measures (e.g., efficient boilers and

11 To facilitate the decisions of the ESD Committee, two working groups were created to examine the

most important issues respectively related to bottom-up and top-down evaluation approaches.



pumps for tertiary heating systems), or on a number of end-use actions covered by one

facilitating measure (e.g., energy performance contracting).

Figure 2: End-use actions and facilitating measures

2.3 Bottom-up calculation methods

The ESD specifies (Annex IV):

„A bottom-up calculation method means that energy savings obtained through the

implementation of a specific energy efficiency improvement measure are measured

in kilowatt-hours (kWh), in Joules (J) or in kilogram oil equivalent (kgoe) and added to

energy savings results from other specific energy efficiency improvement measures.“

EMEEES started its work on bottom-up methods with a report on general methodology

(Broc et al. 2009, Definition of the process to develop harmonised bottom-up

evaluation methods)12. It presents the four steps in bottom-up evaluation (cf. section

2.3.1) and the five general bottom-up methods (cf. section 2.3.3) in detail. EMEEES

partners followed this methodology in the development of the 20 bottom-up case

applications (cf. section 2.4). Appendix 1 lists the individual reports on the case

applications. An overview report (Vreuls et al. 2009)13 with several Appendices

summarises the findings.

The key elements proposed by EMEEES for bottom-up calculations (four steps, three

levels, five types of methods, and three baseline situations, cf. Chapter 2.7.3) form a

framework that is a good basis for a transparent reporting / documentation of the

energy savings. Such transparency is the first required condition for a harmonised 12 http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/D4_EMEEES_Final.pdf 13 http://www.evaluate-energy-

savings.eu/emeees/en/publications/reports/EMEEES_Bottom_up_draft_overview081006.pdf



evaluation system. This was confirmed in the EMEEES pilot tests (cf. Chapter 3.4), in

which the use of this framework enabled describing the tested measures in a compre-

hensive and transparent way.

2.3.1 Four steps in the calculation process

The harmonised rules for bottom-up evaluation methods are organised around four

steps in the calculation process (see figure 3). These steps and their sub-steps are

presented in detail in a separate report (Broc et al. 2009, The development process for

harmonised bottom-up evaluation methods of energy savings)14 and are used in each

case application.

Figure 3: A four steps calculation process.

* the free-rider effect will only be relevant, if the aim of the evaluation is to calculate energy savings additional to those that energy consumers, investors, or other market actor would have achieved by themselves anyway, cf. section 2.7.3. This effect is not mentioned in the ESD.

Bottom-up methods start from calculating annual energy savings for one final

consumer or one piece of equipment. These so-called unitary gross annual energy

savings can normally not be directly measured but need to be calculated from the

difference between the energy-efficient situation after an energy efficiency

improvement measure and a hypothetical baseline. For example, the savings for a

specific dwelling are the calculated or measured gas use after a thermal insulation

14 http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/D4_EMEEES_Final.pdf

Step 1: unitary gross annual energy savings (in

kWh/year per participant or unit, average or individual) Example: how much energy is saved annually by using an A+

fridge instead of an A fridge?

Step 2: total gross annual energy savings (taking

into account the number of participants or units, in kWh/year) Example: how many A+ fridges were sold (within the EEI

programme)?

Step 3: total ESD annual energy savings in the

first year of the EEI measures (taking into account

double counting, multiplier effect, and other gross-to-net

correction factors, in kWh/year) Example: how many A+ fridges are promoted by more than

one EEI programme and might be double-counted?

Step 4: total ESD energy savings achieved in the

year 2016 (in kWh/year, taking account of the timing of the

end-use (EEI) action, and its lifetime)

Example: how many A+ fridges due to the programme are still

in use in 2016?

+ timing and lifetime (within

ESD period)

+ double counting, multiplier

effect, + other gross-to-net

correction factors (e.g., free-

rider effect*)

+ summing up across

participants or units



measure compared to the calculated or measured gas use before, normalising

measured values for fluctuation in heating degree days. In some cases, the choice of

the baseline is decisive for whether all or additional savings will be calculated (cf.

section 2.7.3).

Then these so-called unitary energy savings per consumer or equipment are added

together for all consumers or equipment affected by an energy efficiency improvement

measure. However, the resulting total gross annual energy savings need to be

corrected by some factors. The ESD requires avoidance of double counting but

accounting for multiplier effects15. It does not mention correction for free-rider effects,

i.e., savings by consumers who would have taken the action without energy efficiency

programmes, energy services, and other energy efficiency policies. Correcting for free-

rider effects or not is, therefore, another element in the calculation of all or additional

energy savings (cf. table in section 2.7.3 for details on bottom-up calculations,

baselines, and correction factors).

Two general formulas can be derived from this four-step process for the total ESD

annual energy savings in the first year (cf. also SRCI et al. 2001, p. 65):

1. If average unitary gross annual energy savings for a unit of end-use action can be

defined, the formula will be:

total ESD annual energy savings = average unitary gross annual energy savings per equipment (or participant) * number of equipment (or participants) * (1 - free-rider fraction° + multiplier fraction) * (1- double-counting factor/fraction)

° only if additional energy savings are calculated Equation 1a

2. If individual unitary gross annual energy savings for one (usually larger) final

consumer benefitting from an energy efficiency improvement measure (called a

participant) have to be used, the formula will be:

total ESD annual energy savings = sum of individual unitary gross annual energy savings per participant * (1 - double-counting factor/fraction (average or individual) ) * (1 - free-rider fraction° + multiplier fraction)

° only if additional energy savings are calculated Equation 1b

15 The multiplier (or spill-over) effect enhances the initial effect of EEI measures. According to Annex IV-5

of the ESD the multiplier effect means that “the market will implement a measure automatically without any further involvement from the authorities or agencies referred to in Article 4-4 or any private-sector energy services provider”. The free-rider effect regards market actors who make use of facilities or support, provided for by EEI programmes, policies, or energy services, but would have taken energy-saving actions anyway. It is not mentioned in the ESD and only relevant when calculating additional energy savings.



In both cases of the formula:

• free-rider fraction: share of free-riders between 0 and 1

• multiplier fraction: equivalent to spill-over effect, 0

• double counting factor/fraction: coefficient or fraction between 0 and 1

2.3.2 Three levels of harmonisation

To be as practicable as possible and stimulate continued improvement, the harmo-

nised reporting on bottom-up evaluation is structured on three levels (see figure 4).

Figure 4: Three levels of harmonisation

Data scale Main data sources

Data processing and

documenting

Level 1 European default

values

existing/available

European regulation,

studies and statistics

Reliability coefficient according

to the data basis for the default

value

Level 2 National

representative values

up-to-date national

statistics, surveys,

samples, registries

requirements = minimum set of

data and justifications to be

reported

Level 3 Programme- or

Participant-specific

specific monitoring

systems, registries,

surveys, measurements

requirements to report on the

specific data and justifications in

detail (standard report at least

available)

These three levels correspond to the three main cases (1, 2, or 3) that may occur

when a Member State wants to evaluate the energy savings related to a given EEI

measure:

1 the Member State has only a few data about this measure (e.g. number of

participants) and needs other data sources to complete the evaluation: for that

case, the proposal is to provide MS with European default values (= level 1

evaluation) ;

2 the Member State can evaluate the energy savings by using mainly data available

at national level (e.g. national statistics or surveys): for that case, the proposal is to

provide MS with general guidelines (for ensuring harmonisation at the EU level)

(= level 2 evaluation) ;

3 the Member State can evaluate the energy savings by using mainly data specific

to the evaluated measure (e.g. registry of participants’ data): for that case, the



proposal is to provide MS with detailed guidelines (for ensuring harmonisation at

the EU level), (= level 3 evaluation).

The basic idea behind these three levels is that the more evaluation efforts a Member

State makes, the more accurate the results, and so the more rewarded/recognised the

results should be by the Commission. This approach is indeed to induce a

progressive improvement of the values used by the Member States, rewarding their

evaluation efforts. It is also assumed to be a cost-effective way to address the

uncertainties related to energy savings evaluations. This approach makes it possible to

learn from experience and improve the methods over time.

2.3.3 Five general bottom-up methods

From the general literature on evaluation of energy savings, e.g., CPUC (2006), SRCI

(2001), and ESD Annex IV, five general bottom-up methods have been identified.

Table 2 presents these methods, along with some experience on their costs. The first

two methods rely on measured data. However, due to uncertainties that are always

associated with the determination of baselines, this does not necessarily mean their

results are more accurate than those of the methods 3 to 5. These latter methods are

based on estimates. The accuracy of the methods generally decreases from the

enhanced engineering analysis (method 3) down to the deemed estimate approach

(method 5). In many cases, particularly for large facilitating measures or evaluations,

the first three methods are used on samples to provide the inputs to deemed or ex-post

average savings that are then applied to the total number of participants. This way of

combining evaluation levels and methods is the practice encouraged by EMEEES, as

this offers a great level of flexibility, making it possible to target the evaluation efforts

where they are needed the most and/or where they are the most cost-effective.

Table 2. Classification of bottom-up evaluation methods for energy savings

Methods for

measuring or estimating unitary gross annual energy savings

Methods for

collecting number of units or participants

Methods for

estimating gross-to-net correction factors

Applicable if unit is:

Characterisation of

costs and data collection

1 direct measurement

a) without normalisation

b) with normalisation

A) monitoring of participants and savings per participant

I) and II) participant (usually)

can be costly; suitable for large buildings or sites, or as a basis for deemed estimates

2 analysis of energy bills or energy sales data (sample or all participants)

a) without normalisation

b) with normalisation

A) monitoring of participants and savings per participant

I) and c) comparison with control group;

or d) discrete choice modelling and other in-depth billing analysis

participant (usually)

can be very costly to collect and analyse, particularly d); may be the only way for information campaigns

3 enhanced

engineering estimates for

A) monitoring of participants number of actions

I) and II) participant or specific end-use

can be costly; however, if an energy audit or certification is



Methods for

measuring or estimating unitary gross annual energy savings

Methods for

collecting number of units or participants

Methods for

estimating gross-to-net correction factors

Applicable if unit is:

Characterisation of

costs and data collection

individual units

(e.g., calibrated simulation)

and savings per participant/action

action/ equipment

done anyway, small extra cost of monitoring results

4 Mixed deemed and ex-post estimate, e.g. based on sales data, inspection of samples, monitoring of equipment purchased by participants

A) monitoring of number of actions and savings per action

I) and II) specific end-use EEI action/ equipment (usually)

costs depend on level of accuracy and gross-to-net correction required; monitoring usually straightforward

5 Deemed estimate, e.g. based on sales data, inspection of samples before implementation of the facilitating measure being evaluated

A) monitoring of number of actions and savings per action

maybe II; always simplified;

maybe inclusion of correction factors in deemed savings per unit

specific end-use action/ equipment (usually)

costs can be quite low, monitoring of number of actions and savings per action may be combined with ”anyway” contacts

Typical methods for estimating gross-to-net correction factors (i.e., multiplier, double-counting and, when calculating additional energy savings, free-rider effects) are:

I) surveys of participants (and control group and other market actors) to find out reasons for implementing end-use actions

II) monitoring of participants and end-use actions for different promotion measures to avoid double-counting

It will often be possible to gather the necessary data at quite limited costs, if the

monitoring is planned before implementing an EEI measure. E.g., in an energy audit

programme, only a database has to be created tracking measures proposed in the

audits. Even participant surveys can be combined with the contacts occurring anyway

to provide an EEI measure to the participants. Furthermore, it will only be necessary to

evaluate the influence of the whole package of (EEI) facilitating measures targeting a

specific end use or type of end-use (EEI) action. For the ESD, there is no need to

distinguish, e.g., the energy-saving effects of an information campaign on energy-

efficient lighting in tertiary buildings from the effects of an energy audit programme

and/or a financial incentive programme targeting the same subject. If these

programmes are offered by different actors, it will be their problem if they wish to

distinguish their contributions between each other, but not the Member State’s duty vs.

the ESD. It is, therefore, a task for the analysis of each specific case application (cf.

Table 3) to find a solution for the monitoring that is a good compromise between

evaluation cost and accuracy. In Table 2, only a very broad characterisation of the

costs and data collection issues can be given based on experience, which should be

treated with caution.

In addition to the five general bottom-up methods, there is also the possibility to use

bottom-up modelling of the whole stock or market for one end use. This will be a

bottom-up method, if there are surveys of a sample of final consumers, who are asked



about the end-use actions they have taken and by which facilitating measures these

were influenced.

2.4 EMEEES bottom-up case applications

Each of the 20 bottom-up case applications developed by EMEEES has been

described in a separate report. There is also a compilation of the formulas for unitary

energy savings and the EU level default values that can be used for level 1 calculations

available as an appendix to the overview report (Vreuls et al. 2009)16.

The case applications were selected so as to cover all end-use sectors and some

important multi-sector types of facilitating measures.

Table 3 presents the subjects of the case applications, the sectors covered, whether a

level 1 calculation is possible at all, and whether EMEEES was able to propose a

default value for this. The table then shows, which of the five main methods is used for

level 2 or 3 calculations.

Table 3: Overview of EMEEES Bottom-up case applications

End-use, end-use action, or facilitating measure

Sector

Le

ve

l 1

po

ss

ible

De

fau

lt v

alu

e f

or

sa

vin

gs

by

EM

EE

ES

Method(s) proposed for

Level 2 calculations (and Level 3 where required for larger systems)

Building regulations for new residential buildings

Residential no no Mixed deemed and ex-post

Improvement of the building envelope of residential buildings

Residential no no Mixed deemed and ex-post

Biomass boilers Residential no no Mixed deemed and ex-post

Residential condensing boilers in space heating

Residential yes yes Deemed Savings

Energy efficient cold appliances and washing machines

Residential yes yes Deemed Savings

Domestic Hot Water – Solar water heaters Residential no no Mixed deemed and ex-post

Domestic Hot Water – Heat Pumps Residential no no Mixed deemed and ex-post

Non residential space heating improvement in case of heating distribution by a water loop

Tertiary yes yes Deemed Savings (Enhanced Engineering)

Improvement of lighting systems Tertiary (industry) yes yes Deemed Savings

Improvement of central air conditioning Tertiary yes yes Deemed Savings

16 http://www.evaluate-energy-

savings.eu/emeees/en/publications/reports/EMEEES_Bottom_up_draft_overview081006.pdf



End-use, end-use action, or facilitating measure

Sector

Le

ve

l 1

po

ss

ible

De

fau

lt v

alu

e f

or

sa

vin

gs

by

EM

EE

ES

Method(s) proposed for

Level 2 calculations (and Level 3 where required for larger systems)

Office equipment Tertiary yes yes Deemed Savings

Energy-efficient motors Industry yes no Deemed Savings (Direct Measurement)

Variable speed drives Industry yes yes Deemed Savings (Direct Measurement)

Vehicle energy efficiency Transport yes no Deemed Savings

Modal shifts in passenger transport Transport yes no Mixed deemed and ex-post

Ecodriving Transport yes yes Deemed Savings

Energy performance contracting Tertiary and industry end-uses

no no Mixed deemed and ex-post

Energy audits Tertiary and industry end-uses

yes yes Enhanced engineering

Voluntary agreements – billing analysis method

Tertiary and industry end-uses

no no Billing analysis

Voluntary agreements with individual companies – engineering method

Tertiary and industry end-uses

yes yes Enhanced engineering

2.5 Top-down calculation methods

„A top-down calculation method means that the amount of energy savings is calculated

using the national or larger-scale aggregated sectoral levels of energy savings as

the starting point“

In other words, top-down methods rely on energy efficiency indicators calculated

from national statistics (also called ‚top-down indicators’, e.g., ODYSSEE indicators).

There are several types of indicators:

• Specific energy consumption indicators for a well-defined type of new appliance,

equipment, or vehicle, measuring the average energy consumption of the sold

equipment or the equipment stock in energy/appliance/year or energy/km

• Unit energy consumption indicators of a sub-sector or sector, e.g.,

electricity/employee/year in the tertiary sector, process fuels per ton of cement,

heating energy/m2 of dwelling/year

• Indicators on the diffusion of energy-saving technologies, such as m2 of solar

thermal collectors, or energy-efficient transport modes, such as the share of trains

and ships in goods transport.



Furthermore, a special econometric method based on the analysis of price elasticities

can be used to evaluate the effects of energy taxation from any indicator.

The analysis of top-down methods done by EMEEES is presented in a summary report

(Lapillonne et al. 2009)17 with a separate Annex presenting the ODYSSEE indicators in

more detail18, and a second summary report on the top-down cases analysed in

EMEEES (Lapillonne/Desbrosses 2009)19.

With top-down methods, the overall energy savings are calculated from the difference

in the current value of a particular statistical indicator used in a certain year, and the

hypothetical value that is calculated for that year from a reference trend assumed.

The simplest form of a reference trend is to take the value of the indicator in a base

year as the reference. For example, if the average amount of gas use per dwelling

decreases with respect to a base year, the difference is taken as energy savings. The

resulting energy savings have been called ‘total’ savings (however, ‘apparent total’

savings would be a better name), and the assumption is easily made that these are

equivalent to all energy savings.

However, this intuitive assumption is only meaningful for indicators that have the ‘right’

trend over the years, a trend towards higher energy efficiency. But that is only the case

for about 60 % of all the 14 indicators and countries analysed in EMEEES. For some

indicators, there are all cases of countries with a decreasing, increasing, or stable

trend, cf., e.g., figure 5. This is because there are structural effects that also lead to

changes in the indicator value but have nothing to do with energy efficiency. Therefore,

these structural effects need to be corrected before calculating energy savings, if

possible with a reasonable effort. Such correction could be done by bottom-up

modelling of some of the effects to correct them. With all structural effects removed,

‘apparent total’ energy savings should be equal to all energy savings. It may, however,

be difficult to judge from the results whether all structural effects have been removed,

and it may be costly to do the correction.


savings.eu/emeees/en/publications/reports/EMEEES_WP5_Summary_report_May_2009.pdf 18 http://www.evaluate-energy-

savings.eu/emeees/en/publications/reports/EMEEES__WP5_TD_indicators_overview_final.pdf 19 http://www.evaluate-energy-

savings.eu/emeees/downloads/WP_5_EMEEES_case_studies_report_Final.pdf



Figure 5: Unit energy consumption per m for heating (koe) in the residential sector for three EU Member States

Note: we have selected three Member States showing different trends for this indicator. There

are, however, many other Member States showing similar trends.

An equivalent way, in principle, could therefore be to calculate the reference trend for

all energy savings from bottom-up modelling of the energy consumption underlying the

indicator, with zero energy efficiency changes in the model. However, the feasibility of

this approach was not tested in EMEEES.

For calculating additional energy savings using top-down methods, the approach

taken in EMEEES is a regression analysis of past trends of an indicator that would

reflect the autonomous changes. This was conclusive in some cases but not in others.

In those latter cases, again, bottom-up modelling of the energy consumption underlying

the indicator and the structural changes may provide a way forward, but EMEEES was

not able to test it (cf. table 5 in Section 2.7.3) for details on top-down calculations and

correction factors).

Using such regression analysis allows to evaluate energy savings compared to an

autonomous trend, even if the trend of the underlying indicator does not allow to

calculate ‘apparent total’ energy savings.

Simple econometric methods were used to quantify the impact of energy market prices and trends, on purpose, taking into account several criteria:

• the need for transparency and of harmonisation among countries,

• the easiness of implementation and of their understanding, as such methods would

ultimately need to be applied by the countries;



• finally, the data limitations, in particular for additional explanatory variables (e.g.,

price/tax on cars, cost of equipment) and the uncertainty of the data handled.

The typical regression equation considered was follows:

ln ES = a + b T + c ln P + d ln A + K

with : ln : logarithm; ES: energy saving indicator; a: a constant; b: trend; T: time; P : energy price; c : price elasticity20 ; A: macro economic variable (e.g. GDP) to capture the impact of business cycles; d : elasticity to GDP; K: constant coefficient

Equation 2

The estimate of the regression coefficient is made over a period ending before the

effects of facilitating measures will have to be assessed (e.g., before 1995). Then using

the coefficient, the impact of the different effects can be removed over the period on

which the ESD savings will be calculated (i.e. 2008-2016) (Figure 6). The price effect

can be separated into two components, ex-tax energy price (market component) and

energy tax (policy component), using the same price elasticity .

Figure 6: An example of the calculation of changes in an indicator vs. the reference trend determined through regression analysis (indicator on modal shares in goods transport)

2.6 EMEEES top-down cases

2.6.1 Additional energy savings

Energy savings that are additional to an autonomous trend and to energy savings due

to increases in market energy prices could, in principle, be evaluated from long time-

20 Price elasticity may be differentiated between upward and downward price elasticity.



series of indicators via a regression analysis (cf. the formula in chapter 2.5). This was

our starting hypothesis: a regression over past periods, without facilitating measures in

place, would deliver the baseline projection. Extending this reference trend over the

period 2008-2016 would allow to calculate ESD energy savings by comparison with the

actual development of the indicator.

The positive features of such an approach are twofold:

(1) it allows calculating additional energy savings from statistical data only and

(2) it allows calculating additional energy savings even for indicators that do not

allow to calculate ‘apparent total’ energy savings, since the unit or specific energy

consumption value is increasing or the diffusion of energy-efficient technologies or

transport modes is decreasing (as, e.g., in figure 6).

However, the analysis of case studies was quite inconclusive. In most cases, it was

possible to identify a reference trend for some countries but not for all. An exception

here is the market diffusion of solar water heating, which can be assumed to be

practically entirely due to facilitating measures in the past, so the baseline would be

zero market penetration. Whether this holds for the future, however, would need to be

analysed.

The same picture showed up for the correction for market energy prices. Therefore, the

next hypothesis was to consider using EU default values for both the correction for

market energy prices, and for the autonomous trend for the specific energy

consumption indicators.

• For market energy prices, the EMEEES proposal is to use price elasticities

between 0.1 and 0.2, and only correct for the effects of market energy price

increases.

• For the autonomous trend of specific energy consumption indicators (e.g., for

cars and appliances), the proposal is to use the average trend obtained for the

three countries with the slowest decrease (i.e., lowest percent change per year)

in the value of the indicator. This is based on the assumption that these would be

countries without (strong) national EEI measures in place. Such EU default values

for the autonomous trend of specific energy consumption indicators should be

harmonised with any corresponding EU default values for the percent change per

year of the baseline to be used for bottom-up evaluation methods for the same

type of equipment. The value achieved through the top-down analysis would be the

starting point for such a harmonisation.

Such a default value was developed for the average fuel consumption of cars (cf.

Lapillonne et al., 2009). Due to budget and data constraints, it was not possible

within the EMEEES project to really develop EU average default or country-specific

values for the autonomous development of other specific energy consumption

indicators (e.g., for appliances).



For unit energy consumption or diffusion indicators, countries are usually so

different that it will not be possible to define EU default values for reference trends.

Country-specific trends must be defined. For some countries and indicators, this may

be done using the regression analysis method, as said above.

Furthermore, not all indicators are available for all or most EU Member States (cf.

table 4).

2.6.2 All energy savings

In theory, if all structure effects influencing an indicator are corrected for, all energy

savings can be calculated from the difference between the indicator value in the base

year and the current value of the indicator in the measurement year (e.g., 2016). All

energy savings would then be the same as the ‘apparent total’ energy savings found in

practice.

However, as the analysis of case studies has shown, the indicator is going into the

‘right’ direction to show ‘apparent total’ energy savings at all only for about 60 % of all

the 14 indicators and countries analysed in EMEEES. The reason must be that there

are still some structural effects not yet removed due to lack of data.

Therefore, in practice, it may only be possible for some specific energy

consumption indicators to assume that ‘apparent total’ energy savings are a good

approximation for all energy savings. For all other types of indicators, this would lead to

no savings at all and/or to inconsistent and arbitrary measures of energy savings

between Member States.

2.6.3 Applicable top-down calculation methods

In conclusion, Table 4 summarises, which top-down calculation methods based on

ODYSSEE indicators were analysed in EMEEES, and which of these appear

applicable for a harmonised calculation system for the ESD. These are the five marked

‘yes’ in the column ‘Applicable’ in the table. Three are marked with a “sometimes”, as

they may be applicable depending on the country situation. The method for general

energy taxation cannot be done for a TD indicator, if a correction for energy market

price and a calculation of the energy savings due to taxation is already done in the

case application for that indicator. Otherwise, the effect of energy taxes will be counted

more than once (thus the ‘yes*’).



Table 4: Applicable ‘pure’ top-down methods, if data available and corrections possible

As part of WP5, the data availability has been checked for each indicator/top-down-

method and each MS (cf. Appendix 1 in Lapillonne et al. 200921). The column ‘Data

MS’ gives an overview, with all = 27, most = >22, many >15, EU-15 = 10 to 15 and few

<5. In some cases, the indicators are currently available only for EU-15, but potentially

for all or most of the EU-27 countries. This is mentioned in brackets, e.g., ‘EU-15 (all)’.

The column ‘Robust results?’ refers to whether the regression analysis has delivered

conclusive results, and to methods with specific energy consumption indicators that

can be used with a default value for the autonomous trends.

The energy savings calculated with the two methods New cars and Vehicle stock must

not be added to each other, since the energy savings obtained with Vehicle stock

include those obtained with New cars.

The summary report on the EMEEES top-down case studies (Lapillonne/Desbrosses

2009)22 presents more detail on how to calculate energy savings with the applicable

methods.

2.7 Integration: selection and consistency of methods

Looking at the analysis of bottom-up and top-down methods presented above, for

which end uses and sectors is it most appropriate to use bottom-up and or top-down

methods? Which types of bottom-up or top-down methods would, in principle, be


savings.eu/emeees/en/publications/reports/EMEEES_WP5_Summary_report_May_2009.pdf 22 http://www.evaluate-energy-

savings.eu/emeees/downloads/WP_5_EMEEES_case_studies_report_Final.pdf



appropriate for which type of EEI measure? And how can the results of bottom-up

and top-down methods be made comparable when calculating either all or additional

energy savings? Our replies to these questions can be found in this section.

2.7.1 Selection of bottom-up or top-down methods by end use or sector

On the one hand, the EMEEES project has looked at which methods can be applied for

which sector and/or end use. Our recommendation is as follows:





effects of the whole package of measures, including multiplier (market

transformation) effects.

Bottom-up calculations are possible for appliances and vehicles, too, but it is

often difficult to calculate multiplier (and free-rider) effects with them.

For top-down calculation on appliances and vehicles, a reference trend can be

defined for these specific energy consumption indicators to calculate additional

energy savings; this reference trend should either be a EU harmonised trend or

a national trend based on an EU harmonised coefficient (e.g., for elasticity to

increases in market energy prices23). Furthermore, the base year value may be

assumed to be a proxy for the correct reference trend for calculating all energy

savings for these indicators.

• Top-down methods are the way to calculate the effects of energy taxation24

and add them to the effects of bottom-up calculations for a sector, but only if

these bottom-up calculations exclude free-rider effects. The energy savings due

to taxation must not be added to results of top-down calculations on sectors or








In these areas, structural effects can often not be corrected for in top-down

indicators, or it will need costly bottom-up modelling and gathering the

necessary data for that modelling to do the required corrections. Neither of the

23 We found that it was usually not statistically meaningful to calculate national values for the price

elasticity of the different indicators. Therefore, we propose to use EU harmonised coefficients, although price elasticity is likely to differ between Member States, cf. chapter 2.6.1.

24 This includes general taxes on energy, but not fiscal incentives targeting specific energy efficiency investments, such as tax deductions or rapid depriciation rules. These incentives are specific measures and should be covered by calculations specific to a sector, end use, or type of end-use action.



two reference trends mentioned above for appliances and vehicles are usually

possible for top-down indicators measuring the energy consumption of a sector

per unit of production or per employee. For indicators measuring the diffusion of

energy-efficient transport modes or combined heat and power in industry, the

situation will depend on the country. For most of these sectoral or diffusion

indicators, some Member States may see ‘apparent total’ savings when

comparing the current value of an indicator with its value in the base year, while

others may not. The reason for this is that either these countries really do not

have savings, or their savings are hidden by structural changes that cannot be

corrected for due to lack of data. Using ‘apparent total’ savings as a proxy for all

energy savings for these top-down indicators would, therefore, lead to

inconsistent and arbitrary measures of energy savings between Member States.

This will disable the use of top-down methods in such cases.

By contrast, bottom-up calculations are usually feasible.

This recommendation is based on our analysis of case applications for bottom-up and

top-down methods (cf. sections 2.3 to 2.6), as well as on practical experience in many

countries and our pilot tests (cf. section 3.4). They are based on the general trend of

findings from these sources. For example, we estimate that with our total set of

bottom-up case applications, more than 90% coverage of the energy use subject to

the ESD can be achieved.

Bottom-up calculation needs specific monitoring but can provide information on the

effectiveness and cost-effectiveness of measures, on potential improvements, and on

greenhouse gas emission reductions additional to baseline projections. However,

calculation of multiplier and free-rider effects with bottom-up methods can be costly,

particularly for appliances and vehicles, for which the multiplier effects are particularly

important. Furthermore, they work into the opposite direction, hence partly cancel out

each other. Calculation of both multiplier and free-rider effects could, therefore, be

restricted to measures either yielding at least 40 million kWh of annual electricity

savings or 100 million kWh of annual energy savings of other fuels, or at least 5

percent of a Member State’s ESD energy savings target.

Top-down calculation starts from using existing statistical data and can be easier to

apply, particularly in areas, for which many and overlapping energy efficiency

improvement measures exist. However, it is often difficult to define the reference trend

as stated above, or the indicator is not showing energy savings at all without costly

corrections.

Therefore, the quality of data available in a country will finally determine which





2.7.2 Selection of bottom-up or top-down methods by type of EEI

measure

On the other hand, the EMEEES project has also looked at which types of bottom- up

or top-down methods would, in principle, be appropriate for which type of EEI measure.

Here, the focus was on the type of (EEI) facilitating measure, which can often target

several end uses or sectors. The task has been to classify the energy efficiency

improvement measures (EEI measures) to save energy by the type of evaluation

method that is most appropriate to be used for each measure. The answer to the

question, which type of evaluation method is most “appropriate” for an EEI measure

does not only depend on the type of instrument, it also depends on the availability of

data, which is country-specific.

For this purpose, a concrete classification of EEI measures has been derived. This

classification is based on the classification proposed in the frame of the Odyssee-

MURE project, which is implemented on the Internet (www.mure2.com). Its six main

categories of measures are:

1 Regulation

2 Information and legislative-informative measures (e.g. mandatory labelling)

3 Financial instruments

4 Voluntary agreements and Co-operative instruments

5 Energy services for energy savings

6 EEI mechanisms and other combinations of previous (sub)categories

Appendix 5 presents the main results as an overview by sector in the form of

matrices of EEI measure type vs. evaluation methodology. It can be used as a source

of information in particular for those types of measures not covered by the detailed

EMEEES bottom-up case applications and top-down cases.

The detailed results are available in the report (Eichhammer 2008, Distinction of

energy efficiency improvement measures by type of appropriate evaluation method)25.

They have also been included in the MURE database26.

2.7.3 Consistency between bottom-up and top-down methods

As we recommended in section 2.7.1, the ESD monitoring system can include both

bottom-up or top-down methods for monitoring and evaluation with one Member State.

In order for it to be a harmonised system, the results of either bottom-up or top-down

calculation must be consistent and comparable with each other. This requires that

the elements of calculation need to be chosen in a consistent manner for both

bottom-up and top-down calculations, and for the two evaluation targets introduced

above: additional energy savings and all energy savings.

25 http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP3_Report_Final.pdf 26 http://www.isisrome.com/mure/



This section serves to present the elements that would ensure consistency, in the

table 5 below. It must be noted that only the elements of bottom-up and top-down

calculations in either of the two rows of the table: additional energy savings and all

energy savings, respectively, are consistent with each other. Using the elements of

bottom-up calculation from one and those of top-down from the other row of the table

would be highly inconsistent.

Table 5: Elements of calculation for the evaluation of additional or all energy savings that will ensure consistency between bottom-up and top-down methods

Evalu-

ation

target

Elements of bottom-up

calculation

Elements of top-down calculation

Addi-

tional

energy

savings

Case 1: replacement of existing equipment

Baseline = Without measure situation (market baseline)

Case 2: add-on energy efficiency investment without replacement of existing equipment or building

Baseline = Before action situation

Case 3: new building or appliance: the before situation does not exist and a reference has to be created.

Baseline = A reference situation° (e.g., (2) the existing market)

Apart from avoiding double-counting and taking multiplier effects* into account, also free-rider effects* should be analysed in principle

Case a): for specific energy consumption indicators related to an end-use equipment (e.g., cars, refrigerators): Reference trend = EU default value (based on a regression analysis for all countries with data available, and on the average of the three countries with the slowest trend found in the analysis)

Case b): for other types of indicators (unit energy consumption of sectors, diffusion indicators): b1) if possible, Reference trend for one country = extrapolation of historical trend before measures (from regression analysis for each country) b2) otherwise, the only option that appears consistent, however, feasibility was NOT tested within EMEEES: Reference trend = result of direct (bottom-up) modelling calculation or of correction of the indicator for structural effects, using (bottom-up) modelling

In all cases: correction of reference trend for energy market price increase, using a default value for the short-term price elasticity of 0.1 or 0.2



All

energy

savings

Case 1: replacement of existing equipment

Baseline = Before action situation (stock baseline if aggregated units are used)

Case 2: add-on energy efficiency investment without replacement of existing equipment or building

Baseline = Before action situation

Case 3: new building or appliance: the before situation does not exist and a reference has to be created.

Baseline = A reference situation° (e.g., (1) the existing stock)

Apart from avoiding double-counting, only multiplier effects* have to be analysed in principle

The option that appears most consistent, however, feasibility was NOT tested within EMEEES:

Reference trend = result of (bottom-up) modelling calculation of the development of the indicator without any technical, organisational, or behavioural end-use actions to improve energy efficiency

In particular, zero change of the indicator

between years would only be a correct

reference trend, if all structural effects

influencing the indicator value were

removed**. This may be feasible for specific energy consumption indicators related to an end-use equipment (e.g., cars, refrigera-tors.

In these cases: Reference trend = base year (2007) value of the indicator

This will not be feasible for indicators of sectoral unit energy consumption or diffusion indicators, with the single exception of solar water heaters

* In practice, this is often difficult and the two effects work into the opposite direction, and so it is recommended to only assess multiplier and free-rider effects for EEI measures exceeding a threshold of annual energy savings of, e.g., 40 million kWh of electricity or 100 million kWh of other fuels, or a minimum contribution of 5 % to the ESD energy savings target of a country. According to experience, the additional costs for evaluating these effects would still be below 1 % of the overall costs of measures above this threshold.

° Reference situation could be: (1) the existing stock, (2) the existing market; (3) the legal minimum performance; (4) the Best Available Technology (BAT) (situation (4) only appropriate for technology procurement and similar measures that aim to bring technologies better than BAT to the market)

** Despite the efforts of ODYSSEE to remove structural effects, the ‘apparent total’ energy savings calculated by taking zero change of the indicator between years as the reference trend are, for most ODYSSEE indicators, not consistent with calculating all energy savings. Such ‘apparent total’ energy savings can anyway only be calculated for about 60 % of all ODYSSEE indicators/countries analysed in EMEEES case studies, for which the actual development of the indicator leads into the direction of energy savings vs. the base year. Even in these cases, taking these ‘apparent total’ energy savings for proving the ESD energy savings would be like a lottery for the Member States, because their size is partly determined by structural effects and may thus over- or underestimate the ‘real’ all energy savings. It is impossible to say without further corrections through bottom-up modelling, which part of them is due to energy efficiency and which part is due to structural effects. Such bottom-up modelling may be costly and may be possible based on available data or may not be possible at all.

2.8 Existing evaluation and monitoring methods in the EU

The EU and its Member States do not start from scratch when it comes to evaluation

and monitoring of energy savings from energy policy measures and energy services. At



the beginning of the project, a review was made on existing evaluation and monitoring

methods in the EU and partly the USA and Norway. 26 EEI measures and mechanisms

were chosen as case studies, to cover all end-use sectors relevant for the ESD. Since

the EEI mechanisms usually stimulate a variety of EEI measures, the number of

measures covered is around 220.

The purpose was to review and assess existing evaluation practices and thus provide a

background and basis for developing and proposing evaluation methods under the EU

Directive on energy end-use efficiency and energy services. The review shows that

there is considerable experience of evaluating various types of energy efficiency

improvement measures in all sectors. The methodological approaches for quantifying

energy savings exist in many countries, although accurate quantification and

verification of savings has not always been a priority in past evaluations.

The results of the analysis are presented in detail in the report ‘Assessment of existing

evaluation practice and experience’ (Nilsson et al. 2008)27. The findings have also been

used to upgrade the MURE database28.

The results of the case studies analysis have been summarised in Table 6 indicating

the type of evaluation method (cf. chapter 2.3 for bottom-up methods and chapter 2.5

for top-down methods). Most measures entail regulatory (R), financial (F), and

informative (I) components at the same time. One is an energy service (S). For

example, a white certificates scheme has a strong regulatory component although the

financial incentive is also central, rendering it an R/F classification. It should be noted

that saving energy is often not the only objective of EEI measures. Other objectives

can include social, urban rehabilitation or environmental aspects. For example, the

overall objective of one of the KfW buildings programmes was to provide soft loans to

the general modernisation of buildings in the Eastern parts of Germany. This means

that existing evaluations typically include several aspects other than the energy

savings.

It should be noted that one EEI measure may target several sectors, end-uses or

technologies. The quantification of energy savings in different parts may be more or

less thorough and documented. Hence, the indications given on which bottom-up

evaluation method has been used is based on our overall assessment of evaluations of

the respective EEI measure. Most evaluations rely on deemed savings, mixed deemed

and ex-post, and enhanced engineering estimates. Direct measurements are not

common according to Table 6, but this is hiding the fact that deemed estimates can

often be based on direct measurements, at least in part, and deemed estimates can

therefore be quite accurate, depending on the case.


savings.eu/emeees/en/publications/reports/EMEEES_WP2_D1_Assessment_existing_evaluation_2008-04-21.pdf

28 http://www.isisrome.com/mure/



Table 6: Evaluation methods used for 26 EEI measures

Bottom-up evaluation

method used

Top-down

evaluation

method used

EEI measure29

Co

un

try

Ma

in t

yp

e o

f m

ea

sure

Dir

ect

mea

surem

en

t

Bil

ls &

sa

les

da

ta

an

aly

sis

En

ha

nced

en

g.

Est

ima

te

Mix

ed

deem

ed

an

d e

x-p

ost

Deem

ed

est

ima

te

Bo

tto

m-u

p m

od

ell

ing

Eq

uip

men

t d

iffu

-

sio

n/s

pecif

ic

co

nsu

mp

tio

n

ii

En

d-u

se/s

ecto

r

id

it

Eco

no

mett

ric

Energy Efficiency

Commitment

UK R/F X X X X

Federal Energy

Management Program

US R X X X X X

Building EE, Oregon US F X X X EPS Building Code NL R X X X Building regulation in

Carugate

IT R X X X

Elsparefonden DK F X X X Applicance labelling NL I/F X X X Energy+ – Europe EU I X X X KfW buildings

programme

DE F X X X

Helles NRW DE F/I X

Resi

den

tia

l a

nd

terti

ary

Technology

Procurement

SE I/F X X

Free energy audits DK I/F X X X Investment Deduction

Scheme

NL F/R X

Voluntary Agreement DK F/R X X Programme for EEI in

industry

SE F/R X X X

Energy Audit Program FI I/F X X

Ind

ust

ry

Industrial EE Network NO I/F X X X ACEA agreements EU R X X Ecodriving NL I X X X Congestion charging

Stockholm

SE F/R X X X X X X X

Tra

nsp

ort

Car sharing30

DE S X X Energy taxes SE F X White certificates IT R/F X X White certificates FR R/F X X RUE Obligations BE R/F X X

Gen

era

l

EE Portfolio, California US R/F X X X X X X

EE = energy efficiency; RUE = rational use of energy

29 The EEI measures referred to in the table are described in Annex A of the report (Nilsson et al. 2008) 30 Car sharing means in this context that a fleet of cars is made available to members of a car share

group that typically is organised as a company or a cooperative. The term “car pooling” is sometimes used with the same meaning.



2.9 Need for further research and development

Although the compilation of concrete bottom-up case applications and top-down cases

created by EMEEES is maybe the most comprehensive to date at least in Europe, it

has not been able to answer all questions. Some of these questions cannot be

answered by a research project but only by the European Commission and the ESD

Committee. They are, however, relevant for calculating energy savings, so we list them

here. The questions that need political decision include:

• Are ESD energy savings all energy savings or only those that are additional to

autonomous energy savings and to those induced by increasing market prices for

energy?

• Are ‘early energy savings’ from end-use actions before 2008 eligible?

• What is the most appropriate conversion factor for electricity? Is it 1.0 or rather 2.5

or similar, to better reflect the primary energy savings associated with end-use

actions that affect both electricity and fuels?

• Can there be a conversion factor smaller than 1.0 to reflect that heat generated in a

combined heat and power plant and supplied as district heat or as heat on-site by a

third company achieves primary energy savings from a fuel-switching end-use

action?

• Will biomass heating be counted as replacing fossil fuel in any case, or only if the

biomass is not purchased energy (but harvested by the final consumer him-

/herself)?

On the other hand, there are of course also details that the EMEEES project has not

been able to clarify due to its limited duration and resources. Among these are:

• Activities of support to the European Commission and the ESD Committee:

• Develop a template for the 2011 NEEAPs (both for the plan/measure part and

for the monitoring report);

• With the aim to analyse the energy savings reported in the 2011 NEEAPs:

1) develop a method for comparing results (2010/11) and

2) comparison of results for selected sectors/ end uses / measures (2011/12).

• Activities to support the Member State implementation of the ESD:

• Assist Member States in the application of the harmonised system to be

proposed by the Commission by end of 2010;

• Capacity building materials and courses, particularly for the new Member

States.



• Activities of research aiming to make the methodologies more robust:

• Develop a brief and general evaluation methodology guide (as alternative to

additional case applications) which also instructs the users on which of the

existing case applications can be used for inspiration;

• Additional pilot tests of a number of the developed methodologies with the

purpose to analyse the soundness of the proposed default values before the

2011 NEEAPs; doing more research on additional default values and on

requirements for level 2 and 3 bottom-up calculations to ensure the Commission

that their results are plausible and comparable;

• Additional research on what to do in cases for which the top-down indicator

goes in the ‘wrong’ direction;

• Research on the consequence that the introduction of intelligent integrated

energy systems and individual renewable energy exploitation may have on the

possiblities for assessing savings with a bottom-up approach.

2.10 Evaluation of costs and benefits

The objective of EMEEES was to develop methods for evaluating energy savings.

However, the ESD also mentions that the energy efficiency improvement measures

should be cost-effective. Therefore, we offer here some information what ‘cost-

effective’ means, and how cost-effectiveness can be calculated. Further information

can be found, e.g., in a report coordinated by SRCI (1996).

Basically, the cost-effectiveness of energy efficiency improvement measures is

evaluated by comparing the costs of the corresponding investment in energy efficiency

with its benefits. As the impacts of energy efficiency improvement measures occur at

different points in time and over various time periods, the net present value (NPV) of

annual costs and benefits over the lifetime of the energy efficiency programme is

calculated. Energy savings, for example, usually accrue over several years, while most

of the programme costs arise in the initial years of programme implementation. A net

present value (the difference of total discounted annual benefits and costs) greater

than zero or a benefit-cost ratio above one (ratio of total discounted annual benefits

and costs) indicate that a measure is cost-effective. The calculation of costs and

benefits on a unitary basis (levelized costs in /kWh) allows an immediate comparison

of energy efficiency activities with supply side options and a comparison of the costs of

saved energy by different measures with differing lifetimes. For a comparison on a

level playing field, the same methodology, including measure lifetimes and discount

rates, should be used in the analysis of both the demand side and supply side

(Wuppertal Institute et al., 2000, p. 9).

The impacts of energy efficiency improvement measures on various stakeholders

differ, as each perspective is affected by different kinds of costs and benefits. To

ensure that each affected perspective is considered in the evaluation of energy



efficiency programmes, different cost-effectiveness tests exist. The reference method

for the evaluation of energy efficiency programmes from different perspectives is the

California Standard Practice Manual for Economic Analysis of Demand Side Programs

and Projects published by the California Public Utilities Commission (CPUC, 2001).

Out of the five classes of cost-benefit tests that are described in this source, three tests

from the following perspectives are of general relevance for energy efficiency

improvement measures:

• the final consumer participating in the programme (The Participant Test)

• the national economy (The Total Resource Cost (TRC) Test) and

• the society (The Societal Test)

The cost-effectiveness tests from each perspective are briefly presented below.

The final consumer participating in the programme

The Participant Test analyses if the cost savings realised by the consumer over the

expected lifetime of the end-use action outweigh the costs associated with installing

the measure. This is an essential test, since consumers will be unlikely to participate, if

their benefits are lower than the costs. Costs include incremental equipment costs of

the more efficient technology (which are in general higher than standard equipment

costs) and installation costs that are borne directly by the consumer. Benefits are

mainly the energy bill savings achieved with the more efficient technology and the

eventual incentive payments (including tax credits) obtained.

The Participant Test provides basic information that may be helpful in designing a

programme or energy service. A high benefit-cost ratio indicates that individuals have a

high incentive to participate in a programme. Consequently, a high adoption rate of the

end-use action can be expected. Furthermore, the test result helps in designing the

appropriate level of incentive payments. Accordingly, the donation of too high or low

incentive payments can be avoided (National Action Plan for Energy Efficiency, 2008,

p. 6-1).

The national economy

The TRC test measures the impact on the entire national economy by including all

programme costs and benefits relevant to the national economy. Costs include all

programme costs and the incremental costs for the more efficient technology, whether

paid by the participants or partly by the state, an energy company, or an energy service

company. Benefits comprise mainly avoided energy ‘generation’ and capacity plus

some avoided network costs. Incentive payments are not included in the test as they

represent transfer payments, which do not create added value and are consequently

not relevant from the perspective of the whole economy. The test results are especially

relevant for energy policy, as energy efficiency improvement measures are valued



exclusively as a resource option for the entire national economy in this test (CPUC,

2001).

The society

The Societal Cost Test uses all components included in the TRC Test but also

incorporates the avoided external costs of energy supply. These are used in the test as

an additional benefit component. The external costs comprise all relevant costs

currently not valued in market prices such as environmental costs that are not covered

by emission allowances. A social discount rate is used in the calculations, which may

be in contrast to the discount rate used in the TRC test. A societal discount rate is

lower than a market discount rate to avoid an undervaluation of the interests of future

generations. The test result will indicate, if the whole society is better off by

implementing the energy efficiency improvement measure (CPUC, 2001).

The special perspective of energy companies

Energy companies are one potential actor for implementing energy efficiency

improvement measures and are especially mentioned in ESD article 6. There is also a

CPUC methodology from the perspective of the energy company that implements an

energy efficiency programme. However, this is not entirely relevant for the restructured

energy market existing in Europe.31 The different combinations of unbundled energy

companies existing in Europe make the formulation of a comprehensive methodology

for the cost-effectiveness analysis of energy efficiency programmes from the company

perspective difficult. For each functional unit (generation/import, trading, storage,

transmission, distribution, metering/billing, and supply), different benefit and cost

components are relevant.

What is ultimately relevant from the perspective of an energy company can be called

the Profitability Test. This test indicates whether an energy end-use efficiency

investment is more profitable to the energy company than a supply-side investment or

not. The main cost components included in the test are programme costs and

incentives paid by the energy company that implements the programme. Which benefit

components are considered depends on the specific type of unbundled company that

implements the programme. Avoided power purchase costs, generation and capacity

costs may, for example, be considered. In contrast to the TRC test, the incentive

payments are considered as a cost factor. This makes the test also useful when

designing the appropriate incentive level.

In addition, the lost marginal revenue is considered as a cost component if two

conditions are met: First, the energy efficiency activity would not have been carried out

31 European energy market restructuring is driven by the EU Directives on the internal market for

electricity and gas (2003/54/EC and 2003/55/EC), which establish common rules for the internal energy markets.



without an involvement of the energy company. Therefore, free riders do not contribute

to the lost revenues. Second, the energy company will not loose revenue if it is

integrated into the wholesale market, so that it can sell the amount of excess energy

resulting from the energy savings at the wholesale market. Furthermore, in the

Profitability Test the programme costs passed on through rates or prices, the revenues

from energy services sold to the customer, the compensation for net lost revenue of

network operators, and bonuses or shared net savings from the energy efficiency

improvement measure will also be considered as benefit components for energy

companies, if these cost recovery mechanisms exist in the country the energy

company operates in.

Table 7 summarises the four cost-effectiveness or profitability tests by comparing the

use of important cost and benefit components in the tests. The result of each test

provides different information about the impacts of energy efficiency improvement

measures on stakeholders. Thus, it is of importance that the results of all tests are

considered when decisions regarding the implementation are made. By taking the

whole picture into account, the most information about distributional effects between

stakeholders is provided, it is most likely that incentive payments are set at the

appropriate level, and the implementation of too costly measures can be avoided.

Table 7: Summary of benefit and cost components included in each cost-effectiveness test.

Participant Test

Total Resource Cost Test

Societal Cost Test

Energy Company Profitability Test

Avoided Energy Supply System Costs (Generation and Capacity Costs, Grid Losses, etc.)

Benefit Benefit Benefit

External Environmental Costs Benefit

Pass on of Costs through Rates or Prices, or Revenues from Energy Services

Benefit

Compensation of Net Lost Revenues for Network Operators

Benefit

Bonus or Shared Savings Benefit

Incremental Technology Costs Cost Cost Cost

Programme/Measure Implementation Costs

Cost Cost Cost

Incentive Payments Benefit Cost

Energy Bill Savings Benefit

Lost marginal revenue Cost



3 How to monitor and report to the European Commission?

3.1 The overall monitoring and reporting process

In the EMEEES project, 20 bottom-up (BU) and 14 top-down (TD) case applications

have been prepared for calculation of energy efficiency improvement in various end-

use sectors. The choice of case applications was based on the targeted energy use,

where relatively large energy savings were expected. But available experience with

evaluation methods has played a role as well in the choice.

EU countries can choose from these case applications, use own existing methods, or

develop their own case applications of the general methods descibed by EMEEES

when fulfilling the demands of the ESD:

• proving that the 9% or higher savings target has been met for 2016 (or the

intermediate target for 2010)

• showing that bottom-up methods applied cover at least 20-30% of the energy use

covered by the ESD

• taking account of overlap in the scope of top-down cases and bottom-up case

applications focusing on the same targeted energy use, in order to avoid double

counting of energy savings.

Figure 7 shows how, in an interactive five-step process, countries can choose a set of

case applications that meets the ESD demands.

Figure 7: The process of evaluating ESD energy savings

Step a: in the preparation phase, an inventory is made of available evaluation studies

(for bottom-up methods) or statistical data (for top-down methods) available in the

country.



Step b: a preliminary choice of cases for top-down and bottom-up methods is made.

Step c: a check on the coverage of ESD energy use by bottom-up case applications

is made, taking account of their overlap in scope. The preliminary choice is adapted

until the coverage demanded by the ESD is met (20-30% of the annual final inland

energy consumption for sectors falling within the scope of this Directive in the first

evaluation due in 2011 and much higher in second and third evaluations in 2014 and

after 2016).

Step d: the energy savings are determined for the top-down and bottom-up cases

chosen, eventually correcting various factors (to be decided). To enable corrections for

overlap and interaction, the overlap in scope and double counting is determined for

each pair of top-down and bottom-up cases.

Step e: total ESD savings are calculated and checked. The savings figures for the

chosen top-down and bottom-up cases are summed up, accounting for overlap in

scope and/or interaction between savings. The calculated ESD savings are compared

with the target and, in case the target is not met, a further choice is made between

deploying extra methods or more elaborated methods (probably delivering higher

savings).

More detailed guidelines for this five-step process are provided in the report

‘Harmonised calculation of energy savings for the EU Directive on energy end-use

efficiency and energy services based on bottom-up and top-down methods –

development, assessment and application of a practical approach’

(Boonekamp/Thomas 2009)32.

If the target is met, the results will be reported to the European Commission in the

national Energy Efficiency Action Plans. These are due by 30 June 2011 and 30 June

2014. In order for the results to be comparable, it will be important that the Member

States report a minimum amount of information on

• the EEI measures evaluated,

• the monitoring and evaluation methods applied,

• whether they calculate all or additional energy savings and which

baselines/reference trends and correction factors they have used (cf. chapter 2.7),

and

• whether a part of the energy savings are ‘early energy savings’ stemming from the

period 1995 to 2007 (see also chapter 4.3).


savings.eu/emeees/en/publications/reports/EMEEES_WP61_report_090430.pdf



Apppendix 2 and 3 provide checklists for reporting bottom-up and top-down

calculation results that would assist the Member States in achieving the minimum

amount of information.

By 30 June 2007, the Member States had to file their first National Energy Efficiency

Action Plans (NEEAPs). EMEEES prepared a template for these first NEEAPs, which

some of the Member States took as the basis for structuring their Plans. This is still

available on the project website33 and could be useful for the second and third NEEAPs

as well.

3.2 Coverage of end uses and sectors as well as overlap between

EMEEES bottom-up case applications and top-down cases

One of the most important questions with regard to the evaluation of energy savings for

the ESD is whether there will be enough methods and case applications available to

address all the energy end use sectors and end uses covered by the ESD. That is

required to be able to prove that the Member States have achieved their ESD energy

savings targets.

A look at which end uses in which sectors are covered by the 20 bottom-up case

applications and the 14 top-down cases that EMEEES developed, or maybe by both

bottom-up and top-down cases, is giving a first indication for answering this question.

Table 8, taken from Boonekamp/Thomas (2009)34, provides an overview of which

major end uses are covered by one or more of the EMEEES bottom-up case

applications and top-down cases (dark shaded areas), and which are not (light shaded

areas). Clearly, Member States can use their own methods to address those end uses

that are not covered by the EMEEES cases and also for those that are covered.

33 http://www.evaluate-energy-savings.eu/emeees/downloads/070514_NEEAP_template_final.pdf 34 http://www.evaluate-energy-




Table 8: Overlap and common targeted energy use for the proposed top-down and bottom-up cases including missing parts of sector energy use

Explanations on the table:

Dark shaded areas are those where there is principle coverage by EMEEES bottom-up or top-down cases. Large crosses mark end uses, for which there is match by both an EMEEES bottom-up and top-down case. If there is a box around them, the match will be very good (e.g., between 5 White goods in bootom-up and 3 Large appliances in top-down), otherwise only partly. Light shaded areas are covered either by a bottom-up or top-down case, but the other is missing. Small crosses show potential match between an EMEEES case and its missing counterpart of top-down or bottom-up (e.g., between the EMEEES top-down case 2 Electricity general for households and a missing bottom-up case application A VAC).

The second question, however, is whether there are overlaps between bottom-up case

applications and top-down cases. These would allow to cross-benchmark the results,

but it would also create the need to avoid double-counting of the results. In the

analysis, with only the top-down and bottom-up cases developed by EMEEES, 8 cases

with a clear common energy use were found. These are in residential space heating,

domestic appliances, solar water heaters, tertiary heating and electric end uses,

electric motors and drives in industry, energy-efficient vehicles, and modal shifts in



transport. If cases are added to cover missing parts as shown in this table, the number

of common cases will about double.

3.3 Applicability of EMEEES bottom-up case applications and top-down

cases by EU Member State

3.3.1 Application of top-down and bottom-up methods for countries

A check has been made whether the developed EMEEES cases are applicable for the

monitoring and evaluation per country. For each country, an analysis was made of the

applied measures in the first NEEAPs of 2007, and which top-down and bottom-up

cases could be used to calculate the savings of these measures. It has been tested

whether the countries could prove the savings according to the target and the coverage

of 20-30% of ESD energy use with bottom-up cases.

First, the applicable top-down cases were determined on the basis of the “right”

direction of indicator trends (decreasing) and on data availability for Member States.

From the 14 top-down cases, five were widely applicable at this moment, and a sixth

case on residential space heating in some countries.

For the 20 bottom-up case applications, the application for the countries was assessed

assuming availability of data, thus providing an upper bound. For households, there are

many possibilities for the case application on new dwellings, slightly less on building

envelope and heating, and substantially less on appliances and solar. For the services

sector, application possibilities are lower over the whole range due to less measures

found in the NEEAPs. The same is true for all three EMEEES case applications in

transport. For industry (not being part of emission trading), there are hardly any

opportunities for the four available case applications. General bottom-up methods, like

energy audits, white certificate systems and voluntary agreements, can be applied only

in a limited number of countries. However, due to the large scope of these measures,

they can compensate for few possibilities for the specific cases.

For the applicable five to six top-down cases, the same pattern as for bottom-up cases

emerged. The three top-down cases for households show many possibilities for space

heating, fewer for appliances and fewest for solar boilers. The two cases on transport

show moderate possibilities, but the application is very limited for the top-down case

“taxes”, which has the broadest scope but few Member States have mentioned energy

taxation in their action plans.

Table 9 presents an overview of the results. Details can be found in the report

(Boonekamp/Thomas 2009)35.





A prerequisite for applying bottom-up case applications is availability of data. However,

an inventory on data per Member State was out of the scope of this work. As an

alternative it was looked upon how Member States quantified the savings of the EEI

measures in the NEEAPs. It can be argued that countries with extensive ex-ante

quantitative savings estimates probably have more data for monitoring and evaluation

as well. Therefore, a distinction has been made in table 9 between applicable bottom-

up case applications in case of described-only EEI measures (“Y”) and applicable

bottom-up case applications in case of quantified EEI measures (“Q “).

When bottom-up case applications are not indicated in table 9, this does not mean that

they are completely useless for the Member State. In most cases there are some EEI

measures that fit to the bottom-up case application. However, it is expected that the

amount of savings proved with the (not indicated) bottom-up case application is

relatively low. This is due to the absence of an adjoining set of mutually reinforcing EEI

measures or observed lack of strength for individual EEI measures.

For top-down cases, the same method has been used to assess with which top-down

cases the energy savings due to EEI measures can be calculated. In the lower part of

table 9, the results per Member State are shown. Contrary to the bottom-up case

applications, no distinction is made between a case with described-only EEI measures

and a case with quantified EEI measures, except for a few cases. The reason is that

for the applicable top-down cases it is already clear that savings can be calculated in

principle. This provides no statement on whether the top-down case can be applied in

practice, particularly for the space heating case.



Table 9: Applicable bottom-up case applications (upper part) and top-down cases (lower part) for monitoring energy savings of EEI measures included in the national EEAPs

3.3.2 Meeting ESD demands with EMEEES cases

Given the applicable top-down and bottom-up cases per country, and an estimate of

coverage per bottom-up case application and provable savings per top-down or

bottom-up case, the following conclusions emerge from the analysis (cf.

Boonekamp/Thomas 2009)36:

• All countries except three can prove at least 20-30% coverage with EMEEES

bottom-up case applications

• Large contributions are from space heating in dwellings and passenger transport

• Horizontal measures are important for coverage, as their scope is large

• In case all bottom-up case applications could be applied (in the future), more than

90% coverage of ESD energy use can be achieved

• As to proving the 9% savings target, one-third of Member States could have

problems, due to very different reasons: no transport in the NEEAP, no space

heating (Malta), few measures in general, etc.





• Most important case applications for proving the target regard dwellings and road

vehicles.

Finally, it showed up that some case applications are lacking, e.g. on CHP, street

lighting, and mobility management. It can be concluded that the set of case

applications is sufficient but countries may have problems if they can apply few bottom-

up case applications (due to lack of measures or data), lack also horizontal measures

and cannot prove much savings with the five to six top-down cases.

3.3.3 Comments on applicability from the national workshops and the

Final Conference

The EMEEES project organised two EU-wide events (one workshop rather early on

general approaches and a final conference) and a series of 13 national workshops to

present and discuss the general methods and examples of cases. The presentations

from the EU-wide events can be found at the EMEEES website37.

In general, the work of EMEEES received generally positive reaction and feedback. At

the national workshops, the feedback received from the stakeholders showed that the

project was on the right track and that stakeholders were interested in following the

developments within EMEEES. The effort undertaken was seen as a difficult task, yet

very important and useful, especially for the administration representatives who often

were not aware of the difficulties of the evaluation and monitoring of the ESD and of

the ongoing discussions and debates. Fairness between Member States was valued

high; many of the comments received relate to this issue and were very helpful in

improving opur approaches and methods.

Bottom-up

The reports indicate that the proposed 3-level / 4-steps approach was broadly

accepted at the National Workshops, or at least, there was no (strong) opposition to

the EMEEES proposal, which was praised as pragmatic. There were only some

concerns regarding the difference between level 2 and 3, and the use of level 1 which

may be difficult, due to large national variations.

The concept of the default values was generally seen as a good approach and as a

good basis for discussion. Still, it was pointed out that care should be taken, as default

values may be too detached from the values in concrete projects.

Top-down

The issues related to the top-down methods were very extensively discussed in the

French, Dutch and Finnish workshops, while rather sporadic comments were made at

the other National Workshops. Some particular remarks included, e.g: (i) the

potentially arbitrary nature of some elements used in the calculations (DE, IT), (ii)

37 http://www.evaluate-energy-savings.eu/emeees/en/events/final_conference.php



accuracy of the regression analysis results (FI), or complexity of the calculations (EE,

LV).

3.4 Feedback from the Pilot tests on applicability of methods

In co-operation with Member State governments, energy companies, and other

organizations offering energy efficiency improvement measures, the EMEEES methods

were tested in a series of pilot tests. Each of these evaluated ex post the energy

savings from energy efficiency improvement measures implemented in various

countries for a selected sector and end use, by making use of the methods and case

applications developed.

The table 10 below reports the list of case applications tested, whereas table 11

indicates which energy efficiency improvement measures were evaluated. All the case

applications tested are bottom-up.

Table 10: List of case applications tested

EMEEES case application Sector Italy France Denmark Sweden

Building envelope improvement

Residential X

Energy-efficient white goods Residential X

Biomass boilers in the residential sector

Residential X

Condensing Boilers Residential X X

Improvement of lighting system Tertiary (industry) X

High efficiency electric motors Industry X

Variable speed drives Industry X

Energy audits Tertiary and industry end uses

X

Energy performance contracting

Tertiary and industry

X

Table 11: energy efficiency improvement measures evaluated ex-post

Country Subject Sector(s) addressed

France Condensing boilers, building envelope improvements and compact fluorescent lamps under the French White Certificates.

Residential

Italy Schemes under the Italian White Certificates system Residential, tertiary, industry

Sweden Energy Efficiency Investment Programme for Public Buildings (2005-2008)

Public non-residential buildings



Denmark Energy audits performed in Denmark between 2006 and 2008 Industry, tertiary

The pilot tests performed under the French White Certificate (FWC) scheme focused

on end-use actions addressing space heating in the residential sector, as most of the

white certificates have been issued for such actions so far38. In general, pilot test

outcomes suggest the necessity to keep flexibility for each Member State in order to

use the methods best adapted to its context, provided the global bottom-up methodol-

ogy of 4 steps and 3 levels remains harmonised at the EU level. During the pilot tests,

energy saving amounts estimated resulted to largely depend on parameters describing

the before situation (baseline). The adopted estimates must hence be explicitly

sourced and the assumptions used should be transparent.

In the case of building insulation end-use actions, the savings estimated from the FWC

scheme and the EMEEES case application39 appear to be similar when using the same

main parameters. However, significant differences are observed in the calculated

energy savings in the tested building, mainly due to differences in the values used for

the initial thermal transmission. In fact, EPC results using actual values of the building

are lower than FWC values based on deemed values of baseline parameters (e.g., U-

values before refurbishment). This highlights how important the baseline choice is.

In the case of end-use actions addressing condensing boiler installation, there is a

significant gap between level 1 (European default values) and level 2 (national

average) results. This highlights that either the proposed level 1 default values may be

too conservative even when calculating all energy savings, or the FWC values

overestimate the savings (for condensing boilers), especially due to the low boiler

efficiency used in the FWC baseline. Although it is possible at a national level to use a

particular baseline to encourage a given action, for the ESD reporting, this should be

justified, for harmonisation purpose. In this case, at least, the low default value

provides a clear incentive to proceed towards level 2 or 3 evaluation.

Concerning the pilot tests performed under the Italian white certificate scheme, the

analyses performed indicated that the EMEEES case application related to the

installation of condensing boilers in the domestic sector could be simplified in some

points, as e.g. providing EU default values for energy efficiency improvements related

to parts of the heating system like emitters, heat control systems and heating distribu-

tion systems might introduce large uncertainties in the evaluation. It may be better

providing a single default value to only allow the evaluation of the energy efficiency

improvement due to condensing boiler installation and ensuring that the method is

38 Efficient boilers (condensing and low temperature boilers, both for individual and collective housing)

and insulation actions (roof and windows) represent respectively 40% and 13% of the savings credited in the FWC scheme for the period July 2006-December 2008

39 Engineering estimates obtained from building energy performance certificates (EPC) have been considered according to the proposal in the EMEEES case application.



applied only in case that simple and effective operation standards are fulfilled (e.g. only

in case condensing boilers with modulated burners are installed in heating systems

where the water temperature does not exceed 60 °C) than attempting to capture

effects that are too difficult to estimate. The field tests also indicated that the energy

savings due to domestic hot water production by condensing boilers (not considered in

the EMEEES case application) are considerable.

As far as the EMEEES case application related to energy efficient motors is con-

cerned, the field tests performed showed that the EU default values provided for the

motor load factors in case of different application types may make the EMEEES

evaluation method more reliable than the corresponding method used under the Italian

white certificate scheme. On the other hand, the default values provided for the number

of motor operating hours appear too rough and may make some EMEEES estimates

not conservative. In general, it might be more appropriate and simpler to provide just

the energy saving EU default values for various motor application types and motor

power ranges rather than providing default values for load factors, operating hours and

motor efficiency that are supposed to be used in an energy saving calculation formula

by the evaluator.

In case of the EMEEES case application related to the installation of variable speed

drives (VSDs), the field tests showed that this case application aims at covering a

range of VSD technical applications that is probably too wide. This would cause that

this method would disadvantage, e.g., Italy with respect to other EU countries. Indeed

a highly specific method for VSDs used for water pumping systems has been devel-

oped under the Italian white certificate scheme and results in less energy savings than

the energy savings estimated through the EU default values provided by the EMEEES

case application. Such tests seem to confirm the general principle that is better to

propose a sufficiently accurate calculation method for a specific application than aiming

at covering very different applications by too rough energy saving estimates.

Pilot tests performed in Sweden were mostly based on interviews with public authori-

ties, government representatives and energy service companies, to whom a summary

of the tested case applications have been presented. Although the calculations and

data requirements for each case application test are relatively straightforward, the pilot

test outcomes identified some complexities in gathering this data. The main issues

identified by interview respondents regarding EMEEES method applicability to the

ongoing Energy Efficiency Investment Programme (OFFROT) included quality of data

reported, especially given the amount of information requested, and range of motiva-

tion among individuals in voluntarily reporting evaluation results. Moreover, respon-

dents noted that there is often a mismatch in purpose: energy savings is but one of the

broader set of priorities for the public agencies and authorities that administered the

OFFROT program. Based on the pilot test outcomes, the accuracy of the EMEEES

case application related to energy efficient lighting systems in the tertiary sector has

been improved by including estimates of lighting system operating hours per building



category addressed, and by defining which stakeholders should be in charge of each of

the four EMEEES evaluation steps.

The pilot tests performed in Denmark seem to indicate that the EU default values

provided by the EMEEES case application on energy audits do not in all cases provide

an incentive for choosing a high level evaluation approach (i.e. level 2 or level 3). This

seems to happen in particular for the EU default values related to energy savings from

end-use actions addressing heat and fuel. However, this outcome might be due to the

particular situation existing in Denmark, where a lack of incentives with regard to heat

and fuel energy savings is registered compared to electricity savings, and the EMEEES

EU default values hence resulted to largely overestimate the heat and fuel savings.

Under this point of view, it is however the strength of the proposed EMEEES case

application that values to estimate electricity savings and heat and fuel savings are

provided separately. The Danish pilot tests on energy audits also highlight that free-

rider fractions may vary between countries and may be higher in countries, where

energy saving obligations or white certificate schemes are in place (e.g., EMEEES

experts estimated a 50% free-ridership for the energy savings from energy audits

carried out under the Danish energy companies’ energy saving obligations, whereas

the EMEEES case application indicates a 10-15% free-ridership for Finland).



4 How can the European Commission judge the results?

4.1 The tool for NEEAP assessment

One task in the EMEEES project was to develop an assessment tool that can be used

by the European Commission for a quantitative assessment of the measures included

by the Member States in their National Energy Efficiency Action Plans (NEEAPs).

These NEEAPs had to be sent to the European Commission by 30 June 2007. Their

objective is to describe the energy efficiency improvement measures planned to reach

the targets of the Member States (Art 14 (2) of the EU Directive on energy end-use

efficiency and energy services). The tool developed has the objective to support the

Commission in assessing whether it is plausible that the energy efficiency improvement

measures proposed by a Member State will be sufficient to reach the Member State’s

energy savings target under the ESD.

The EMEEES Assessment Tool is a bottom-up modelling tool based on a stock

modelling of residential and tertiary buildings, residential appliances, and vehicles, as

well as main industrial processes and cross-sectoral technologies for industry, and

tertiary electricity consumption. It is an adaptation of the model developed in the MURE

project40. The structure of the tool is described in a special report (Faberi et al., 2009:

Report on the integrated assessment tool and results of the assessment tests for

Germany, Italy and Austria).

4.2 Test assessments of the energy savings expected in selected

NEEAPs

The assessment tool developed by EMEEES has also been used for the analysis of

selected measures from the NEEAPs of Austria, Germany, and Italy. The results are

presented in detail in the special report (Faberi et al., 2009).

The aim of this exercise was to assess, whether the ex-ante estimates made by these

three Member States of the energy savings by measure and sector are plausible. This

will, in turn, answer the question whether it is plausible that the energy efficiency

improvement measures proposed by a Member State will be sufficient to reach the

Member State’s energy savings target under the ESD. The Member States Germany

and Italy were chosen since they display estimates of energy savings by measure in

their NEEAPs, which many other Member States did not do. Austria is one of these

latter Member States. It served as an example to test, whether the EMEEES

assessment tool is able to calculate ex-ante estimates of the energy savings, if the

correct country-specific parameters are chosen.

40 http://www.mure2.com/



4.2.1 Germany

The German NEEAP envisages the implementation of 32 measures. Of these 32

measures, 24 carry the quantitative impact data while for the remaining 8 only a short

qualitative description is provided. The sectoral split of the fully commented 24

measures is as follows: household 6, tertiary 5, industry 5, transport 4, cross sectoral 4.

The savings target calculated in accordance to the Directive 2006/32/CE is 833 PJ and

the savings provided by this set of 24 measures are expected to provide, from 503 to

746 PJ, to which have to be added 375 PJ coming from the early actions, thus arriving

at a total of 878-1121 PJ. All of these numbers and the other numbers in this chapter

are calculated with an electricity conversion factor of 1.

17 out of the 24 German NEEAP measures have been simulated with the EMEEES

tool, of which: 6 in the household sector, 2 in the tertiary, 4 in the industry, 4 in the

transport and 1 cross-cutting. Table 12 shows the expected NEEAP savings broken

down per sector as well as the corresponding values provided with the EMEEES

assessment.

Table 12: German NEEAP, overall results

Savings 2016 PJ Sectors

NEEAP EMEEES

Household 198-315 178-252

Tertiary (Measures evaluated with EMEEES) 13-22 16,0

Tertiary Other measures 31-44 -

Industry (Measures evaluated with EMEEES) 16-24 19,0

Industry (Other Measures) 29-40 -

Transport 159-231 153.3

Cross Cutting (Measures evaluated with EMEEES) 50-60 34,0

Cross Cutting (Other Measures) 6,5-9,5 -

Total Expected energy saving (national target) 503-746 n.a.

Early energy savings 375,0

Total (Sectors + Early energy savings) 878-1121

ESD Target 833,0

Since it was not possible to assess all measures in the German NEEAP with the

EMEEES tool, it is impossible to say whether the German NEEAP in total will be able

to achieve the energy savings it estimates. However, for those measures, which were

tested with the EMEEES tool, the estimates are quite similar to those in the German

NEEAP (see table 12, rows: Household, Tertiary and Industry (Measures evaluated



with EMEEES), Transport, and Cross Cutting (Measures evaluated with EMEEES)).

For the households/residential sector, however, this is only true if all energy savings

are calculated. With additional energy savings, the result is some 70 PJ lower.

4.2.2 Italy

The Italian NEEAP foresees a total of 19 measures, of which 10 in the household

sector, 4 in the tertiary, 5 the industry and 1 in transport. The target calculated in

accordance to the Directive 2006/32/CE is 425,8 PJ and the savings provided by this

set of measures should provide, according to the Italian NEEAP estimations, 455,3 PJ.

This number and the other results in this chapter are calculated with an electricity

conversion factor of 1.

17 out of the 19 Italian NEEAP measures have been simulated with the adapted MURE

model, of which: 9 in the household sector, 4 in the tertiary, 4 in the industry and 1 in

transport. Table 13 shows the estimated target NEEAP savings per sector and the

corresponding values provided with the EMEEES assessment. The latter are in total

about 20 % lower than those from the NEEAP, and below the national ESD target. If

this deficit is not made up by the two measures that the EMEEES tool was not able

evaluate, it will indicate that Italy will need to implement additional measures to achieve

its ESD energy savings target.

Table 13: Italian NEEAP, overall results

Savings 2016 PJ Sectors

NEEAP Italy EMEEES

Household 204,4 156,8

Tertiary 88,8 88,9

Industry 77,4 42,0

Transport 83,6 74,2

Total Expected energy saving (national target) 454,2 361,9

ESD Target 425,8

4.2.3 Austria

This section illustrates the use of the EMEEES adapted MURE tool for the evaluation

of energy-saving measures using the case of the Austrian residential sector as an

example. The analysis conducted here is complementary to the analysis of the German

and Italian NEEAPs described above. Still, it differs from them in that the

parameterisation of the energy-saving measures is not rooted in the NEEAP itself but



in other sources, because the Austrian NEEAP does not provide a quantification of the

effects of the energy-saving measures described therein. Together with those

exercises, this analysis has demonstrated the capabilities of the EMEEES assessment

tool in providing an adequate common platform for both the stand-alone evaluation of

energy-saving measures and their comparison across countries.

However, the results obtained in the analysis of the Austrian residential sector are

mainly illustrative. The main value added of this exercise is that it provides a good

example on how these types of measures can be evaluated as well as gives an

example of the data requirements, typical outputs and level of efforts required for doing

so.

Table 14: Results of the Impact Assessment of the Austrian energy efficiency improvement measures.

Achieved savings of the policy scenario in 2016 with respect to:

Measures

Frozen EE Baseline (all energy savings) PJ

Autonomous Baseline (additional energy savings) PJ

Frozen EE Baseline (all energy savings)

%

Autonomous Baseline (additional

energy savings) %

Building envelope existing 25,09 14,71 16,66% 7,41%

Building envelope new 13,00 2,51 37,12% 10,25%

Building Envelope total 38,09 17,22 18,93% 8,75%

Heating equipments 82,85 33,14 38,07% 19,74%

Sanitary Hot Water

(installation of solar thermal heaters for hot water and phase out of electric storage water heaters as of 2010) 4,10 1,5 25,40% 10,90%

Lighting 0,9 0,4 16,30% 8,90%

Refrigerators 1,77 0,65 37,71% 18,17%

Freezers 1,12 0,24 37,17% 11,13%

Washing Machines 0,28 0,14 9,37% 5,10%

Dishwashers 0,11 0,05 3,94% 0,67%

Driers 0,28 0,08 7,92% 2,41%

Large Appliances 3,56 1,16 24,0% 9,2%

Information and Communication Technology appliances 0,1 0,1 28,80% 22,90%

Total Household Sector 129,59 53,53



4.2.4 Conclusions

The EMEEES adapted MURE tool can well be used by an overseeing institution such

as the European Commission to compare energy efficiency improvement measures in

different countries as well as by single countries or even specific actors within those

countries for the ex-ante evaluation of measures.

The EMEEES tool is suitable for conducting analyses using either a dynamic baseline

reflecting autonomous progress, or a so-called “frozen efficiency” baseline as the

reference scenario. Using the former, EMEEES computes additional energy savings.

Using the “frozen efficiency” baseline, EMEEES computes all energy savings.

In addition, sensitivity analysis can be parameterised and carried out. Thus, if some

output is not satisfactory, it is easy to come back to the simulation scenarios and carry

out a sensitivity analysis in order to check, which kind of efficient technology

penetration is required to obtain the expected results. Obviously, the technology

penetration levels are strictly dependent on the way the measures are structured and

implemented. The EMEEES tool allows analysing, which type of efforts are required to

obtain the expected results, but does not suggest the way these results can be

obtained.

In general, results will mainly be highly dependent on the choice of the baseline

scenario and on the parameterisation of the measure. In some cases, measures as

defined in policy documents or other sources may have an intrinsic ambiguity as to the

definition of a clear parameterisation, thus making a sensitivity analysis necessary.

In synthesis, the advantages of using a structured model like EMEEES/MURE are:

• Consistent evaluation method for all the measures

• More transparent procedures and data, if they are open for scrutiny and discussion

• Possibility to easily carry out sensitivity analysis

• The evaluation efforts are focussed on the parameterisation for the measures and

not on the calculation of the results

In contrast, some disadvantages of using the EMEEES/MURE tool are as follows:

• Intrinsic rigidity of the modeling approach (trade off between the data collection and

modeling costs and the end uses/subsectors coverage)

• Relatively high level of efforts for updating of the baseline scenario.



4.3 How to compare individual evaluation results?

While the EMEEES assessment tool presented in the previous sections is mainly

suitable for ex-ante evaluation of the energy savings to be expected from packages of

energy efficiency improvement measures, it could also be used to check and verify the

reported results of ex-post evaluations reported by Member States. This possibility

exists mainly for bottom-up calculations, since the EMEEES/MURE tool is a bottom-up

model.

The basic precondition for this use of the model, and for any verification of results, is

that the Member States report as precisely as possible on the formulas, input

parameters, and the correction factors and operations used in their calculations.

For such verification in general, the most important input parameters are the energy

efficiency values for the baseline and for the energy-efficient solutions, as well as the

lifetimes used in bottom-up calculations, and the reference trends used for top-down

calculation.

Therefore, EMEEES has developed Reporting checklists for bottom-up and top-down

evaluations, which contain all the main data needed to know in order to assess the

plausibility of results, and to make them comparable between Member States. This

These checklists are included in Appendices 2 and 3 of this report.

The ESD has required the European Commission to propose a harmonised system of

bottom-up and top-down calculation methods for ESD energy savings. What could

harmonisation mean in practice (cf. also section 2.2)?

The highest level of harmonisation in results is certainly defined by default or even

harmonised values. In top-down calculations, EMEEES has recommended to use

default values only for specific energy consumption indicators for appliances and

vehicles, and for the diffusion indicator on solar water heaters (cf. section 2.6). In

bottom-up calculations, there are of course the default or harmonised values on

lifetimes of energy savings, as proposed by the CEN (2007) or the list to be published

in 2009 by the Commission. Furthermore, EMEEES has prepared a number of

proposals for default values for unitary gross annual energy savings in bottom-up

calculations, or for some input parameters for these. However, these default values are

often quite conservative to reflect uncertainties and differences between Member

States. They are thus a tool that allows calculation in Member States that do not yet

have own data and that stimulates creation of own data, rather than a tool for

harmonisation.

The second level of harmonisation in methods and results can be achieved by

harmonised rules for a) definition of formulas, parameters, monitoring, and

calculation procedures in level 2 and 3 calculations (bottom-up) and national reference



trends (top-down), and b) harmonised reporting of results. EMEEES has presented

proposals for the definition of formulas, parameters, monitoring, and calculation

procedures in level 2 and 3 calculations in the 20 bottom-up case applications. We

have also described the steps in top-down calculation for 13 types of indicators and the

energy taxation method (Lapillonne /Desbrosses 2009). However, there is still quite

some flexibility for the Member States, making these proposals currently belong to the

third level of harmonisation, which has the status of supporting resources (cf.

section 2.2). It needs further analysis and discussion, to which extend such supporting

resources can be moved on to the second level, the harmonised rules. This is certainly

also an area, in which more experience needs to be collected in the next round of

NEEAPs in 2011. These NEEAPs will include the first ex-post calculations of energy

savings. And we again very strongly recommend that the European Commission

require harmonised reporting using at least a format such as the reporting checklists

developed by EMEEES and presented in Appendices 2 and 3. Probably the most

effective way to achieve harmonisation between Member States is to encourage and

facilitate sharing of experience. If Member States learn from each other, this will lead

to more harmonised practices. Harmonised reporting will be highly important to

facilitate such sharing of experience and mutual learning.

Harmonised reporting will also allow the Commission to better judge the plausibility and

comparability of savings (and hence efforts) between Member States and in many


assessment tool that is another product of EMEEES.



5 Conclusions

Monitoring and evaluation of the energy savings is very relevant and important for the

impact that the Energy Services Directive may finally have. The Member States have

to prove to the European Commission that they have saved enough energy to reach

their targets. This is why the methods and tools by which the Member States calculate,

evaluate, and report their energy savings towards achieving the targets adopted for the

Directive are very important. The results must be comparable to build confidence that

all Member States have taken comparable effort to stimulate the markets for energy

services and energy end-use efficiency in general.

At the same time, the effort for monitoring, evaluation, calculation, and reporting must

be limited to a reasonable level. However, it should be kept in mind that there is a

trade-off between effort and accuracy. The higher the accuracy, the fairer the

comparison between the results presented by Member States. Therefore, there is also

a trade-off between effort and fairness.


tion of the Directive have not yet published a decision on how to deal exactly with the

many open issues that the ESD has left open, notably the additionality or not of energy

savings, and the interpretation of ‚early action’. It is not in the competence of the

EMEEES project to decide this. The methods and case applications developed by

EMEEES, therefore, enable Member States to both calculate all energy savings

(including those through ‘autonomous improvement’) and additional energy savings

(excluding those through ‘autonomous improvement’). Furthermore, the methods and

case applications enable Member States to assess whether early energy savings

achieved before 2008 still exist in 2016. This does not prejudice the choice of

calculating all or additional energy savings, nor whether early energy savings should

count towards the ESD target or not.

Within these limitations, EMEEES has been able to prepare general methods for

bottom-up and top-down methods plus guidelines for ensuring consistency between

the results of bottom-up and top-down calculations. The project furthermore developed

20 bottom-up case applications and 14 top-down cases of these general methods,

which together already cover the largest part of potential ESD energy savings from the

energy efficiency improvement measures the Member States have pledged to

implement in their National Energy Efficiency Action Plans. For example, we estimate

that with our total set of bottom-up case applications, more than 90% coverage of the

energy use subject to the ESD can be achieved.

Therefore, EMEEES has been able to show that evaluation energy of efficiency

improvement measures and calculation of ESD energy savings is possible. In

summary, we recommend to use the following methods:







effects of the whole package of measures, including multiplier (market

transformation) effects. Bottom-up calculations are possible for appliances and

vehicles, too, but it is often difficult to calculate multiplier (and free-rider) effects

with them.

• Top-down methods are the way to calculate the effects of energy taxation and

add them to the effects of bottom-up calculations for a sector, but only if these

bottom-up calculations exclude free-rider effects. The energy savings due to

taxation must not be added to results of top-down calculations on sectors or








These recommendations are based on our analysis of case applications for bottom-up

and top-down methods, as well as on practical experience in many countries and our

pilot tests (cf. section 3.4). They are based on the general trend of findings from these

sources.

However, the quality of data available in a country will finally determine which



The ESD has required the European Commission to propose a harmonised

calculation model of bottom-up and top-down calculation methods for ESD energy

savings. In chapters 2 and 4, we analysed what harmonisation could mean in practice.

Certainly, the Commission and the Member States could decide to use as many

default values as possible. EMEEES has developed some proposals in this area, too.

On the other hand, the precision of results will deliberately be higher if national level 2

and 3 calculations (bottom-up) and national reference trends (top-down) are used, but

with harmonised rules for a) definition of formulas, parameters, monitoring, and

calculation procedures, and b) harmonised reporting of results. This is certainly also

an area, in which more experience needs to be collected in the next round of NEEAPs

in 2011. These NEEAPs will include the first ex-post calculations of energy savings.

And we again very strongly recommend to require harmonised reporting using at

least a format such as the reporting checklists we have developed and present in



Appendices 2 and 3. This will then allow the Commission to better judge the plausibility

and comparability of savings (and hence efforts) between Member States and in many


adapted MURE assessment tool developed by EMEEES.

Some further challenges for development and application of methods remain. They

include:

• What is the relationship between EU-level and Member States’ measures – can the

effects of EU-level measures be counted, and how can they be counted bottom up,

or how can they be distinguished top down?

• For the ESD, it does not matter if a measure is undertaken by the Member State’s

central government or by regional or local governments, or by market actors and

stakeholders such as energy companies and energy service companies. Few

Member States have included measures by these actors in their plans, but

hopefully they will try to monitor their energy savings, too. However, the issue of

potential double-counting remains, as does the issue that all of these actors also

would like to know what the energy savings due to their measures are. More

practical analysis of case studies will be needed here.

• This issue is related to the question, whether always only packages of measures

targeting an end use or sub-sector shall and can be evaluated, and / or whether

there should be a ranking of measures between ‘measurable’ (e.g., energy

performance contracting or financial incentive programmes) and ‘supporting’ (such

as general information campaigns) ones.

• What linkages and differences are there between CO2 abatement measure impact

monitoring (which must prove additionality) and ESD energy savings evaluation?

Can evaluation costs be kept down by joining the two evaluation streams?

• Could the EMEEES methods or, more generally, the harmonised system of

calculation methods to come, enable the proof of fulfillment and thereby the setting

of mandatory energy savings or energy efficiency targets?

• How shall evaluation of cost-effectiveness, i.e., costs and benefits be performed,

what are the rules and methods? Can the information presented in section 2.10

provide guidance here?

• The demarcation and interlinkages to energy supply are still not perfectly clear, e.g.,

the treatment of district heating from cogeneration of heat and power, or biomass

boilers.

• Can infrastructural measures (e.g., the construction of railways) be counted as

energy-saving measures, and how should the resulting energy savings be

calculated?

Finally, it should be noted that evaluation is not only possible, it is also necessary for

the continuous development and refinement of energy efficiency policies and other



energy efficiency improvement measures. However, evaluation is an area where a

single best method cannot be devised. Methods must evolve and be adapted to the

measure and context at hand. The quality of evaluations will improve as experience

accrues through learning-by-doing. This will not only benefit the cost-effective

implementation of the Directive, but also provide a basis for future and probably more

ambitious energy efficiency policy efforts.



6 References

Active Implementation of the European Directive on Energy Efficiency (AID-EE) project team, Jamil Khan, Mirjam Harmelink, Robert Harmsen, Wolfgang Irrek, Nicola Labanca, 2007: From Theory Based Policy Evaluation to SMART Policy Design, Summary report of the AID-EE project. ECOFYS Netherlands bv, Utrecht, Lund University, Lund, eERG/Politecnico di Milano, Milan, Wuppertal Institute, Wuppertal

Jean-Sébastien Broc, Jérôme Adnot, Bernard Bourges, Stefan Thomas, and Harry Vreuls, 2009: The developing process for harmonised bottom-up methods. ARMINES, Nantes/Paris, Wuppertal Institute, Wuppertal, SenterNovem, Sittard. Available at: http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/D4_EMEEES_Final.pdf

P.G.M. Boonekamp, 2009: How much will the Energy Service Directive contribute to EU energy and emissions goals? (to be published in Energy Efficiency)

P.G.M. Boonekamp and Stefan Thomas, 2009: Harmonised calculation of energy savings for the EU Directive on energy end-use efficiency and energy services based on bottom-up and top-down methods – development, assessment and application of a practical approach. ECN, Petten and Wuppertal Institute, Wuppertal. Available at: http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/EMEEES_WP61_report_090430.pdf

California Public Utilities Commission (CPUC), 2001: California Standard Practice Manual (CSPM): Economic Analysis of Demand-Side Management Programs and Projects. CPUC, San Francisco

California Public Utilities Commission (CPUC), 2006: California Energy Efficiency Evaluation Protocols: Technical, Methodological, and Reporting Requirements for Evaluation Professionals. Report prepared by the TecMarket Works Team, CPUC, San Francisco, CA, April 2006.

Comité Européen de Normalisation (CEN), 2007: Saving lifetimes of energy efficiency improvement measures in bottom-up calculations, CWA 15693, April 2007

European Commission (EC), 2006: Action Plan for Energy Efficiency: Realising the Potential. Communication from the Commission COM(2006)545 final. European Commission, Brussels, 19 October 2006.

Wolfgang Eichhammer, 2008: Distinction of energy efficiency improvement measures by type of appropriate evaluation method. Fraunhofer ISI, Karlsruhe. Available at: http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP3_Report_Final.pdf

Bruno Lapillonne, Didier Bosseboeuf, and Stefan Thomas, 2009 : Top-down evaluation methods of energy savings, Summary report. Enerdata, Grenoble, ADEME, Paris, Wuppertal Institute, Wuppertal. Available at: http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/EMEEES_WP5_Summary_report_May_2009.pdf, Annex to the report: http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/EMEEES__WP5_TD_indicators_overview_final.pdf

Bruno Lapillonne and Nathalie Desbrosses, 2009 : Top-down evaluation methods of energy savings, Case studies summary report. Enerdata, Grenoble. Available at: http://www.evaluate-energy-savings.eu/emeees/downloads/WP_5_EMEEES_case_studies_report_Final.pdf

National Action Plan for Energy Efficiency, 2008: Understanding Cost-Effectiveness of Energy Efficiency Programs: Best Practices, Technical Methods, and Emerging Issues for Policy-Makers. Edited by Energy and Environmental Economics, Inc. and Regulatory Assistance Project

SRC INTERNATIONAL APS (SRCI), 1996: European B/C Analysis Methodology (EUBC), A Guidebook For B/C Evaluation Of DSM And Energy Efficiency Services Programs. SRC INTERNATIONAL APS

(DENMARK), February 1996

SRCI, NOVEM, Electricity Association, MOTIVA, Norsk Enok og Energi AS, Centre for Energy Conservation of Portugal, Elkraft system, SEVEn, Energy Saving Trust, Wuppertal Institute, 2001: A European Ex-Post Evaluation Guidebook for DSM and EE Service Programmes. SAVE Project No. XVII/4.1031/P/99-028, April 2001

TecMarket Works, Megdal & Associates, Architectural Energy Corporation, RLW Analytics, et al., 2004: The California Evaluation Framework. Report prepared for the Southern California Edison Company as mandated by the California Public Utilities Commission, K2033910, Revised September 2004. Available at: http://www.calmac.org/toolkitevaluator.asp



Edward Vine and J. Sathaye, 1999: Guidelines for the Monitoring, Evaluation, Reporting, Verification and Certification of Energy-Efficiency projects for Climate Change Mitigation. Report prepared for the U.S. Environmental Protection Agency, LBNL-41543, March 1999

Harry Vreuls, Wim De Groote, Peter Bach, Richard Schalburg, Kirsten Dyhr-Mikkelsen, Didier Bosseboeuf, O. Celi, J. Kim, Lena Neij, M. Roosenburg, 2005: Evaluating energy efficiency policy measures & DSM programmes - volume I : evaluation guidebook. Report for the IEA-DSM task IX, October 2005. Available at: http://dsm.iea.org/Publications.aspx?ID=18

Harry Vreuls, Stefan Thomas, and Jean-Sébastien Broc, 2009: General bottom-up data collection, monitoring, and calculation methods, Summary report. SenterNovem, Sittard, Wuppertal Institute, Wuppertal, ARMINES, Nantes. Available at: http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/EMEEES_Bottom_up_draft_overview081006.pdf

Wuppertal Institute et al., 2000: Completing the Market for Least-Cost Energy Services, Strengthening Energy Efficiency in the Changing European Electricity and Gas Markets, A Study under the SAVE Programme, Project Final Report, Wuppertal et al



7 Ackowledgements

This project would not have been possible without the financial support from the

European Commission IEE programme, as well as from a number of Member State

governments. We also very much appreciated the discussion of issues related to the

project and the comments received from European Commission staff (mainly DG

TREN, the EACI, and the JRC), experts from Member State governments and

agencies, as well as all participants in the national and EU workshops and the Final

Conference. The many colleagues from the project partner organisations and

institutions, who contributed to the results of the project, are listed in the table below.

Project Partner Staff contributing to the project

Wuppertal Institute for Climate, Environment and Energy (WI)

Stefan Thomas, Ralf Schüle, Vera Höfele, Susanne Böhler, Frederic Rudolph, Renate Duckat, Carolin Schäfer-Sparenberg, Wolfgang Irrek, Felix Suerkemper

Agence de l’Environnement et de la Maitrise de l’Energie (ADEME)

Didier Bosseboeuf, Robert Angioletti, Pascal Larsonneur, Sarah Dukhan, Luc Bodineau, Hervé Lefebvre, Therese Kreitz

SenterNovem Harry Vreuls, Michel de Zwart, Annemie Loozen, Cees Maas, Tom Monné (SenterNovem); Michiel Beeldman (PwC Nederland); Robert van den Brink (Goudappel Coffeng)

Energy research Centre of the Netherlands (ECN)

Piet Boonekamp, Coen Hanschke, Yvonne Boerakker, Floor van der Hilst

Enerdata sas Bruno Lapillonne, Nathalie Desbrosses

Fraunhofer-Institut für System- und Innovationsforschung (FhG-ISI)

Wolfgang Eichhammer, Barbara Schlomann, Tobias Fleiter

SRC International A/S (SRCI) Kirsten Dyhr-Mikkelsen

Politecnico di Milano, Dipartimento di Energetica, eERG

Nicola Labanca, Lorenzo Pagliano, Daniele Palma, Andrew Pindar, Tommaso Toppi

AGH University of Science and Technology (AGH-UST)

Adam Gula, Arkadiusz Figorski, Anna Barcik, Pawel Wajss, Dominik Jezierski, Boguslaw Reich

Österreichische Energieagentur – Austrian Energy Agency (A.E.A.)

Leonardo Barreto-Gomez, Klemens Leutgöb, Susanne Geissler, Christof Amman

Ekodoma Dagnija Blumberga, Claudio Rochas, Marika Rochas

Istituto di Studi per l’Integrazione dei Sistemi (ISIS)

Stefano Faberi, Michela Fioretto

Swedish Energy Agency (STEM) Carlos Lopes, Martina Högberg, Anna Forsberg (STEM); Lars J. Nilsson, Christian Stenqvist, Katie Lindgren (Lund University); Jenny Gode, Rebecka Engström (IVL - Swedish Enviromental Research Institute)

Association pour la Recherche et le Développement des Méthodes et Processus Industriels (ARMINES)

Jean-Sébastien Broc, Jérôme Adnot, Bernard Bourges, Daniela Bory, Philippe Rivière, Stefano Grassi

Electricité de France (EdF) Paul Baudry, Dominique Osso

Enova SF Andreas Krüger Enge (Enova); Nils Borg, Anne Bengtson (eceee)

Motiva Oy Ulla Suomi, Lea Gynther

Department for Environment, Food and Rural David Cope, Carsten Rohr, Tom Bastin, James Accord (DEFRA); Heather Haydock, Diana



Affairs (DEFRA) Parusheva-Lowery (AEA Technology)

ISR – University of Coimbra (ISR-UC) Paula Fonseca, Anibal de Almeida, Tiago Fernandes

DONG Energy (DONG) Tine Florin, Søren Vontillius

Centre for Renewable Energy Sources (CRES) Konstantinos Lytras, Kostas Tigas



8 Appendices

8.1 Appendix 1: List of EMEEES Reports and Bottom-up Case

Applications

WP N

o

Author(s), Title, and Link (if the document is publicly available)

2 Lars J. Nilsson et al., 2007: Assessment of existing evaluation practice and experience

http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/EMEEES_WP2_D1_Assessment_existing_evaluation_2008-04-21.pdf

3 Wolfgang Eichhammer (Fraunhofer ISI), 2008: Distinction of energy efficiency improvement measures by type of appropriate evaluation method http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP3_Report_Final.pdf

4 Jean-Sébastien Broc, Jérôme Adnot, Bernard Bourges, Stefan Thomas, and Harry Vreuls, 2009: The development process for harmonised bottom-up evaluation methods of energy savings

http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/D4_EMEEES_Final.pdf

4 Harry Vreuls, Stefan Thomas, and Jean-Sébastien Broc, 2009: General bottom-up data collection, monitoring, and calculation methods, Summary report

http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/EMEEES_Bottom_up_draft_overview081006.pdf

4 20 bottom-up case applications

EMEEES case application 1: Building regulations for new residential buildings (residential sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_1_Building_regulations_for_new_buildings_Final.pdf

EMEEES case application 2: Improvement of the building envelope of residential buildings (residential sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_2_Building_Envelope_Final.pdf

EMEEES case application 3: Biomass boilers (residential sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_3_Biomass_boilers_Final.pdf

EMEEES case application 4: Residential condensing boilers in space heating (residential sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_Method_4_resboilers_080609.pdf

EMEEES case application 5: Energy efficient cold appliances and washing machines (residential sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_5_White_Appliances_Final.pdf

EMEEES case application 6: Domestic Hot Water – Solar Water Heaters (residential sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_6_Solar_Water_Heaters_Final.pdf

EMEEES case application 7: Domestic Hot Water – Heat Pumps (residential sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_7_Heat_Pumps_Final.pdf

EMEEES case application 8: Non residential space heating improvement in case of heating distribution by a water loop (tertiary sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_8_non_residential_heating_Final.pdf

EMEEES case application 9: Improvement of lighting systems (tertiary sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_Method_9_Lighting_final.pdf

EMEEES case application 10: Improvement of central air conditioning (tertiary sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_10_centralairco_Final.pdf

EMEEES case application 11: Office equipment (tertiary sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_11_Office-equipment_Final.pdf



EMEEES case application 12: Energy-efficient motors (sector industry)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_12_EEM_Final.pdf

EMEEES case application 13: Variable speed drives (sector industry)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_13_VSD_Final.pdf

EMEEES case application 14: Vehicle Energy Efficiency (transport sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_Method_14_Vehicle_EE_080226.pdf

EMEEES case application 15: Modal shifts in Passenger Transport (transport sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_15_Modal_Shifts_Final.pdf

EMEEES case application 16: Ecodriving (transport sector)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_16_Ecodriving_Final.pdf

EMEEES case application 17: Energy Performance Contracting (combined sectors tertiary, industry and residential)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_17_EPC_Final.pdf

EMEEES case application 18: Energy Audits (tertiary and industry sectors)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_Method_18_Energy_Audits_Revised_draft_080530.pdf

EMEEES case application 19: Voluntary Agreements - billing analysis method (industry and tertiary sectors)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_19_VA_industry_billing_analysis_Final.pdf

EMEEES case application 20: Voluntary agreement with individual industrial companies - engineering method (industry and tertiary sectors)

http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP42_20_VA_Industry_Engineering_method_Final.pdf

5 Bruno Lapillonne, Didier Bosseboeuf, and Stefan Thomas, 2009 : Top-down evaluation methods of energy savings, Summary report

http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/EMEEES_WP5_Summary_report_May_2009.pdf

Annex to the report: ODYSSEE and ODEX indicators that can be used in top-down evaluation of energy savings

http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/EMEEES__WP5_TD_indicators_overview_final.pdf

5 Bruno Lapillonne and Nathalie Desbrosses, 2009 : Top-down evaluation methods of energy savings, Case studies summary report

http://www.evaluate-energy-savings.eu/emeees/downloads/WP_5_EMEEES_case_studies_report_Final.pdf

6 P.G.M. Boonekamp and Stefan Thomas, 2009: Harmonised calculation of energy savings for the EU Directive on energy end-use efficiency and energy services based on bottom-up and top-down methods – development, assessment and application of a practical approach

http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/EMEEES_WP61_report_090430.pdf

6/7 Stefano Faberi et al., 2009: Report on the integrated assessment tool and results of the assessment tests for Germany, Italy and Austria

7 Klemens Leutgöb and Stefan Thomas, 2007: Template and guide for NEEAPs

http://www.evaluate-energy-savings.eu/emeees/downloads/070514_NEEAP_template_final.pdf

8 Reports from pilot tests of selected bottom-up case applications

Tine Florin and Søren Vontillius, 2009: National report from the pilot tests of Case application 18, Energy audits: Energy audits in Denmark performed by DONG Energy in the period 2006-2008

http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/D15_DONG_Energy_pilot_test_Final.pdf

Christian Stenqvist and Katie Lindgren, 2009: National report from the pilot tests of case application 9, Improvement of Lighting Systems (tertiary sector) and case application 17, Energy Performance Contracting.



Performed under the Swedish Investment Support for Energy Efficiency Improvements and Conversion to Renewable Energy Sources in Public Non-residential Buildings

http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/D15_STEM_Pilot_Test_Final.pdf

Nicola Labanca, 2009: National report from the pilot tests of Case application 12, Energy Efficient Motors, Case application 13, Variable Speed Drives, Case application 4, Condensing Boilers in space heating, Case application 5, White appliances. Performed under the Italian white certificate scheme

http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/D15_eERG_Pilot_test_report_Final.pdf

Jean-Sébastien Broc et al., 2009: National report from the pilot tests of case application 2, Residential building envelope improvement and case application 4, Residential condensing boilers in space heating. Performed under The French White Certificates Scheme http://www.evaluate-energy-savings.eu/emeees/en/publications/reports/D15_France_Pilot_Tests_Final.pdf

9 Adam Gula, 2008: Reports from National Workshops – Short Summary

http://www.evaluate-energy-savings.eu/emeees/en/events/national_expert_workshops/EMEEES_NAT_WS_SHORT_SUMMARY.pdf

10 Project fact sheet and summary slides



8.2 Appendix 2: Proposal for a reporting checklist for bottom-up

evaluations

Measure(s) and evaluation details

Name of the measure (or group of measures):

Contact person(s) for the measure(s):

Organisations involved in the measure(s) implementation:

Contact person(s) for the evaluation:

Organisations involved in the evaluation:

Short description of the measure(s)

Target group:

Targeted type of final energy (fuel) and end use:

Concrete end-use actions facilitated (please list)41

Period for which the measure has been evaluated:

Short description of the measure(s) (including eligibility requirements for

participation/actions, level of financial incentives, if any, and role of actors)42

:

Main results

It is possible to present values for all energy savings (compared to the status quo

without any of the targeted end-use actions) and for energy savings additional to the

end-use actions taken autonomously by final consumers or other actors.

All annual energy savings in 2016 (or 2010) (in GWh):

Additional annual energy savings in 2016 (or 2010) (in GWh):

Other important results:

41 ESD Annex III provides examples (a) to (o) of end-use actions, which are not exhaustive 42 The Appendix to this checklist provides a non-exhaustive list of types of measures



Calculation process, STEP 1: Unitary gross annual energy savings

• Is an average or a participant-specific value used (in kWh per unit, per action or

per participant):

is it :

o a level 1 default average value ? Please provide the value:

o a level 2 national average value? Please provide the value:

o a level 3 measure-specific average value? Please provide the value:

o a level 3 participant-specific value ?

• calculation method(s) used:

o direct measurement

o billing analysis

o enhanced engineering estimates

o mix of ex-ante and ex-post data

o deemed savings

o other:

• definition of the baseline:

for the calculation of the unitary gross annual energy savings:

o level 1: implicit baseline in the default value ?

o level 2: average energy consumption based on national statistics or samples ?

o level 3 (measure-specific): average energy consumption based on measure-specific

definition/eligibility requirements of energy-efficient end-use actions, regional

statistics or samples ?

In these cases, is the baseline based on

o the stock (before action) situation?

o the inefficient market (without measure) situation?

o another reference situation for new buildings or equipment? If yes which?

Or is the baseline a

o level 3 (individual) baseline: before action energy consumption specific to the

participants, based on measurements, metered data or energy bills ? Or energy

consumption of the participants if they would not have taken advantage of the

evaluated measure ?

• definition of the value of specific energy consumption for the energy-efficient

solution:

for the calculation of the unitary gross annual energy savings:

o level 1: implicit average value in the default value ?

o level 2: average energy consumption based on national statistics or samples ? How

has the energy-efficient solution been defined (e.g., threshold value for specific

energy consumption or specific technology parameter or choice):

o level 3 (measure-specific): average energy consumption based on measure-specific

definition/eligibility requirements of energy-efficient end-use actions, regional

statistics or samples ?



• main data used:

Level 1 (European) data (and vintage):

Level 2 (national) data (and vintage):

Level 3 (measure-specific) data:

• normalisation factors:

What normalisation factors (see ESD Annex IV(1.2)) were taken into account?

How were they applied:

• conversion factors:

Was it necessary to use conversion factors (see ESD Annex II + caution: for the

ESD, Net Calorific Values shall be used)?

If yes, specify the factors used:

• rebound effect (optional):

Was a possible rebound effect considered:

If yes, how:

Calculation process, STEP 2: Total gross annual energy savings

• accounting method:

o national (or specific) register or database of final consumers or other actors

benefiting from the measure

o other direct accounting (e.g. by vouchers or applications):

o market analysis

o survey among participants (all or sample?)

o survey among a sample of the whole population

Was the accounting completed by ex-post verifications (e.g. on-site inspections):

• main data used:

Level 2 (national) data (and vintage):

Level 3 (measure-specific) data:

Calculation process, STEP 3: Total ESD annual energy savings

• double counting43

(see ESD Annex IV(5)):

are other measures targeting the same end-users’ group or the same energy end-uses

and/or end-use actions?

If yes, how were double counting risks managed:

43 Double counting may occur when two measures overlap (e.g. grants and energy audits schemes for

industrial companies).



is there any risk of overlap between national and regional or local measures?

If yes, how was it addressed:

• technical interaction44

:

is there any possibility of overlap between actions?

If yes, how was it considered:

• multiplier effects (see ESD Annex IV(5)):

was the evaluated measure (or group of measures) designed to have multiplier

effects?

what multiplier effects were expected?

how were these multiplier effects monitored over time:

what was the result (in GWh/year of the all or additional annual energy savings

reported above)?

• free-rider effects (only if additional energy savings have been calculated):

were possible free-rider effects considered?

If yes, how were they taken into account:

what was the result (in GWh/year of the all annual energy savings reported above)?

• time lag (if relevant):

was there any risk of time lag in the measure implementation?

If yes, how was it addressed:

does the evaluated measure (or group of measures) include energy efficiency

requirements?

If yes, how was the compliance ensured / monitored:

Calculation process, STEP 4: Total ESD annual energy savings in 2016 (or

2010)

• lifetime of energy savings:

o a level 1 European value ? if yes, harmonised or default value?

o a level 2 national value ?

o a level 3 measure-specific value ?

o a level 3 participant-specific value ?

• persistence effect (optional):

were the results monitored / controlled over time?

If yes how (and what reasons of changes in the results were considered):

44 Technical interaction may occur when two actions overlap (e.g. improving both the insulation and the

heating system of a building). This is considered a special form of the double-counting issue.



• early energy savings (see ESD Annex I(3)):

were energy savings from end-use actions taken before 2008 but after 1995 (in

special cases 1991) reported?

If yes,

o how much savings do they represent (in GWh/year of the all and additional annual

energy savings reported above)?

o were special calculation rules applied (e.g., a different baseline)?

o how is it ensured that they will be still effective in 2016?

Evaluation quality and uncertainties

what are the specifications / guidelines used to ensure the evaluation quality?

how are missing data handled?

can the uncertainties on the results be assessed or qualified? If yes, please provide

the results

Monitoring and evaluation costs

What types of costs are related to the monitoring and evaluation of the measure (or

group of measures)?

Can these costs be assessed (e.g. in for the whole evaluation, or in /kWh saved)?

References

(mention here the reports produced and any document used for the evaluation)



Appendix to the Bottom-up Reporting Checklist: Non-exhaustive list of

energy efficiency improvement measures and mechanisms

Category Subcategories

1 Regulation Standards and norms:

1.1 Building Codes and Enforcement

1.2 Minimum Equipment Energy Performance Standards

2 Information and legislative-informative measures (e.g. mandatory labelling)

2.1 Focused information campaigns

2.2 Energy labelling schemes

2.3 Information Centres

2.4 Energy Audits

2.5 Training and education

2.6 Demonstration*

2.7 Exemplary role of the public sector

2.8 Metering and informative billing*


3.1 Subsidies (Grants)

3.2 Tax rebates and other taxes reducing energy end-use consumption

3.3 Loans (soft and/or subsidised)


4.1 Industrial Companies

4.2 Commercial or Institutional Organisations

4.3 energy efficiency public procurement

4.4 Bulk Purchasing

4.5 Technology procurement


5.1 Guarantee of energy savings contracts

5.2 Third-party Financing

5.3 Energy performance contracting

5.4 Energy outsourcing

6 EEI mechanisms and other combinations of previous (sub)ca-tegories

6.1 Public service obligation for energy companies on energy savings + “White certificates”

6.2 Voluntary agreements with energy production, transmission and distribution companies

6.3 Energy efficiency funds and trusts

* Energy savings can be allocated to these subcategories only if a direct or multiplier effect can be proven. Otherwise they must be evaluated as part of a package.



8.3 Appendix 3: Proposal for a reporting checklist for top-down

evaluations

Evaluation overview

Contact person(s) for the evaluation:

Organisations involved in the evaluation:

Energy consumption or diffusion indicator used:

Statistical basis of the indicator:

Main results

It is possible to present values for all energy savings (compared to the status quo

without any of the targeted end-use actions) and for energy savings additional to the

end-use actions taken autonomously by final consumers or other actors.

All annual energy savings in 2016 (or 2010) (in GWh):

Additional annual energy savings in 2016 (or 2010) (in GWh):

Other important results:

Calculation process

• Normalisation and other corrections for weather, structural effects, etc that

were made (please list the factors corrected for, how it was done, and which data

were used):

1.

2.

3.

4.

(please add more lines if needed)



• Reference trend chosen:

o The value of the normalised and corrected indicator in the base year (for calculation

of all energy savings); base year chosen:

o A reference trend of energy efficiency improvement of the indicator reflecting

‘autonomous improvements’ (for calculation of additional energy savings); value

chosen (in % of improvement per year):

How was this reference trend reflecting ‘autonomous improvements’ developed (e.g.,

regression analysis, expert assessment)?

• Was a correction of the reference trend for increases of energy market prices

made?

If yes, what was the price elasticity used:

How was it developed (e.g., regression analysis, expert assessment)?

• Unitary annual energy savings values for diffusion indicators or specific energy

consumption indicators of equipment

Was such a value needed to calculate total ESD annual energy savings from the results

directly derived from the indicator?

If yes, what is this value (e.g., annual final energy savings per m2 of solar water heater

surface; number of standard annual cycles per washing machine; number of annual

vehicle km travelled):

How was it developed, what were the data sources?

Short description of the measure(s) having an effect on the energy savings

evaluated with the indicator (please provide one box per measure)

Name of the measure (or group of measures):

Contact person(s) for the measure(s):

Organisations involved in the measure(s) implementation:

Target group:

Targeted type of final energy (fuel) and end use:

Concrete end-use actions facilitated (please list)45

Period for which the measure has had an effect:

45 ESD Annex III provides examples (a) to (o) of end-use actions, which are not exhaustive



Short description of the measure(s) (including eligibility requirements for

participation/actions, level of financial incentives, if any, and role of actors)46

:

Evaluation quality and uncertainties

what are the specifications / guidelines used to ensure the evaluation quality?

how are missing data handled?

can the uncertainties on the results be assessed or qualified? If yes, please provide

the results

Monitoring and evaluation costs

What types of costs are related to the monitoring and evaluation of the measure (or

group of measures)?

Can these costs be assessed (e.g. in for the whole evaluation, or in /kWh saved)?

References

(mention here the reports produced and any document used for the evaluation)

(The same Appendix as for the bottom-up reporting checklist would be added to this

top-down reporting as well, cf. Appendix 2 to this report)

46 The Appendix to this checklist provides a non-exhaustive list of types of measures



8.4 Appendix 4: why the harmonised system of evaluation methods

needs to be able to calculate both additional and all energy savings

In Annex IV, the ESD states (section 1.1, general): “In measuring the realised energy

savings as set out in Article 4 with view to capturing the overall improvement in energy

efficiency and to ascertaining the impact of individual measures, a harmonised

calculation model… shall be used to measure the annual improvements…”

The ESD requires MS to achieve their indicative 9 percent annual energy savings

target by 2016 “by way of energy services and other energy efficiency improvement

measures. Member States shall take cost-effective, practicable and reasonable

measures designed to contribute towards achieving this target.” (Art. 4 ESD). On the

other hand, the ESD defines energy efficiency improvement measures as “all actions

that normally lead to verifiable and measurable or estimable energy efficiency

improvement” (Art. 3h ESD). The ESD does not explicitly mention that these actions

and the resulting energy savings shall be additional to the so-called autonomous

savings47 that energy consumers, investors, or other market actor would have done by

themselves anyway.

In October 2006, the European Commission published the Action Plan for Energy

Efficiency: Realising the Potential (COM(2006)545 final). It stated that there is a cost-

effective potential for energy savings of over 20 % compared to baseline projections by

2020. Based on this Action Plan, the European Council on 8/9 March 2007 stressed

“the need to increase energy efficiency in the EU so as to achieve the objective of

saving 20 % of the EU’s energy consumption compared to projections by 2020, as

estimated by the Commission in its Green Paper on Energy Efficiency, and to make

good use of their National Energy Efficiency Action Plans for this purpose.” This is,

therefore a target for additional energy savings. The reference to the National Energy

Efficiency Action Plans suggests that the European Council expects a significant

contribution from the ESD towards these additional energy savings.

This is the case, although the two targets are not directly comparable, since the ESD

target is on final energy savings and for each Member State, and the 20 % target is on

primary energy savings (hence, includes savings in power and district heat generation

and transmission, and oil refineries) and for the EU as a whole. Final energy savings

directly translate into primary energy savings. And the 20 % target is so high that all

Member States will at least have to come close to 9 % additional energy savings for

the Union to meet the 20 % target48.

Furthermore, the ESD states that ‘early action’ can be counted towards the national

energy savings target, albeit subject to guidelines by the European Commission.

47 “brought about by natural replacement, energy price changes, etc.” as stated in the EU Action Plan

(EC, 2006) 48 See also the analysis in Boonekamp, 2009



However, the ESD text can be interpreted in two ways: ‘early action’ could mean

energy savings from technical or organisational action taken by market actors between

2008 and 2016 but facilitated by measures created before 2008 by Member States to

achieve energy efficiency improvements (e.g., a building code revised in 2005 with

tightened requirements) (we shall call this interpretation ‘early measures’), or it could

mean energy savings achieved between 1995 and 2008 due to energy efficiency

improvement measures (we shall call this ‘early energy savings’). A number of Member

States have claimed early energy savings in their first national energy efficiency action

plans (NEEAPs) filed in 2007. Up to 45 % of the 9 % target would be achieved through

early energy savings by these Member States.

An analysis of these two issues has led to the following conclusions:

• If all energy savings, including those due to autonomous changes are allowed to

count towards the ESD target, in the extreme case that all autonomous change is

due to energy end-use efficiency and the Commission’s estimate of 0.85 % per year

of autonomous improvement (EC, 2006) is correct for energy end-use efficiency

improvements in the end-use sectors covered by the ESD as well, only ca. 0.15 %

additional annual energy savings each year (or 1.35 % in 9 years) would be needed

to achieve the target (cf. figure A1).

• If ‘early energy savings’ from action taken between 1995 and 2007 are allowed, if

their average saving lifetime according to CWA (2007) is 15 years, and if they reach

0.6 % per year in each year from 2002 to 2007, only ca. 0.6 % new annual energy

savings would be required in each year from 2008 to 2016 (or 5.4 % in these 9

years together; cf. figure A1).

• If both energy savings due to autonomous changes and ‘early energy savings’ from

action taken between 1995 and 2007 are allowed, no additional energy savings at

all may be needed between 2008 and 2016. The energy savings due to

autonomous changes could be higher than those that remain to be made, after

‘early energy savings’ from action taken between 2002 and 2007 are counted

towards the target of 9 % (cf. figure A1).

What does this mean for a harmonised model of methods to evaluate energy savings

for the ESD? If the ESD is to make a significant contribution to achieving the EU’s

target of 20 % additional energy savings by 2020, as the 2006 EU Action Plan for

Energy Efficiency assumed, the following political conclusions will most likely need to

be drawn for the implementation of the ESD:

1. Not all energy savings from all end-use actions to improve energy efficiency

should be allowed to count for the ESD energy savings target but only energy

savings additional to autonomous changes of energy efficiency. Member States

should, under this condition, try with the highest appropriate effort to exclude

energy savings due to autonomous changes from the calculation of ESD energy



savings. Section 2.7.3 of the report has presented how to make bottom-up and top-

down calculations of additional energy savings consistent with each other.

2. The best solution regarding ‘early action’ would be not to allow ‘early energy

savings’ to count towards the ESD target. This will not put forerunners at a

disadvantage, since they already have good experiences and have many – early –

measures in place, which will create new energy savings during the 2008 to 2016

period.

Figure A1: The potential effects of counting energy savings due to autonomous changes and ‘early energy savings’ (example)

However, it is not up to the EMEEES project to decide on the interpretation of the ESD.

Therefore, the methods and case applications were developed by EMEEES to enable

Member States to both calculate all energy savings and the additional energy savings

that are an impact of energy efficiency improvement measures. Furthermore, the

methods and case applications aim at enabling Member States to assess whether

early energy savings achieved before 2008 still exist in 2016.

9 % savings target by 2016 diluted to 0.6% per year by accepting savings from 2002 onwards (15 instead of 9 years)

Autonomous savings higher than the 0.6% per year needed if accepting early savings from 2002 onwards



8.5 Appendix 5: Types of evaluation methods vs types of EEI measures

This Appendix presents an overview of results from a special analysis within the

EMEEES project. The full report (Eichhammer 2008) is available at the EMEEES

website49.

49 http://www.evaluate-energy-savings.eu/emeees/downloads/EMEEES_WP3_Report_Final.pdf



Table 8-1. Which types of evaluation methods are most appropriate to apply for which type of energy efficiency improvement measure

Category Subcategories Bottom-up methods Bottom-up or top-down

methods: Diffusion indicators/ Stock modelling

Top-Down methods**

1 Regulation Standards and norms:

1.1 Building Codes and Enforcement

1.2 Minimum Equipment Energy Performance Standards

Building stock modelling/ building certificates

Equipment stock modelling

Monitoring diffusion of equipment or buildings meeting performance standard

(Specific energy consumption indicator)

2 Information and legisla-tive-informa-tive measures (e.g. mandato-ry labelling)

2.1 Focused information campaigns

2.2 Energy labelling schemes

2.3 Information Centres

2.4 Energy Audits

2.5 Training and education

2.6 Demonstration*

2.7 Exemplary role of the public sector

2.8 Metering and informative billing*

Deemed savings + surveys


Deemed savings + monitoring + surveys

Enhanced engineering estimates/direct measurement + monitoring


Deemed savings + monitoring

Enhanced engineering estimates/ billing analysis + monitoring

Deemed savings + monitoring of end-use actions + surveys or billing analysis with a control group

Diffusion of label classes

Diffusion of efficient IT appliances Building stock modelling/ Building certificates/

(Specific energy consumption indicators)



Table 8-1 (continued)



Top-Down methods**


3.1 Subsidies (Grants)

3.2 Tax rebates and other taxes reducing energy end-use consumption

3.3 Loans (soft and/or subsidised)

All: Mixed deemed and ex-post estimates / Enhanced engineering estimates/ Deemed savings + monitoring (all) / (building) stock modelling + surveys

Diffusion indicators (where available)


Energy taxes: Econometric modelling / special analysis of specific energy consumption indicators


4.1 Industrial Companies

4.2 Commercial or Institutional Organisations

4.3 energy efficiency public procurement

4.4 Bulk Purchasing

4.5 Technology procurement

Benchmarking of targeted sectors or end-uses (e.g. industrial cross-cutting technologies) / Mixed deemed savings and ex-post estimates + monitoring

4.3 to 4.5: Deemed savings / Mixed deemed and ex-post + monitoring or surveys


Specific diffusion indicators



5.1 Guarantee of energy savings contracts

5.2 Third-party Financing

5.3 Energy performance contracting

5.4 Energy outsourcing

All: Enhanced engineering estimates / Billing analysis/ Mixed deemed savings and ex-post estimates / Direct measurement

+ monitoring

Diffusion indicators Specific energy consumption indicators



Table 8-1 (continued)



Top-Down methods**

6 EEI mechanisms and other combinations of previous (sub)ca-tegories

6.1 Public service obligation for energy companies on energy savings + “White certificates”

6.2 Voluntary agreements with energy production, transmission and distribution companies

6.3 Energy efficiency funds and trusts

Depending on the types and targets of EEI measures (from 1 to 5 above) implemented under the EEI mechanism or as part of the combination;


Building stock modelling with surveys

Integrated bottom-up and top-down methods

Specific energy consumption indicators, depending on the types and targets of EEI measures (from 1 to 5 above) implemented under the EEI mechanism or as part of the combination

* Energy savings can be allocated to these subcategories only if a direct or multiplier effect can be proven by specific monitoring efforts. Otherwise they must be evaluated as part of a package.

** Top-down methods can usually only measure the combined effect of packages of EEI measures targeting one sector (specific energy consumption indicators, econometric methods) or end use (diffusion indicators).

measuring and reporting energy savings for the energy ... · measuring and reporting energy savings...

Documents