evaluation of the market transformation programme final report
TRANSCRIPT
Evaluation of the Market Transformation Programme
Final report
April 2012
Evaluation of the Market
Transformation Programme
Final report
Evaluation of the Market Transformation Programme
Final report
April 2012
Evaluation of the Market
Transformation Programme
Final report
Project
Directors:
Charles Michaelis &
David Kenington
Report Authors: Richard Carter
John Fawcett
Jennifer Hindson
Paula Owen
Additional Project
Team:
Katherine Netherwood
Debs Secker
Date: 19th April 2012
Evaluation of the Market Transformation Programme
Final report
April 2012
Contents
GLOSSARY OF KEY TERMS .............................................................................................................................. 1
1 EXECUTIVE SUMMARY ............................................................................................................................. 5
1.1 INTRODUCTION.................................................................................................................................................. 5 1.2 OBJECTIVES OF THE RESEARCH .......................................................................................................................... 5 1.3 KEY FINDINGS ................................................................................................................................................... 6
1.3.1 Achieving positive outcomes .................................................................................................................... 6 1.3.2 How positive outcomes are achieved........................................................................................................ 6 1.3.3 How effectively the programme is managed and delivered ...................................................................... 7
2 INTRODUCTION ........................................................................................................................................... 9
2.1 BACKGROUND ................................................................................................................................................... 9 2.1.1 What is the Market Transformation Programme and what is it for? ........................................................ 9 2.1.2 What are the activities of MTP? ............................................................................................................... 9
2.2 OBJECTIVES OF THE RESEARCH ........................................................................................................................ 10
3 METHODOLOGY ........................................................................................................................................ 11
4 PROCESS EVALUATION .......................................................................................................................... 13
4.1 PROGRAMME LOGIC MAP ................................................................................................................................. 13 4.2 POST-2007 CONTRACT CHANGES AND RATIONALE ........................................................................................... 14 4.3 HAVE THE INTENDED BENEFITS BEEN ACHIEVED? ............................................................................................. 16 4.4 OTHER ISSUES ARISING .................................................................................................................................... 18 4.5 IMPLICATIONS ................................................................................................................................................. 19
5 POLICY IMPACT DATA USED ................................................................................................................ 20
5.1 MEPS / LABELLING ......................................................................................................................................... 20 5.1.1 ‘Policy vs. reference’ impact .................................................................................................................. 20 5.1.2 Lifetime impact ....................................................................................................................................... 21
5.2 LIMITATIONS OF THE CARBON BUDGETS AND MTP IA DATA: THE NECESSITY TO EXPLORE ‘POLICY A VS. POLICY
B’ IMPACT ............................................................................................................................................................ 22 5.2.1 Limitations of the Impact Assessment and carbon budgeting figures ..................................................... 22 5.2.2 Rationale for establishing a ‘policy A vs policy B’ scenario .................................................................. 24
5.3 OTHER POLICY IMPACTS .................................................................................................................................. 26 5.3.1 Voluntary agreements ............................................................................................................................ 26
5.4 GREEN PROCUREMENT .................................................................................................................................... 27 5.5 OTHER ............................................................................................................................................................ 27 5.6 WIDER BENEFITS ............................................................................................................................................. 27
5.6.1 European Benefits .................................................................................................................................. 27 5.6.2 Global impacts ....................................................................................................................................... 28 5.6.3 MTP work achieving non-carbon impacts .............................................................................................. 28
6 ATTRIBUTION: EVIDENCE OF MTP INFLUENCE ............................................................................. 29
6.1 STAKEHOLDER AWARENESS OF AND ENGAGEMENT WITH MTP. ........................................................................ 29 6.1.1 Perceptions of MTP ................................................................................................................................ 29
6.2 VERIFICATION OF MTP INFLUENCE ON MEPS AND LABELLING ........................................................................ 30 6.2.1 Phase 1 anticipated influence ................................................................................................................. 30 6.2.2 Acknowledged general UK / MTP influence .......................................................................................... 31 6.2.3 Barriers to identifying and achieving influence ..................................................................................... 33 6.2.4 Exploring the counterfactual .................................................................................................................. 34
Evaluation of the Market Transformation Programme
Final report
April 2012
6.2.5 Specific influences at European level ..................................................................................................... 35 6.2.6 Refutation of influence and critiques of MTP data and the UK position ................................................ 36 6.2.7 Summary of influence on MEPS and labelling ....................................................................................... 36 6.2.8 Influences on non-priority products ....................................................................................................... 38
6.3 INFLUENCE ON PUBLIC (GREEN) PROCUREMENT .............................................................................................. 38 6.4 INFLUENCE ON VOLUNTARY AGREEMENTS ....................................................................................................... 39 6.5 INFLUENCE ON ENERGY STAR .......................................................................................................................... 40 6.6 OTHER UK-SPECIFIC POLICY INITIATIVES......................................................................................................... 41 6.7 ASSIGNING IMPACTS TO ATTRIBUTION BANDS .................................................................................................. 43 6.8 VALUE FOR MONEY ANALYSIS ......................................................................................................................... 45
6.8.1 Factors considered in the analysis ......................................................................................................... 45 6.8.2 MTP Value for money ............................................................................................................................ 46
7 CONCLUSIONS AND RECOMMENDATIONS ...................................................................................... 47
7.1 IS THERE VALUE IN THE OUTPUTS? ................................................................................................................... 47 7.2 OPPORTUNITIES FOR MORE EFFICIENT DELIVERY? ............................................................................................ 48
7.2.1 Why MTP? .............................................................................................................................................. 48 7.2.2 Improving efficiency within the current framework................................................................................ 49
7.3 ENHANCING PROGRAMME EFFECTIVENESS ....................................................................................................... 50 7.3.1 Retaining work in key areas of impact ................................................................................................... 50 7.3.2 Enhancing the impact of current outputs................................................................................................ 50 7.3.3 Future opportunities ............................................................................................................................... 51
8 APPENDIX A: FULL METHODOLOGY STATEMENT ........................................................................ 54
8.1 PHASE 1: DESK RESEARCH ............................................................................................................................... 54 8.2 PHASE 1: CONSORTIUM INTERVIEWS ................................................................................................................ 55 8.3 CONSORTIA WORKSHOP ................................................................................................................................... 55 8.4 PHASE 2: STAKEHOLDER INTERVIEWS .............................................................................................................. 56
8.4.1 Impacts explored .................................................................................................................................... 56 8.4.2 Stakeholder interviews conducted .......................................................................................................... 56 8.4.3 Challenges encountered in securing interviews in phase 2 .................................................................... 57
8.5 IMPACT METHODOLOGY .................................................................................................................................. 57 8.5.1 What impact data was required and why? ............................................................................................. 57 8.5.2 Sources of Impact Data .......................................................................................................................... 57 8.5.3 Overlap with other policies .................................................................................................................... 59 8.5.4 Impact data quality and limitations ........................................................................................................ 60
8.6 ATTRIBUTION METHODOLOGY ......................................................................................................................... 61 8.6.1 Stage 1: identifying potential areas of MTP influence ........................................................................... 61 8.6.2 Stage 2: verifying the claimed influences ............................................................................................... 61 8.6.3 Challenges in verifying MTP influence .................................................................................................. 62 8.6.4 Stage 3: Arriving at an assessment of attribution ................................................................................... 62 8.6.5 Stage 4: Integrating impact data with attribution data .......................................................................... 63
Evaluation of the Market Transformation Programme
Final report
April 2012
1
Glossary of key terms
Terminology / acronym
Definition (in the context of this report)
IEA Implementing Agreement
for a Co-operating Programme
on Efficient Electrical End-Use
Equipment (4E)
An international collaboration between International Energy Agency
members to promote wider use of more energy-efficient electrical
equipment.
Attribution
An assessment of the extent to which policy outcomes and thus
improvements in energy performance can be ascribed to a
particular organisation or activity.
Attribution factors
Different levels at which the programme could have influenced
outcomes e.g. strong influence, some influence, little or no
influence.
Blocking minority
The number of votes required at EU Council votes to block a
decision. In EU 27, a blocking minority requires 91 votes of the 345
votes in the Council. This only applies when the ‗qualified majority‘
voting system is used.
Circulators
A circulator is a passive three- or four-port device, in which
microwave or radio frequency power entering any port is
transmitted to the next port in rotation (only). There are circulators
for LF, VHF, UHF, microwave frequencies and for light.
Cold appliances Appliances that use energy to cool e.g. refrigerators or freezers.
Commission policy leads
The Commission representative responsible for coordinating the
policy process – including assimilation of advice and information
from member states and other stakeholders - on a particular
product group within the Ecodesign of Energy-related products
Directive.
Compliance
Ensuring that product manufacturers are producing products that
meet the energy performance requirements of the relevant
regulations.
Confidence level
A statistical term assessing robustness of findings. In the context of
this report, this term refers to the extent to which we can be
confident of MTP claims of influencing particular policy outcomes
e.g. green (strong verification), amber (some verification), red
(little or no verification).
Consortium
Refers to the overall organisation of the MTP, with Defra managing
AEA who in turn contract and coordinate a range of sector and
product experts.
Consultation / negotiation
stages
Proposed EU legislation under the Ecodesign of Energy-related
products Directive must pass through a consultation stage (inviting
stakeholder views on the existing proposals) and a subsequent
negotiation stage (where the proposals are discussed, finalised and
voted upon by member states).
Cost effectiveness
Essentially an assessment of value for money, usually expressed as
a ratio (x benefit for every £1 spent). In the context of this report,
refers to assessment of the benefits delivered through MTP outputs
when compared to the costs of delivering the programme.
Counterfactual
Important in understanding the influence / impact of an
intervention, the counterfactual is the establishment – either
hypothetical or observed – of what would have happened without
that intervention.
Evaluation of the Market Transformation Programme
Final report
April 2012
2
Domestic lighting Fixed lighting found in homes.
Enhanced Capital Allowance
(ECA) scheme
Capital allowances allow the costs of capital assets to be written off
against a business's taxable profits. They take the place of
depreciation in the commercial accounts, which is not allowed for
tax. These schemes have been introduced by government to
encourage investment in particular assets or by particular sorts of
businesses (in the context of this report, energy efficient products
and machinery).
Ecodesign of Energy-related
products Directive
Sometimes refereed to as simply the Ecodesign Directive, this set
of regulations aims to provide consistent EU-wide rules for
improving the environmental performance of energy related
products (ERPs) through eco-design. It prevents disparate national
legislations on the environmental performance of these products
from becoming obstacles to the intra-EU trade. It covers both
Energy-using products (EUPs), which use, generate, transfer or
measure energy and energy related products (ERPs) which do not
use energy but have an impact on energy and can therefore
contribute to saving energy, e.g. windows, insulation, taps etc.
Under the directive, performance standards are established for each
product group.
Electric motors
An electromechanical device that converts electrical energy into
mechanical energy. Electric motors are found in applications as
diverse as industrial fans, blowers and pumps, machine tools, and
household appliances.
Emergency Power Supply Unit
(EPSU)
A device which produces energy in case of failure of the primary
systems.
Energy Star
ENERGY STAR is a voluntary label developed by the US that
identifies the most energy efficient products on the market. Since
2000, the EU has had an agreement with the US to use the label for
office ICT equipment (such as computers, monitors and imaging
equipment). To be eligible for ENERGY STAR, products must meet
specified criteria including energy efficiency minimum standards.
European Commission
The executive body of the European Union, responsible for
proposing legislation, implementing decisions, upholding EU treaties
and the general day-to-day running of the Union.
Government Standard Briefing
Notes (GSBNs)
Notes containing evidence base information and details all
references and assumptions behind the figures in the evidence
base. These Briefing Notes are public consultation documents that
allow stakeholders to examine the data and assumptions behind the
proposed Government Standards and related projections.
Green Procurement
Taking environmental considerations into account within the
procurement process. For governments, introducing green
procurement means directing the combined spending power of
government towards environmentally friendly products and
services. Green Procurement encourages public sector buyers to
take account of environmental factors by setting out key
considerations for procuring different types of goods as well as
technical specifications and criteria that products should meet.
Impact Assessments
The principal outputs of the MTP programme to Defra. These
documents are product group-specific and provide UK government
with a briefing as to the current state of the market and the likely
impacts of introducing different requirements around product
energy performance.
Industry In the context of this report, refers collectively to manufacturers,
retailers and their trade associations / federations which consult
Evaluation of the Market Transformation Programme
Final report
April 2012
3
upon and are affected by the MEPS and labelling legislation coming
out of the EU.
Labelling
A method of visually displaying the performance of products to
allow straightforward comparison or assessment of different
products. In the context of this report, a method of establishing a
mechanism for comparing the efficiency of energy-using products.
Lifetime impacts
For this report, the expected ongoing annualised financial /
environmental benefits arising as a result of policies to improve
product energy performance.
Member State One of the currently 27 countries that comprise the EU.
Minimum Energy Performance
Standards (MEPS)
Specifications containing a number of performance requirements for
an energy-using device, which effectively limits the maximum
amount of energy that may be consumed by a product in
performing a specified task. A MEPS is usually made mandatory by
government and generally requires use of a particular test
procedure that specifies how performance is measured.
National Measurement Office
(NMO)
An executive agency of the UK Department for Business, Innovation
and Skills (BIS), NMO is responsible for managing and developing
the National Measurement System (NMS) which is a network of
laboratories and processes that provide measurement standards
and calibration testing facilities. It maintains the measurement
infrastructure, represents the position of the UK internationally and
influences the development of standards.
Net benefits
Extent of benefits remaining after the costs of delivering those
benefits have been taken into account. In the context of MTP‘s
work, an example would be the overall financial / sales benefits
delivered to UK manufacturers by a drive for more energy efficient
products after the
Non-priority products
For the purposes of establishing MTP impact, non-priority products
are those which the consortium did not feel the MTP had influenced
much in terms of eventual MEPS / labelling policy outcomes.
Policy A vs Policy B scenario
The marginal / additional impact of the MTP-influenced policy
outcome compared to the policy outcome that would otherwise
have been realised.
Policy impact / benefit
In the context of this report, the annualised financial and
environmental benefits arising as a result of a particular energy
performance policy / standard being established for a product
group.
Policy outcome
The policy position arrived at, generally used in this report in
reference to the final MEPS / labelling legislation at EU level
following member state negotiations.
Policy vs Reference scenario
The impact of the MTP-influenced policy outcome compared to a
counterfactual / (‗reference‘) scenario in which no policy is
introduced.
Product area / group The groups into which products governed by EU legislation are split.
Super-Efficient Equipment and
Appliance Deployment (SEAD)
Similar to the 4E, the SEAD initiative is a voluntary scheme with
government members which seeks to transform the global market
for efficient equipment and appliances—reducing energy
consumption while simultaneously saving money for consumers.
SEUP Benefits document
A document produced internally within the MTP, which estimated
the extent to which UK benefits derived from EU MEPS / labelling
policies could be attributed to MTP activities. This formed the basis
of the impacts that this evaluation sought to verify.
Simple Set Top Boxes (STBs) A device that generally contains a tuner and connects to a TV,
turning the signal into content which is then displayed. Set-top
Evaluation of the Market Transformation Programme
Final report
April 2012
4
boxes are used in cable and satellite TV systems to transform the
signal from the cable or satellite to a form that can be used by the
television set or other receiver. It also enhances the quality of
signal.
Stakeholders In the context of this report, refers to any individuals involved in
MTP consortium work i.e. Defra representatives, AEA PAs, or PAMs.
Standby Refers to the electric power consumed by electronic and electrical
appliances while they are switched off or in a standby mode.
Subcontractors
The experts / PAMs contracted by AEA to provide product specific
data and expertise to enable development of the Impact
Assessments and other MTP outputs.
Tertiary lighting Apart from fixed lighting, the other lighting found in homes or
businesses e.g. lamps.
Voluntary agreements In this context, agreements – generally between businesses - that
achieve higher than required environmental outcomes.
Wet appliances Appliances that use water to perform their function e.g. washing
machines or dishwashers.
Evaluation of the Market Transformation Programme
Final report
April 2012
5
1 Executive Summary
1.1 Introduction
The Market Transformation Programme is a consortium of expert contractors managed by
the Department for Environment, Food and Rural Affairs (Defra) and coordinated by the
lead consortium partner, AEA Technology (working alongside Consumer Research
Associates).
The aim of the Market Transformation Programme (MTP) is to drive improvements in
energy and environmental performance of energy using products, through the provision of
a robust evidence base on the current and likely future performance of energy using
products for policy makers in the UK, the European Commission, industry, and global
projects such as Energy Star and 4E.
The main outcomes that the programme aims to support are a real reduction in emissions
(e.g. CO2) and energy use through:
New and refined policies and standards;
Better informed policy decisions (through provision of robust information);
Innovation and improved product design.
1.2 Objectives of the research
Defra‘s contract with AEA Technology to manage the MTP programme is due to end at the
end of October 2012; this presented an ideal opportunity to explore the outcomes and
impacts of the programme through its activities conducted between 2007 and 2011,
assessing the overall contribution of the programme to reducing energy use.
The primary focus of the evaluation was therefore to quantify the impacts influenced by
MTP and to therefore provide an estimate of value for money.
It was expected that the evaluation would inform Defra‘s decisions about:
The value of retaining an evidence base.
How the focus and management of the MTP contract post-2011 could be refined or
re-defined.
This evaluation of the MTP programme has therefore sought to establish:
1. Whether, and to what extent, the programme influences positive outcomes, with a
primary – but not exclusive – focus upon reduction in CO2 emissions.
2. On the basis of the impact attributed to MTP activities, whether or not value for
money has been achieved in securing outcomes.
3. How – and how effectively - the programme influences positive outcomes.
4. How effectively the programme is managed and how effective the current MTP
process has been.
Evaluation of the Market Transformation Programme
Final report
April 2012
6
1.3 Key findings
1.3.1 Achieving positive outcomes
The evaluation has demonstrated that the programme does influence positive outcomes
and – in doing so – has achieved very strong value for money. Using the total additional
impact of the MTP-influenced policy outcome (when compared to the policy outcome that
would otherwise have been achieved), the impact and value for money of the programme
are estimated as follows:
Taking into account the lifetime benefits of policy outcomes which MTP outputs
have had a strong influence upon, the total programme benefits derived from
2007-11 activities are at least £348m and 1.61MtCO2. Compared with a total
contract spend of £12.2m, this equates to influenced savings of
approximately £28.50 and 0.13tCO2 per £1 spent.
If lifetime savings upon which Defra – underpinned by MTP – have had some
influence are also taken into account, this equates to influenced savings of at least
£954m and 5.38MtCO2. Compared with a total contract spend of £12.2m,
this equates to influenced savings of approximately £78.20 and 0.44tCO2
per £1 spent.
With a number of Ecodesign and Energy Labelling implementing measures still to be
negotiated, ongoing development of UK procurement standards and voluntary agreements,
and an important role for programmes such as Energy Star and 4E, there is also ongoing
potential for the benefits derived from 2007-11 activities to continue to be realised.
1.3.2 How positive outcomes are achieved
The evaluation shows that development and possession of the types of output that MTP
produces have:
Given the UK a prominent role in pre-consultation scoping and a prominent
position at the European negotiating table;
Ensured a level of (almost automatic) support from some Member States for the
UK position;
Provided an important counter-balance to less progressive national / industry
interests;
Helped the UK to maintain some degree of control over the direction of policies
that could otherwise carry detrimental effects nationally;
Built good relations and reputation with stakeholders both domestically and
internationally;
Meant the UK does not have to use more political means of negotiating to a
desired outcome;
Helped to embed more effective monitoring and compliance.
For specific policy outcomes, there is a level of uncertainty as to the counterfactual – i.e.
what would have happened without MTP outputs – but overall it is clear that some policy
Evaluation of the Market Transformation Programme
Final report
April 2012
7
outcomes might have been very different without the UK taking an evidence-based
position.
A small number of stakeholders implied that other Member States can get by in
negotiations without the same support and that the UK – with a large number of votes –
could probably still enjoy some influence regardless of MTP. However:
Most respondents felt that a Member State would struggle to play an active role in
shaping policy without technical evidence bases and data.
Without MTP work, the UK could be less clear on which policy outcome would be in
the national / environmental interest and therefore what position to support.
The benefits of MTP outputs are not only realised in Commission negotiations, but
across a range of national and international initiatives and programmes.
1.3.3 How effectively the programme is managed and delivered
Whilst most respondents agreed that the type of outputs produced by MTP are beneficial,
most other Member States do not have a separate government-funded programme to
deliver these. The alternatives – as demonstrated by other Member States – were to have
advisors from within government (often energy / environmental departments or agencies)
or industry.
Alternative delivery methods – e.g. Defra directly managing experts - are options that
Defra may wish to consider. However, it is also important to highlight that the research
identified a number of arguments for continuing to source the required data through MTP
e.g. consistency, impartiality and building upon lessons learned.
Overall, whilst this evaluation indicates that there is value in both alternative
arrangements and retaining the current set up, it has certainly highlighted a number of
ways in which the current MTP process could work more efficiently and effectively:
Defra still need the outputs from MTP, but how much of the process of obtaining
them is now necessary?
There are recognised to be occasions where the UK has not been able to influence
policy outcomes. This suggests a need for more strategic assessment of whether or
not, for a particular product group or policy:
a. The effects of changes to a product group upon the UK are of enough
significance to warrant full investigation and analysis of data.
b. The eventual policy outcome is likely to be affected by the UK position.
c. The eventual policy outcome is likely to be influenced by possession of robust
data.
It should be borne in mind that where there is at least some impact of a policy /
product group upon the UK, there will still be a degree of analysis required for
carbon budgeting purposes. However, the resources devoted should be
proportionate rather than uniform.
Could the consortium be organised in line with Defra team i.e. on a product by
product basis?
Evaluation of the Market Transformation Programme
Final report
April 2012
8
One of the key issues cited by MTP for turnaround of required outputs was the
management and availability of subcontractors. This is an issue common to
subcontracting that perhaps should have been anticipated. Solutions include a
requirement in the contracts of independent experts to commit to delivering work
or making themselves available, or an expansion of the panel in order to minimise
the risk of expertise being unavailable for a particular product group or policy area.
A separate study is currently being undertaken to test the feasibility of the second
option.
Retain resources for workstreams where MTP impact is clear e.g. Eco-Design,
international programmes and future studies.
Although there is generally agreed to be no modelling system currently better
placed to form an assessment of the impacts of the policies under consideration,
there is a need for enhancements to be made to the quality of the assumptions
and modelling used in the MTP evidence base. It could be that more frequent
revisions to the IAs may be necessary to update the evidence bases used for
modelling, particularly in fast-changing markets such as consumer electronics and
ICT. Clear markings of the updates and the significant changes that have been
made is also recommended.
Furthermore, there are a number of policies for which the outcomes and benefits
at a UK level are unclear; there would be benefit in seeking to evaluate these
policies and quantify their impacts more accurately.
Evaluation of the Market Transformation Programme
Final report
April 2012
9
2 Introduction
2.1 Background
2.1.1 What is the Market Transformation Programme and what is it for?
The Market Transformation Programme is a consortium of expert contractors managed by
the Department for Environment, Food and Rural Affairs (Defra) and coordinated by the
lead consortium partner, AEA Technology (working alongside Consumer Research
Associates). There are also a number of additional technical advisers to the programme,
including the Building Research Establishment (BRE) and Intertek Research and Testing
Centre.
The aim of the Market Transformation Programme (MTP) is to drive improvements in
energy and environmental performance of energy using products, through the provision of
a robust evidence base on the current and likely future performance of energy using
products for policy makers in the UK, the European Commission, industry, and global
projects such as Energy Star and 4E.
The main outcomes that the programme aims to support are a real reduction in emissions
(e.g. CO2) and energy use through:
New and refined policies and standards;
Better informed policy decisions (through provision of robust information);
Innovation and improved product design.
2.1.2 What are the activities of MTP?
MTP is directly involved in the following activities:
Managing the energy using product evidence base, a key element of which is the
producing market intelligence and indicative standards for energy using products;
Assisting Defra (in negotiations with other member states) and the European
Commission in developing the eco-design of Energy Related Products (ErP)
Directive, which sets standards for the minimum energy performance of products
sold in the EU;
Contributing to international collaborations around standards and labelling e.g.
Energy Star and the International Energy Agency implementing agreement on
Efficient Electrical End-use Equipment (IEA 4E);
Influencing UK Government policy development;
Providing accurate assessments of policy impact for UK Carbon Budgets
accounting;
Up to 2009-10 - when this role was transferred to the National Measurement Office
(NMO) – assisting with compliance testing and the formulation of compliance
testing methods and standards.
MTP supports and drives forward these activities through its programme of work
(agreed with Defra).
Evaluation of the Market Transformation Programme
Final report
April 2012
10
2.2 Objectives of the research
The 2007-11 Defra contract with AEA Technology to manage the MTP programme is due to
end at the end of October 2012; this presented an ideal opportunity to explore the
outcomes and impacts of the programme through its activities conducted between 2007
and 2011, assessing the overall contribution of the programme to reducing energy use.
The primary focus of the evaluation was therefore to quantify the impacts influenced by
MTP and to therefore provide an estimate of value for money.
The evaluation was not expected to be an assessment of the rationale for the programme,
or the policies it assists with, particularly on the basis that Defra are looking to re-tender
the contract. However, Defra did see the contract renewal as a logical point at which to
test the focus, structure and delivery of the contract throughout the 2007-11 contract
period. In addition, the assessment of the impact of the programme inevitably raises
questions around the value of particular activities and / or how impacts could be enhanced.
It is expected the evaluation will inform Defra‘s decisions about:
The value of retaining an evidence base.
How the focus and management of the MTP contract post-2011 could be refined or
re-defined.
This evaluation of the MTP programme has therefore sought to establish:
1. Whether, and to what extent, the programme influences positive outcomes, with a
primary – but not exclusive – focus upon reduction in CO2 emissions.
2. On the basis of the impact attributed to MTP activities, whether or not value for
money has been achieved in securing outcomes.
3. How – and how effectively - the programme influences positive outcomes.
4. How effectively the programme is managed and how effective the current MTP
process has been.
Evaluation of the Market Transformation Programme
Final report
April 2012
11
3 Methodology
The section below provides a short overview of the research elements undertaken, the
rationale for these, and a brief summary of the approach to establishing programme
impact and attribution of that impact. A much more detailed methodology – and the
rationale for decisions around use of sources of impact data - can be found in
appendix A to this report.
The evaluation was split into two phases:
Phase 1 comprised desk research and interviews with MTP consortium
representatives. The purpose of this phase was to:
a. Ascertain the resources used to deliver the programme, and how these are
distributed.
b. Identify activities undertaken by the programme and outputs and impacts
arising from these.
c. Provide feedback on the programme logic, structure and day-to-day working in
order to enable a full process evaluation.
d. Enable a short-listing of product groups and policy outcomes where MTP
influence was felt to be strong. This was done on the basis that MTP activities
were numerous and wide-ranging, therefore it would be much more effective
for Defra and MTP to highlight areas where they believed there had been MTP
impact (which could then be verified) rather than seeking to gain specific
impacts from a large number of wider stakeholders.
There then followed an interim workshop with consortium representatives at
which phase 1 findings were presented and discussed, and attendees were able to
input further into any research questions pertaining to phase 1 objectives.
Phase 2 comprised interviews with 32 stakeholders, including Defra and AEA
product leads as well as representatives of the European Commission, member
states, industry and Energy Star programme. The purpose of these interviews was
to verify the influence of MTP upon the policy outcomes shortlisted in phase 1.
The attribution factors which would ascertain the extent of MTP influence upon
policy outcomes were developed into a two-factor assessment:
1. An assessment of how important MTP input was to the policy outcome realised,
on a scale of strong influence – some influence – weak influence – no
influence;
2. An assessment of how much confidence can be placed in this importance
rating, based upon such factors as number of respondents verifying, the source
of those verifying (were they authoritative / potentially biased?) and the level
of disagreement between respondents on the extent of MTP influence upon a
particular outcome. This was rated through a traffic light system of:
Green – good level of confidence
Amber – some confidence
Red – low confidence
Evaluation of the Market Transformation Programme
Final report
April 2012
12
To enable an assessment of impact influenced by MTP, the attribution factors had
to be applied to projections of the impact that influenced policies would have in
terms of UK CO2 and cost savings. These impact figures were drawn from the latest
carbon budget figures for policy impacts in each product group. These figures are
derived from the MTP Impact Assessments and reflected (in an aggregated form)
in the DECC Carbon Plan.
The attribution and impact data were integrated to produce categories of impact
i.e. impact from policy outcomes which MTP had a strong influence upon, impact
from policy outcomes which MTP had some influence upon etc.
The policy IA figures are calculated by taking the difference in impact between the
policy outcome and a scenario in which there is no policy at all. We have therefore
also reported impact in a different way to provide a more accurate representation
of the likely impact MTP directly influenced i.e. taking the difference in impact
between the actual policy outcome and the policy outcome that would have arisen
without MTP input. The IA and our approach are described respectively as the
‗policy vs. reference‘ impact and the ‗policy A vs. policy B‘ impact. The latter was
calculated by the following process:
- In MTPs internal ‗SEUP Benefits‘ document1, an attempt was made to estimate
the extent to which the impacts achieved by MEPS policy outcomes for each
product group were attributable to Defra / MTP contributions (looked at
another way, what impacts from policy for this product group would not have
been achieved). This effectively points to a policy B / counterfactual policy
outcome that would have occurred without Defra / MTP input. Due to the
qualitative nature of this exercise, these estimates were reported as % ranges
i.e. Defra / MTP can claim between x% and y% of the impacts derived from the
policy outcome for product group z.
- The mid-point of this % range was applied to the latest carbon budget impact
figures for each product group to provide an estimate of the additional impact
of the MTP-influenced policy compared to the policy that would otherwise have
arisen i.e. ‗Policy A vs Policy B‘.
Both sets of final attributed impacts were then compared to MTP budget data to
enable ‗value-for-money‘ assessments.
1 This document was produced by Andy Cotterill and Davide Minotti as an attempt to quantify the
impact MTP has achieved through its work. The impact figures presented in the document are based
on the impact assessments used in MTP‘s modelling work.
Evaluation of the Market Transformation Programme
Final report
April 2012
13
4 Process evaluation
This section uses data from the desk research, phase 1 consortia interviews and consortia
workshop to enable an assessment of the key successes and challenges of programme
delivery between 2007 and 2011. This feeds into the overall assessment of programme
appropriateness and effectiveness in section 7.
4.1 Programme logic map
The diagram below shows a logic map for the way in which the MTP programme results in
positive impacts for the UK in terms of reduced carbon emissions and financial savings.
Programme Inputs: Defra funding,
AEA and subcontractor expertise
Programme Workstreams:
Programme Management, EuP
Directive, Government Standards,
Evidence Base, International, Future
Studies, OGD and Ad-hoc, Specialist
Advice
Programme Impacts: Reduction
in energy usage, reduced CO2
emissions, financial savings.
Policies:
MEPs
and
labelling
Voluntary
Agreements
Energy
Star
Green
Procurement
Programme Outputs:
MTP
Evidence
Base
GSBNs Direct contact
with industry
Direct liaison
with
European
Commission
Ensure
s
com
pliance
Ensure
s s
ett
ing
of re
asonable
sta
ndard
s
Incre
ases
effectiveness o
f
policie
s
Evaluation of the Market Transformation Programme
Final report
April 2012
14
4.2 Post-2007 contract changes and rationale
The MTP programme has operated as a discrete programme since 1998. Since 2002, a
panel of experts, including ECI Oxford University academics and AEA Technology staff,
have worked with Defra to develop an evidence base which would allow Defra to influence
policy negotiations.
In 2007, the current MTP contract was tendered. This contract renewal contained three key
changes:
1. A change of structure to encourage the use of a wide panel of product and
energy experts; this was a stipulation explicitly introduced as part of the new
contract specifications. Guidelines were set that limited funding for programme
management to 20% of the total budget, with the remaining 80% ring-fenced for
procurement of outputs from the expert panel.
The benefits of this approach were argued to be a wider engagement and drawing
in of sector-specific and specialist expertise, a more open contract, potentially new
approaches / data and less reliance upon one contractor. The purpose of
continuing to outsource the management of the programme and coordination of
the panel was that this would ensure a consistent formatting of outputs,
streamlined reporting to Defra, and ensure that there were clear lines of
responsibility for managing and transforming Defra requests into outputs.
The programme management element was won by AEA Technology, who had also
been the previous lead contractor. Generally 10-20% of MTP analysis and reporting
is done by in-house AEA staff, and 80-90% is done by external subcontractors.
The outputs generally take the form of a short report with both high-level findings
and some detail of the analysis conducted. MTP managers will then communicate
this work back to Defra. Defra will then use the outputs for influencing EU policy
making and standard setting and reporting on UK carbon budgets progress.
2. A continued shift of focus from UK to European policy: this change occurred
during the course of the 2007-12 contract. Initially, MTP focused upon
development of the evidence base, modelling and UK policy such as the
Government Standards work. This came about as a result of the Energy White
Paper in which developing UK product policy was seen a key approach in helping to
meet 2050 carbon reduction targets.
Over time, with the development of the EU Ecodesign of Energy Using Products
(lately re-cast as Energy Related Products) Directive in particular product groups,
the programme focus has evolved towards responding to policy and standard
setting by the European Commission, in the form of minimum energy performance
standards, labelling and other interventions. This was intended to ensure that MTP
outputs could be relevant and influential at the European level, though the outputs
are also required for carbon budget accounting purposes.
Evaluation of the Market Transformation Programme
Final report
April 2012
15
3. A move from a more collaborative arrangement to one of Defra as clear
lead, especially on policy: with regards to MTP programme structure, the
number of Defra policy staff has increased over the contract from three to a team
of seven. In line with this, the extent to which the MTP consortium gets involved
with direct policy development and influencing has decreased. Defra now leads on
policy work with the European Commission, whereas before 2007 there was more
of a partnership approach to this between Defra and the consortium members.
The rationale for this change was that now that the Ecodesign Directive is being
progressed, there are potential risks for the UK government of being represented
by – and having UK policy outcomes negotiated by - independent consultants.
Defra policy leads now focus on specific product groups whilst the MTP consortium
now focuses more on producing the evidence base and modelling for two main
purposes:
Modelling and impact assessments (IAs) for Defra to use in EU negotiations
and carbon budget accounting.
IAs for economists, industry and other organisations e.g. other government-
funded programmes or agencies.
Defra continues to provide information to MTP on the product areas and analysis
required in response to EU negotiations. Defra may ask for MTP to conduct work at
a number of different points in the European Commission‘s legislative process,
including:
Pre-consultation (preparatory studies).
At the consultation stage, when a product area is being discussed, but before a
policy has been drafted.
At formal meetings after a policy has been drafted, where Defra would seek to
use the MTP evidence base to influence other member states.
Evaluation of the Market Transformation Programme
Final report
April 2012
16
4.3 Have the intended benefits been achieved?
The table below sets out the benefits arising from each of the three principal contract changes outlined above, and any issues arising from these, to
provide an overall assessment of how effectively the changes have delivered the benefits anticipated:
Programme
element Benefits observed Issues arising
Moving to the 80% -
20% delivery and
management split.
The programme is drawing in more individual expert resource and
knowledge through the sub-contracting arrangement.
This means more organisations are potentially able to contribute to
outputs which can deliver benefits through identifying new data
sources or analysis approaches.
For Defra, this represents an expansion of contacts and knowledge
base.
To some extent, the apparent diversity of the expertise drawn
upon in the new process is misleading, as a proportion of the
current panel of individual experts are ex-AEA staff who left the
organisation post-2007 and are now freelance.
AEA and Defra recognised that the change had led to a loss of in-
house management and expert skills, thus impacting upon quality
of outputs and coordination where inexperienced individuals were
taking on these management roles.
In addition, some AEA staff argued that the requirement to
outsource delivery of work-streams previously delivered in-house
has led to inefficiency and delay in turning around work as time is
needed for these external resources to be commissioned and work
sub-contracted i.e. experts were not available when required
because they have other work commitments which are now outside
of the contract and which AEA cannot manage / control. The
programme work plans did show certain outputs being delayed and
delivered beyond schedule, yet it is not clear how far this was the
case pre-2007.
Concern about delays and inefficiency in the process has led to
occasional deviation from the intended process i.e. one respondent
said that on some occasions the contract procedure is being
bypassed and the sub-contracted expert is reporting directly to
Defra rather than passing outputs through AEA.
Evaluation of the Market Transformation Programme
Final report
April 2012
17
A greater focus upon
European level policy
Assuming that MTP‘s core purpose is to provide data to inform
policy positions and outcomes, this change was essential to the
ongoing value and relevance of the programme.
It has ensured that Defra can bring useful and persuasive evidence
bases to the negotiating table to support their policy position and
influence the Commission. Section 6 explores and demonstrates
the influence that the UK – through use of MTP data – has enjoyed
in setting product policy.
Notice given for draft regulation by the EU can be very short,
giving little time for MTP to produce impact work on different
product policy scenarios; the inference from some respondents
was that this could impact upon quality and accuracy, or
turnaround times.
With support from Defra, MTP have attempted to organise
preparatory work to try to pre-empt which product groups might
be covered in legislation, though some requirements have still
necessitated long working hours for consortium staff. Attempting
to pre-empt requirements had also proved difficult due to the
changing nature of European policy formation. Whilst the
consortium felt that they had responded appropriately wherever
possible, this had been disruptive to normal, planned work and
resulted in working very long hours, with again the risk of
compromised quality.
Defra taking the lead
on policy
negotiations
This has ensured that the UK government are representing the UK
in all negotiations, ensuring impartiality and accountability.
However, this has not meant that MTP cannot be directly involved
in the process. Some experts are still being seconded to work with
the Commission on particular product groups and policies in initial
stages of development.
For some in AEA, the change in relationship between Defra and
MTP from one of a ‗partnership‘ (in which MTP‘s experts were
involved in shaping policy) to MTP becoming more of a service
delivery agent was seen to have led to less direct collaboration
between MTP and other EU and UK stakeholders and so less
strategic understanding from within MTP. This was felt by AEA
respondents to have led both to a stifling of innovation within MTP
– as they are reactive to Defra priorities - and a limit to the
additional impacts MTP could have through broader engagement.
One respondent strongly felt that the MTP has been “rather left
behind” as the policy agenda has moved on from that when the
programme was developed. They felt that this was at least in part
due to a failure of AEA to accept their altered role from leading the
thinking.
Evaluation of the Market Transformation Programme
Final report
April 2012
18
4.4 Other issues arising
Although not an inherent result of the changing programme process or structure, certain
aspects of the contract and programme delivery were highlighted by respondents in phase
1 as having created issues:
Some AEA respondents stated that quite high staff turnover at Defra (as well as
product group complexities) has created inefficiencies, in that it takes time to bring
new Defra staff up to speed with the background to – and technical issues
surrounding – a particular product group area. Whilst AEA have worked with new
Defra staff to bring them up to speed with particular product groups and the
policies affecting them, this is an ongoing requirement not explicitly accounted for
in the contract.
Closer Defra scrutiny of the assumptions and modelling used to produce the IA
documents have identified issues, in particular the economic basis for assumptions
about changes in product markets was cited as an area where there are problems
with accuracy. This has sometimes led to the need for Defra economists to re-work
figures or request re-calculation. It was perceived by several respondents that
there was a lack of economics expertise within AEA and the subcontractor panel;
despite Defra providing some training on this issue to MTP, it was felt by one
respondent that the wrong individuals had been sent to this and that the learnings
had not been used fully. It should be noted that Defra have since seconded an
economics expert to AEA.
It was felt by some AEA respondents that there could have been wider
communication of MTP outputs across industry and other national stakeholders.
That this was not done was felt to be largely due to a lower priority being placed
on communication of outputs to a wider audience, as well as a lack of specialist
marketing expertise. Examples were cited where MTP outputs had been used and
communicated by others (e.g. The Energy Saving Trust) more than by the
programme itself. Whilst MTP not promoting their own works is not necessarily an
issue, where others (such as EST) do not do it, engagement with wider
stakeholders could be more difficult.
One respondent felt that the work on the programme is not well targeted,
providing too much theoretical evidence which is “miles away from policies” and
that the individuals in AEA providing advice don‘t have a good policy
understanding2. They argued that Defra being tied into the 2007-11 contract and
not being able to renegotiate has been problematic and led to situations where AEA
would cite contract limitations as a barrier to taking on particular workstreams.
They felt that AEA has “a strong institutional memory and talks in 2005-speak”.
They did accept that Defra had been tough on the contract costs but also felt that
AEA could have been more creative in suggesting savings from the core tasks.
2 Examples cited were:
a. Work on potential tightening of government buying standards, which while theoretically
possible, went against three key constraints – the government‘s commitment to no
regulations, George Osborne‘s position on climate change and the need for growth.
b. A desire for fiscal measures as well as regulations and labels which the respondent felt was
bad policy making as there is loss of value in multiple measures.
Evaluation of the Market Transformation Programme
Final report
April 2012
19
A technical challenge highlighted is that as the programme has evolved, there has
been an increase in the complexity of the modelling system and the amount of
data it holds. The consortia interviews conducted indicate that for at least some
product areas, the system of Excel spreadsheets used to model different policy
scenarios has become inefficient to maintain and update. This was seen as a
potential risk to the modelling process if the demands on this system were to
further increase. One respondent described the MTP model as “a mess”, citing it as
one of the reasons why data provision from the programme is so expensive and
subject to delays. AEA were paid an extra £150k through the 2010-11 contract to
revamp the model but it is not clear how well some of the issues with it have been
resolved.
4.5 Implications
Overall, the process established for the 2007-11 MTP contract has still enabled the desired
outputs to be delivered and – as shown in section 6 – outcomes to be influenced.
In addition, despite the current contract structure and process being cited as responsible
for a number of issues, these should all be viewed in the context of:
a. The benefits obtained by Defra (outlined in the table in section 4.3) of the changes
to the contract;
b. Loss of influence and contract involvement for AEA; which always had the potential
to result in negative perceptions of some of the changes from AEA‘s point of view.
Some of the issues cited e.g. loss of proximity of MTP to European policy makers,
may not be seen by Defra as an issue at all.
c. Attempts have been made to address some of the issues that have arisen e.g.
training on economics to MTP policy advisors.
However, there is a suggestion that outputs produced are sometimes of questionable
quality and accuracy. Also, what is surprising about some of the issues raised is that some
issues arising from the 2007-11 changes could have been anticipated and processes /
procedures embedded to mitigate these. For example, the issues highlighted by AEA
around supplier delays, commissioning bureaucracy and lack of subcontractor control are
common to any subcontracting relationship.
Section 7 of the report considers the issues raised and provides some recommendations on
the ways in which these could be resolved.
Evaluation of the Market Transformation Programme
Final report
April 2012
20
5 Policy Impact data used
This section sets out the impacts of policies which MTP had potential to influence.
Rationale for the sources of these figures, their limitations and overlaps with other policies
can be found in the final sub-section of this section of the report, in the methodology
appendix and in the conclusions of this report.
As most robust monetary and carbon impact figures available for MTP‘s work, DECC-
approved policy impact data was used as the basis for programme impacts.
In general, impacts associated with the implementation of the EU Ecodesign and Energy
Labelling Directives have been drawn from the ‗SEUP Benefits‘ analysis and reported on a
product group by product group basis. Whilst some estimated impacts were found for other
policies (such as Green Procurement and voluntary agreements), these were generally EU-
wide, rather than UK-wide impacts, and often lacked the analytical detail of the impact
assessments which make up the ‗SEUP Benefits‘ analysis.
5.1 MEPS / labelling
5.1.1 ‘Policy vs. reference’ impact
Since the MTP Impact assessments view minimum energy performance standards (MEPs)
and energy labelling together, we have used the combined impacts of MEPs and labelling
for the relevant product groups when assessing impact. It was also felt that phase 2
respondents would be better able to assess MTP contribution to the suite of policies in each
product group rather than separating them out.
The table below shows the average annual net financial benefit to the UK of each product
group as set out in the latest departmental figures. These figures – drawn from the latest
carbon budget figures - are based upon (but updated from) MTP impact assessments.
The savings presented in the table below are savings net of the reference scenario - a
projection of what is likely to happen to the energy consumption of each product area if no
new policies were implemented i.e. a reference scenario.
The effect of three different scenarios on the product markets is modelled in MTP‘s work:
The Reference Scenario is a projection of what is likely to happen to the energy
consumption of each product area if no new policies are implemented; for
clarification, this scenario does include all policies that have been implemented by
a certain point.
The Policy Scenario is a projection of what will happen if a defined set of
additional product-specific and related cross-cutting policies are implemented fully
and in a timely manner;
The Best Available Technology (BAT) Scenario is a hypothetical projection of
impact if the best available technologies on the market (current and future) are
Evaluation of the Market Transformation Programme
Final report
April 2012
21
adopted. The scenario tends to assume implementation at the earliest possible
date and is unlikely to occur in reality, but gives an indication of potential savings.
The table below shows the average annual net financial and CO2 emissions benefit to the
UK of each product group as set out in the carbon budget figures (themselves derived from
the MTP IAs):
Product groups Average annual UK
net benefit from policies
Lifetime UK net benefit
from policies
Net annual CO2
savings predicted to be achieved in 2020 as a result of policies (Mt
CO2)
Standby £288m £3,165m 1.80
ESPUs £11m £125m 0.10
Electric Motors £92m £1,007m 1.30
Circulators -£11m -£119m 0.30
Cold appliances and labelling £44m £487m 0.60
TVs and labelling £64m £707m 0.70
Wet appliances and labelling £14m £149m 0.20
Domestic Lighting £124m £1,368m 0.50
Tertiary Lighting £214m £2,349m 1.30
Simple STBs £82m £899m 0.40
Total £922m £10,137m 7.10
5.1.2 Lifetime impact
The table below shows an estimate of the likely annual CO2 savings as a result of policies
in each product from 2010 to 2020.
This is included to illustrate the likely savings that would result from policies in each area
over time. These estimates have been produced by applying the net CO2 saving in 2020
produced in the departmental figures (itself derived from the published IAs and DECC
Carbon Plan) to the profile of net CO2 savings reported in each impact assessment. In
most cases, the impact assessments assume an s-shaped take-up curve between 2010
and 2020, resulting in higher net savings figures in 2020 than in the preceding years.
A take-up curve of this shape seems sensible intuitively, as it will take some time in the
early years of a policy for it to be widely adopted, and take-up will gradually plateau as the
market becomes entirely compliant. However, a critical issue is that the curve does not
seem to take into account the inevitable ramp down of the additional impact of the policy
as the market / general environment could have progressed to delivering these impacts
eventually3. Although this ‗ramp down‘ effect was adjusted for in carbon budgeting work,
this does not seem to have resulted in the declining net annual impacts that might be
expected.
3 In recent IAs this has been picked up by Defra economists and a revised approach is being
developed in response to this.
Evaluation of the Market Transformation Programme
Final report
April 2012
22
Product Area
Estimate of Net CO2 savings as a result of policies (MtCO2)
201
1 2012 2013 2014 2015 2016 2017 2018 2019 2020
2010-2020
lifetime
Standby 0.5 0.8 1.0 1.2 1.4 1.5 1.6 1.7 1.7 1.8 13.2
ESPUs 0.0 0.0 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.8
Electric Motors
0.1 0.2 0.3 0.4 0.5 0.7 0.8 1.0 1.1 1.3 6.4
Circulators 0.0 0.0 0.0 0.1 0.1 0.2 0.2 0.2 0.3 0.3 1.4
Cold appliances
and labelling
0.1 0.1 0.1 0.2 0.3 0.3 0.4 0.5 0.5 0.6 3.1
TVs and labelling
0.0 0.1 0.2 0.3 0.4 0.4 0.5 0.6 0.6 0.7 3.8
Wet appliances
and labelling
0.0 0.0 0.0 0.0 0.1 0.1 0.2 0.2 0.2 0.2 1.0
Domestic
Lighting 0.1 0.3 0.4 0.5 0.5 0.5 0.5 0.5 0.5 0.5 4.3
Tertiary Lighting
0.4 0.6 0.9 1.0 1.1 1.1 1.2 1.2 1.3 1.3 10.1
Simple STBs
0.2 0.2 0.3 0.3 0.3 0.4 0.4 0.4 0.4 0.4 3.3
5.2 Limitations of the carbon budgets and MTP IA data: the
necessity to explore ‘policy A vs. policy B’ impact
5.2.1 Limitations of the Impact Assessment and carbon budgeting figures
The MTP Impact Assessments attempt to quantify the net benefits of policies by looking at
the cost of implementation to manufacturers (through making products compliant) and
consumers (through increased product costs). This is compared to the benefits the policy
would bring (in terms of the value of additional energy and CO2 savings, and the value of
air quality damage avoided) to obtain an overall net benefit of the policy.
Other government policies which are expected to affect the energy efficiency of products in
each product group are included in the ‗baseline scenario‘ in each Impact Assessment to
ensure that neither the costs nor the benefits are double counted.
Most interview respondents felt that MTP modelling produced reliable and robust data
which they could trust. Interview respondents also mentioned the effect that central
government backing had upon the perceived quality of the data MTP produces.
However, some of those closest to the review and use of the IAs expressed some
reservations:
That the IAs are all ex-ante and there is insufficient ongoing tracking (though this
is being explored). A significant number of the GSBNs which are relevant to the
Impact Assessments appear to have been first published some time ago (some as
far back as 2008), although most appear to have been updated or reviewed in
2010 or 2011.
Evaluation of the Market Transformation Programme
Final report
April 2012
23
It can be difficult to establish from the IAs what is assertion and what is evidence
based. As with any modelling, there are limitations to the assumptions that might
underpin parts of MTP‘s impact assessments. Assumptions need to be made about
the future of fast-changing markets which, even when based on the best available
evidence, can deviate significantly from reality.
The savings are based on average annual savings over 10 years; therefore where a
small timing change has been enacted, this would make very little difference from
the reference scenario e.g. on standby.
There is an assumption that extra savings are as cost effective as the baseline
would have been; this does not seem a reasonable assumption.
Some research participants, particularly industry representatives, felt that some
modelling assumptions had been made while only considering part of the relevant
product market, rather than all of it. These interviewees did concede, however,
that getting the relevant data from their members in order to provide holistic
market representation can be difficult.
As well as these comments from the consortium interviews, some additional limitations –
specific to the MTP evidence base - were identified through an internal Defra review:
Benefits from minimum energy performance standards (MEPS) seem to increase
past 2020 despite rising energy prices. This has been highlighted as being contrary
to economic principles and suggests the modelling does not take account of the
interaction between energy prices and sales.
Costs to manufacturers upon the introduction of MEPS decline more sharply than
intuition suggests they should.
In some cases, little sensitivity analysis has been conducted to assess what would
happen to the modelled outputs if some assumptions were to change.
The models do not have the flexibility for analysis to be re-run with different input
values quickly.
It is sometimes unclear what data sources have been used or where assumptions
have come from.
Finally, having reviewed the data sources, our additional comments would be:
The policy impact cited in IAs is based upon a reference scenario of no policy being
introduced at all. Such a scenario is very unlikely. It is far more likely that a less
stringent policy would have been introduced, or a policy would have been
introduced later. This is the key driver for considering a ‘policy A vs policy B’
scenario.
The baseline for product groups is usually a straight line extrapolation of past
trends; the justification for this is not clear.
Looking at the impacts for product groups in isolation ignores real-world effects
that might mean increased use of one influences decreased use of another. For
example, heating and air conditioning.
The general principle of an s-shaped curve for impacts between 2010 and 2020
seems intuitive and sensible, because it will take some time for the benefits to be
fully achieved (standards complied with) and these will plateau as the market
becomes fully compliant with a policy.
However, the curve does not account for ‗ramping down‘ of policy impact over time
i.e. one has to assume that the market may have reached the standard regardless
of a specific policy; it may not be realistic to assume that net benefits will continue
Evaluation of the Market Transformation Programme
Final report
April 2012
24
to increase all the way up to 2020 and will then be consistent beyond that point, as
seems to be the case in most impact assessment curves.
It is important that the reader is aware of these limitations. However, the carbon budget
figures – partly derived from the IAs - have been used as the basis to calculate the
impacts of MTP-influenced policies because:
a. They have taken account of some of the limitations highlighted above.
b. In terms of ascertaining UK impact, the IAs are based upon primary data that is at
least as robust as that of other datasets that either Defra or other stakeholders
were aware of4.
c. The IAs and carbon budget figures have taken into account – and attempted to
address – issues that could have arisen from simply aggregating impact
assessment data e.g. policy overlap and years from which figures are drawn.
d. The IAs and carbon budget figures disaggregate impacts by product group in a way
that fits with the attribution approach.
In addition, Defra representatives felt that the standard of IA - in terms of transparency
and publishing – has improved recently, and that more conservative assumptions are
starting to be used, driven by the need for carbon budgeting.
For comparison, the lifetime impacts from the IAs explored was compared to the predicted
impacts of the E3 standards and labelling Program in Australia and New Zealand. The
estimated lifetime economic benefit of this was estimated at AUD$22.4 billion by 2024 plus
NZD$5.1 billion2 to the New Zealand economy by 2036. The UK impact assessment
estimates therefore seem proportionate and within the right order of magnitude when it is
considered that:
The IA data in the table covers only the product groups that have been selected for
attribution evaluation; the total lifetime figure taking into account all products is
larger.
The E3 impact reflects business and industry benefits as well.
5.2.2 Rationale for establishing a ‘policy A vs policy B’ scenario
In summary, although there are significant limitations to the IA data that underpin the
calculations of MTP impact and value for money in this evaluation, it was generally agreed
that these figures were the best available data and that the IAs were broadly accurate in
assessing MEPS & labelling policy impact against a counterfactual of no policy at all.
However, as MTP have rarely influenced the idea of introducing a policy for a particular
group, it would be a substantial overestimate if the current IA and departmental figures
were used as a guide to the impact directly influenced by MTP.
For example, for standby, MTP data was predominantly used in order to secure the
introduction of more stringent standards 1-2 years earlier than would otherwise have been
4 A secondary aim of the phase 2 interviews was to allow stakeholders the opportunity to cite any
alternative impact figures that might be more credible than those in the MTP IA documents. Although
some stakeholders expressed reservations about the way in which the MTP impact assessment and
forecasting model data may have been calculated, they generally conceded that this data was ‗as good
as it gets‘.
Evaluation of the Market Transformation Programme
Final report
April 2012
25
the case. A MEPS policy would have been introduced regardless and more stringent
standards would almost certainly have been in place by the third year from the
introduction of the MEPS. It is therefore misleading to suggest that MTP influenced
anything more than the marginal impacts derived from two years of having more stringent
requirements, as the vast majority of the lifetime savings from the policy would have been
delivered anyway. Yet the IA does not take account of this.
Therefore, as well as showing the level of MTP influence against total ‘policy vs.
reference’ impacts, section 6 of this report also attempts to provide a more
accurate estimate of actual MTP impact using ‘policy A vs. policy B’ impacts i.e.
the marginal / additional impact of the MTP-influenced policy outcome compared
to the policy outcome that would otherwise have been realised.
This ‗policy A vs. policy B‘ impact was established through:
a. Using the qualitative data on attribution to understand how the MTP-influenced
policy differed from the policy that would otherwise have been introduced.
b. Using the section of the SEUP benefits document which estimates a portion of the
overall ‗policy vs. reference‘ impact which MTP had a direct influence upon. As
described in section 4, MTPs internal ‗SEUP Benefits‘ document5 estimates the
extent to which the impacts achieved by policy outcomes for each product group
are attributable to Defra / MTP contributions. This effectively points to a policy B /
counterfactual policy outcome that would have occurred without Defra / MTP input.
Estimates were reported as % ranges i.e. Defra / MTP can claim between x% and
y% of the impacts derived from the policy outcome for product group z. The mid-
point of this % range was applied to the carbon budget impact figures for each
product group to provide an estimate of the additional impact of the MTP-
influenced policy compared to the policy that would otherwise have arisen i.e.
‗Policy A vs Policy B‘. Using this approach ensures a more conservative
estimate of total impact.
The % ranges shown in the SEUP Benefits document are as follows:
Implementing Measure Percentage
influence in EU
Standby 25% - 30%
ESPUs 7.5% - 12.5%
Electric Motors 20% - 30%
Circulators 30% – 50%
Cold + Label 1% to 3%
TVs + Label 20% - 30%
Wet + Label 5% - 10%
Domestic Lighting 25% - 35%
Tertiary Lighting 7.5% - 12.5%
Simple STBs 15% - 25%
5 This document was produced by Andy Cotterill and Davide Minotti as an attempt to quantify the
impact MTP has achieved through its work. The impact figures presented in the document are based
on the impact assessments used in MTP‘s modelling work.
Evaluation of the Market Transformation Programme
Final report
April 2012
26
5.3 Other policy impacts
In addition to input into the setting of standards for the above product areas, phase 1
interviews identified other policies to which MTP may have contributed. The impacts of
these policies are discussed below.
There are several issues regarding the impact data available for these policies:
1. Since the majority of the impact figures discussed below are EU-wide impacts
rather than UK-specific impacts, disaggregation to a UK impact has not been
possible.
2. The basis for some of the calculations is not clear.
3. The ranges on some of the impact estimates indicate that these estimates may not
be robust.
On this basis, MTP influence on most of these policies is discussed without attempting to
quantify their UK impact.
5.3.1 Voluntary agreements
Voluntary agreements take the form of European Codes of Conduct and individual
agreements with particular industry sectors. Voluntary agreements can also take the place
of a regulation under the EU Ecodesign Directive e.g. complex set top boxes and imaging
equipment. A list of current codes of conduct is given below, along with an estimate of EU-
wide savings resulting from the code (or, in the case of the code of conduct for data
centres, UK-wide savings):
Code of Conduct for Data Centres – estimated UK savings from this code are
4.7MtCO2 between 2008 and 2014.6
Code of Conduct for Digital TV Services – an EU wide implementation of this code
could result in an overall reduction in energy consumption of 8TWh per year in the
EU, an equivalent saving of around €750m per year.7
Code of Conduct on Energy Consumption of Broadband Communication Equipment
- an EU wide implementation of this code could result in an overall reduction in
energy consumption of 25TWh per year in the EU, an equivalent saving of around
€7.5bn per year.8
Code of Conduct on Efficiency of External Power Supplies – this code of conduct
will result in a maximum reduction in electricity usage in the EU of 5TWh per year,
equivalent to a total saving of €500m per year.9
Code of Conduct on AC Uninterruptible Power Systems (UPS) – no estimated EU-
wide or UK-wide savings for this code of conduct were available.
6 http://efficient-products.defra.gov.uk/cms/eu-code-of-conduct-for-data-centres-launched/ 7 http://re.jrc.ec.europa.eu/energyefficiency/pdf/CoC_Digital_TV-version%208_2009.pdf 8
http://re.jrc.ec.europa.eu/energyefficiency/pdf/CoC%20Brodband%20Equipment/Code%20of%20Con
duct%20Broadband%20Equipment%20V4%20final%2010.2.2011.pdf 9 http://re.jrc.ec.europa.eu/energyefficiency/pdf/CoC_Power_Supplies_Version4-March2009.pdf
Evaluation of the Market Transformation Programme
Final report
April 2012
27
5.4 Green Procurement
Government Buying Standards cover the minimum performance of energy using products
which can be purchased by government. Currently only one impact assessment of the
effect of this policy has been undertaken, relating specifically to ICT equipment. This
impact assessment estimates the net benefit of the policy on consumer savings to be
£24m over the next 10 years10.
5.5 Other
MTP has also provided some information and data to help develop the Carbon Emissions
Reduction Target (CERT) and Energy Saving Trust Recommended (ESTR) schemes.
These initiatives have not been explored at the attribution stage as there is limited data
relating to their individual impacts on the UK, and attribution of any impacts to MTP was
considered to be negligible by phase 1 respondents.
5.6 Wider benefits
5.6.1 European Benefits
The internal ‗SEUP Benefits‘ document attempted to quantify the additional impact that EU
policies will have Europe-wide, by calculating a rough ‗approximating factor‘ to scale up UK
savings to a European level. This approximating (extrapolating) factor is derived by
dividing EU carbon emissions by UK emissions. The document acknowledges that this
method of estimating EU-wide savings is only illustrative, and relies on a number of
assumptions, some of which are clearly not without issues; these assumptions include:
That the relationship between energy and CO2 impact is constant throughout the
EU;
That there is similar potential for energy efficiency improvements in products
throughout the EU;
That sales and use of products is constant throughout the EU;
That there are relative uniform prices (e.g. for energy, values of CO2) across the
EU.
In addition, applying the same factor to all product areas implies that EU member states
will be impacted to the same degree as the UK by the policies in each product group. In
reality there are likely to be product areas which impact some member states more than
others depending on the relative abundance of each product within that country. Despite
the limitations of this method and inherent complexity of producing EU-wide impacts for all
product groups, it is clear that EU energy efficiency standards will have some impact on
other EU member states in the same way in which impacts have been quantified for the
UK.
10 Based upon figures provided by Defra in the document ―Impact Assessment of Revised
‗Government Buying Standards‘ specifications for Information and Communications Technology –
Computers‖. It is important to note that although this document is still a draft, the impact figures in it
are not expected to change. The £24m is a midpoint between the predicted impact range of £16m and
£32m.
Evaluation of the Market Transformation Programme
Final report
April 2012
28
5.6.2 Global impacts
The Energy Star programme also influences benefits beyond the UK. For example, Energy
Star estimates that in the last five years, the US Energy Star programme for office
equipment resulted in savings of more than 223TWh and energy bill savings of $22 billion
for the US. It also estimates that in the last three years, the EU Energy Star office
equipment programme is estimated to have resulted in savings of around 10TWh, or more
than 2 billion euros11. MTP provides independent data to the programme to support the
development of product standards criteria.
MTP has contributed to the International Energy Agency‘s 4E‘s programme (Implementing
Agreement for a Co-operating Programme on Efficient Electrical End-Use Equipment). The
programme‘s Mapping and Benchmarking annex makes use of MTP data and publications
in order to achieve its aims:
The mapping provides an overview of the energy efficiency performance of certain
electrical end-use equipment (or energy using products) in several countries
and/or regional areas and a brief summary of the main policy measures regarding
energy efficiency of products in those areas;
The benchmarking compares average and best performance of products put on the
market in different countries and/or regional areas; analysing variations between
different markets; sharing best practices and lessons learnt by, for example,
highlighting potential for further improvement of energy efficiency levels globally.12
MTP also provides data and evidence to support the efforts of the Super-Efficient
Equipment and Appliance Deployment (SEAD) in influencing transformation of global
markets.
5.6.3 MTP work achieving non-carbon impacts
The work of MTP in influencing UK and EU product policy has other potential impacts aside
from the reduction in carbon emissions and UK consumer savings associated with an
increase in efficient products in the market. These include raw material savings, water
savings and additional revenue and market share increases associated with the
introduction of these products.
As well as these more tangible benefits, some interviewees felt that by using MTP‘s
outputs, Defra was improving the UK‘s international reputation and / or influencing other
countries to carry out similar modelling on their own product markets, leading to better
standards of efficiency beyond the UK.
Although these non-carbon impacts should be considered as part of the overall value of the
MTP, they have not been quantified or included in the value for money analysis.
11 Promotional email from the Energy Star programme 12 http://mappingandbenchmarking.iea-4e.org/
Evaluation of the Market Transformation Programme
Final report
April 2012
29
6 Attribution: Evidence of MTP influence
6.1 Stakeholder awareness of and engagement with MTP.
As a criteria of their selection, all phase 2 stakeholder respondents were aware of MTP
(though respondents did note that some of their colleagues / other member states may
not be). However, the majority of respondents at the Commission and from other Member
States had limited direct contact with MTP, reflecting the division of responsibilities under
the current contract.
Industry representatives‘ main contact with MTP arises through consultations issued by
Defra. A few respondents also noted further engagement around data sharing but this still
tends to be largely within the context of a consultation response. All industry respondents
have an on-going dialogue with Defra involving regular meetings with Defra
representatives.
Respondents‘ understanding of the distinction between MTP and Defra was mixed. Many
respondents at the Commission were unclear if MTP was a separate programme or whether
it was a section of Defra itself. Often respondents were not clear if they were engaging
with someone from Defra or MTP, though this does not seem to have influenced their
perception of the effectiveness of Defra or MTP. Industry respondents in UK trade bodies
were clearer on the distinction.
6.1.1 Perceptions of MTP
Respondents‘ perceptions of MTP were generally positive, though given the limited direct
contact of many with MTP, few respondents were able to comment authoritatively. Some
commented more generally about the objectives of a programme such as MTP rather than
MTP‘s work in particular.
Member State and Commission representatives were generally very positive. In particular,
MTP was praised for its independence, technical capability and the extent of its evidence
base and research. The long term strategic approach to developing the evidence base was
also commended. The individuals working on MTP were seen as at least competent and
often expert in their field. Two respondents felt that the MTP research and evidence base is
more extensive than that developed by equivalent sources for other member states.
Positive statements from European-level respondents regarding MTP included the
following:
“The UK, through MTP, has the most extensive database. It is a serious partner,
also trusted by industry. MTP has a good line between industry and member
states. They provide good technical input.”
“MTP is valuable as it is the most complete source; it‟s seen as a reliable and
government-backed official source that can be trusted.”
―An excellent evidence base. No other country undertakes this type of research.”
―It is a thoughtful and well set up programme. It is good that it had a very
consistent philosophy on the role of evidence making.‖
Evaluation of the Market Transformation Programme
Final report
April 2012
30
―A good structure and long term plan which is very useful for focusing on technical
information and data which is very important and lacking at the EU level.....there is
a clear advantage of MTP, who have a continuous database that they are building
up. Here we could learn from MTP.”
The industry view of MTP was more mixed. MTP was praised for its engagement with
industry and noted for its independence. However, one industry respondent was negative
about MTP‘s work, feeling that their association was having to drive the moves towards
adoption of higher standards in the sector, with little or no enthusiasm or technical input
from MTP (who they felt were under-resourced). Positive responses from industry
representatives included:
“MTP play a very useful role. They are very responsive and very active. They
engage well with industry.‖
“It is good to have a programme to show actors in the market that there are
alternatives with a better environmental footprint and that they usually make
economic sense as well. There is a need to cover the „death valley‟ between proof
of concept and market adoption.‖
“Broadly, the MTP process as a whole is healthy. I think the fact that Defra
established an independent evidence base is good. I think the fact that it is
consultancy-based is reasonable – I can‟t judge if outsourced consultancy is better
than in house. The people we deal with are genuine and competent.‖
6.2 Verification of MTP influence on MEPS and labelling
6.2.1 Phase 1 anticipated influence
Minimum energy performance standards were identified as likely to be the area where MTP
has had the clearest positive impact, through Defra and the Commission using the MTP
evidence base to:
1. Prompt the development of new standards for product groups which otherwise
would not have minimum standards set;
2. Influence higher standards to be set by the Commission, usually through increased
scope or more stringent requirements;
3. Bring forward the timescale for introduction of standards that would have been
introduced anyway but less quickly;
4. Influence the maintenance of a standard proposed by the Commission, which
would otherwise have been lowered/diluted due to influencing agendas of other
member states.
Labelling follows the same regulatory process as MEPs and has been evaluated alongside
them, but where possible impacts have been identified separately as they act on product
markets in a different way i.e. more on demand side. MTP is likely to be having an impact
on labelling by:
1. Influencing the introduction of labelling on products that would otherwise not have
had labelling;
2. Influencing the energy performances pertaining to different labelling
categorisations;
Evaluation of the Market Transformation Programme
Final report
April 2012
31
3. Influencing the format and structure of the label to make them clearer for
consumers;
4. Influencing to maintain energy performance labelling categorisation standards
proposed by the Commission, which would otherwise have been lowered/diluted.
6.2.2 Acknowledged general UK / MTP influence
The UK was noted by almost all member states and Commission representatives as being
amongst the member states most consistently calling for more stringent requirements or
for earlier adoption of measures and is generally viewed as making a contribution to the
outcome of negotiations in most areas to some degree.
UK representatives to the Commission are highly regarded by all respondents. They were
described as active, authoritative, constructive, well informed and good at engaging
stakeholders. The UK was repeatedly identified as one of a small number of very active
and vocal Member States; other cited were the Netherlands, Germany, Italy, Sweden, and
Denmark:
[The UK is] one of, if not the most active of, Member States.‖
“Together with the Netherlands and sometimes Sweden, the UK were the most
constructive Member State. They have a really in depth knowledge of the subject.
Their comments were extremely well founded, and constructive, they have solid
data to back up their arguments. This proved to be extremely useful for the
Commission. In the end it improved the quality of the regulation.”
―The UK delegation is well informed, gives clear opinions and proposals to push
things forwards.”
―There is no doubt that the UK is one of the active Member States on both Eco-
design measures and Energy Labelling (and Energy Star too) and in most cases
advocates more stringent levels or a more early adoption. Although such action is
certainly needed, it does not always result in changes because other Member
States are advocating in the other direction.‖
―The UK government representatives are well respected in the EU and in
Commission work. There is an advantage for them in terms of both language and
in a long experience of stakeholder consultation. Whenever Defra speak it is
authoritative. The feedback I get is that there are only four to five member states
speaking up and Defra is one of them. Defra acts as a counter weight against large
member states that have a different agenda.‖
One Commission respondent commented that the UK almost always inputs and even if it
does not get a majority vote will likely be able to secure a blocking minority. Respondents
felt that this position partly reflected the fact that the UK does not a have a large
manufacturing base and that other Member States were more protective of national
competitiveness reflecting large national manufacturing interests. Another noted that it
reflected that the responsibility for product policy sat within the UK environment ministry
and therefore was more influenced by ambitions under climate change policy.
Furthermore, the evidence base possessed by Defra was viewed by the Commission,
member states and Defra themselves as being a contributory factor to the strength of the
UK contribution. Those involved in negotiations felt that MTP data has played an important
role in supporting Defra to defend their own position in the face of opposition from other
Evaluation of the Market Transformation Programme
Final report
April 2012
32
Member States, though it is not clear the extent to which this is also supporting a common
position with the Commission. The UK was cited by several member state respondents as
backing up their position with data and using that to be constructive; this was not felt to
be the case for all Member States. The UK was noted for being specific about the
requirements they were seeking; again due to the work conducted. Respondents felt that
the evidence base was an advantage for Defra and strengthened their position.
“[The UK is] trying to use evidence as the basis for their positions and I think in
this the MTP was quite useful to sharpen their arguments.”
―It is already quite an advantage that the UK gathers this evidence. Often the
positions of some members states are purely based on the input they get from
industry; not that the UK does not listen to industry. But there is this added value
here of trying to make this evidence based policy making.‖
―It adds a lot of weight to the debate – it is a transparent approach, which is open
for all to see.”
―It [the UK input] was very technical in a positive sense, very focused on product
and issues that could come up, definitions. Very informed, very detailed technical
knowledge and analysis. The UK is making a strong contribution with strong
technical content.”
―[The UK is] quite influential; on one hand because of being large and having lots
of votes, but normally they give constructive comments based on data, which you
don‟t have in every case. It is to do with their ability to provide numbers, to
specify the precise value they want for requirements, not just call for it to be
stricter in general.”
One respondent noted that the UK used the evidence base in a way that it was able to
change the nature of the debate. The UK representatives present the evidence and then
invite discussion around the findings. Therefore it is not simply a debate around the
validity of the UK position but also one about the evidence base itself: ―They did not have
to bring forth arguments like others, the normal diplomatic wheeling and dealing. Their
evidence base elevated the discussion beyond a power struggle with the votes.‖
Another noted the importance of the technical evidence base in adding to confidence in the
UK position and authority at the negotiating table; the UK was more likely to be heard in
the debate even if other states might theoretically disagree with the UK. Other
respondents noted more generally the importance of an evidence base for being
influential: ―Having evidence is very important. You normally can‟t change Commission
proposals without the evidence that it is possible to have stricter requirements without
compromising function or products or the structure of the market.”
In several areas MTP data has supported the Commission by filling a gap in the evidence
base or providing more up to date data: ―When there are new products, products that
have not been dealt with before, there are very large gaps in the technical data available
and often we are scrabbling around for information. Commission preparatory studies are
often a collection of what information is already out there rather than commissioning new
data collection and scientific studies.‖
Several respondents noted the lack of available data when putting together preparatory
studies and that the Commission is often reliant on data from member states or industry
when it comes to negotiations that might take place several years later. Regarding the
Evaluation of the Market Transformation Programme
Final report
April 2012
33
latter source, respondents commented on the value of having a view independent of
industry, who would be aware how data was likely to be used: ―Having the technical input
was very important, for the Commission too. The regulations are based on the preparatory
studies and these are sometimes finished two or three years before the adoption
procedure so the data can often be out of date. We are counting on data provided by
industry and they always downgrade the real efficiency improvement potential as they
know the information they give will be used for setting legal requirements. So it is useful
for the Commission to have an outside independent evaluation which is up to date at the
time of negotiation. And either confirms or pushes upwards the requirements we adopt.‖
One respondent identified an important issue that the current Commission requirement –
being pushed by the UK - for centralised EU-wide data to be made available is not being
met and there is no clear responsibility for who should coordinate it. The Directive states it
should be provided by manufacturers on free and accessible websites but someone needs
to collect it. The Commission is planning to centralise collection of product test data with
the development of a database where manufacturers will be encouraged to voluntarily
submit their product data for new products. The aim is for the market to push action by
manufacturers, though the degree to which this is achieved and the speed at which a
substantial volume of accurate data can be collated is not clear.
6.2.3 Barriers to identifying and achieving influence
Despite this general affirmation of the scale and authority of UK input (and the importance
of the MTP data in enabling this), most respondents from Member States were unwilling or
unable to give a clear statement that the UK had been responsible for specific outcomes.
All commented on the consensus-based nature of the negotiations and the balance of
technical and political arguments.
The process takes on average four years to complete; two years for the preparatory study
and two years for consultation and negotiation. In particular, the consistently divergent
views of Member States with or without manufacturing interests was noted as a common
theme of most debates and this always necessitates the need to move to a compromise
position that was acceptable to a majority of parties. The challenge of attributing specific
policy outcomes to the UK, let alone MTP, was articulated by respondents as follows:
“It is difficult - or even impossible - to verify claimed impacts of one specific actor
in these processes...in general any impact is „group work‟.‖
“It is collaboration. Even if you disagree, as you discuss you find a compromise
which is a good solution for both countries. All the information the UK provides is
considered but it is not possible to say if any UK information is considered more
than others.‖
Another reason respondents at the Commission appeared to be unwilling to comment too
explicitly on the UK influence was the politically sensitive nature of commenting on the
influence of any one Member State. One explicitly stated he was unable to comment on the
basis of confidentiality linked with the commitology procedure; others made similar
comments.
Evaluation of the Market Transformation Programme
Final report
April 2012
34
A few respondents from the Commission and Member States were more direct about the
influence of the UK but were still unable to be very specific in identifying examples of the
UK influencing a particular outcome.
“The UK provide clear positions and lots of ideas. They are one of the few active
Member States. They are perceived as one of the driving forces in these
processes.”
“A position is always shaped by the UK.”
As well as highlighting unwillingness to directly attribute policy outcomes to the UK, the
interviews identified limitations to the extent to which Defra can use MTP‘s work to
influence EU policy are limited by a number of factors:
Some EU member states – especially those with large manufacturing bases –
approach the EU negotiations with different priorities to the UK. Possession of a
strong evidence base may be insufficient in the face of national priorities.
Inevitably, much of MTP‘s evidence base work is UK-focused and designed to
ascertain the benefits of particular policies to the UK market, therefore the extent
to which it is felt to be authoritative outside the UK may be affected by differences
in product use, predominant product models and the extent of other demographic
differences.
Where there have been concerns about MTP modelling or assumptions, the
evidence base was less likely to be used or pushed forward in negotiations,
therefore less likely to influence.
There are also likely to be instances where the MTP data has endorsed what the Defra
position would have been anyway, or where the Defra position has aligned with the most
likely outcome at European Commission level. In these instances, evidence from MTP will
only have been used to endorse an outcome that probably would have occurred anyway,
and so would not result in any additional impacts.
6.2.4 Exploring the counterfactual
To attempt to overcome respondents‘ inability to precisely assess the influence of the UK –
and MTP – on policy outcomes, respondents were asked about their views on how
effective Defra (or the UK) would have been if operating without the evidence base.
Respondents commented that the UK would not be as effective without the technical
evidence base they currently have. Respondents felt that without MTP, the UK would have
to default to a political negotiating position and rely more on gathering support from
industry: ―I think they would have negotiated like other big member states, to gather
people to support their case and not necessarily back things up. They did not have to bring
forth the normal diplomatic wheeling and dealing.” Another respondent went further,
stating that the lack of an evidence base could mean the member state cannot exercise
much influence: “Those that can afford it invest in technical resources. This is very
important if a Member State is to be active in the discussions.‖
It was noted by some respondents that the UK has a large number of votes and that other
Member States without MTP-type technical support can also be active. However, given the
counter discussion on the importance of the evidence base for the UK position it can be
inferred that the UK would not be in such a strong position as States that have no evidence
Evaluation of the Market Transformation Programme
Final report
April 2012
35
base but have a national interest to defend and are pushing for maintaining the status
quo. It was certainly felt that the UK would not be able to specify a specific position that
benefitted a progressive policy agenda.
It was also felt that the Commission would lose an important and consistent data source.
Although one respondent felt that the UK itself would not feel the impacts of a world
without MTP for around two years, they did state that the Commission would feel the loss
sooner.
Another source of insight into the importance of MTP is provided by comparison with the
situation in other Member States. The UK is unique amongst the other Member States
interviewed in having a technical resource that is structured and funded like the Market
Transformation Programme. However, in many cases, Member State representatives will
also have technical support in the meetings with the Commission. Where there is a
national energy agency, the technical experts are from that agency. If there is no agency,
the ministry representative tends to bring external experts. Where their support is limited
in terms of resource, there may be more collaboration with like-minded Member States.
Germany, Italy and the Netherlands all have technical support provided by a government
agency, though it was also noted by some respondents that Italy‘s manufacturing
companies may prepare evidence bases to demonstrate the implementation costs / issues
around certain standards.
Therefore, in comparison, some Member States who are also proactive in the discussions
at the Commission have also invested in a technical resource, but this is usually within the
existing government administration / energy related agencies. The scale of that investment
tends to reflect the scale of the national manufacturing base for energy using products.
This also influences which part of the government lead on the discussions, with economics
ministries leading where there is a large manufacturing base and environment ministries
leading where it is predominantly a consuming country. Where there is less investment
and a limited technical resource, this is strengthened by building closer links with similar
minded Member States.
6.2.5 Specific influences at European level
As well as strengthening the data input to the regulatory process, MTP has also
strengthened the technical capability. One Commission respondent noted that the technical
input from MTP for standby and TVs has been influential in clarifying queries and speeding
up the regulatory process.
For ICT products - computers, displays and imaging equipment - MTP is currently providing
additional technical staff to work directly for the Commission given the limited resources in
the Commission and a delay after the preparatory study. MTP felt that this was a good way
of influencing the Commission and being able to push for more stringent requirements and
was able to help the Commission distinguish technical from political views. Overall, it is not
clear to what extent MTP has contributed to more stringent outcomes in this area as the
process is not yet finished. However, it is clear that MTP has made a positive contribution
enabling the Commission to develop their proposal in the direction of more stringent
proposals.
Evaluation of the Market Transformation Programme
Final report
April 2012
36
MTP is noted by several respondents for its input to the development of test standards for
verification and compliance testing. This was noted for standby with Bob Harrison (an
independent MTP expert consultant in the area of consumer electronics) being noted
particularly for his contribution to test standards for simple set top boxes and standby. The
contribution to test standards for TVs was also noted but there was some uncertainty
around the specific outcomes claimed by MTP. Other specific contributions to verification
and test standards were verified - to varying degrees - for wet appliances. MTP also
claimed a contribution to test standards on EPSUs, electric motors and tertiary lighting
although this was not verified by any other respondents.
Many Commission respondents noted that Defra has been very active on market
surveillance although Defra funds the NMO to principally overseeing compliance at the UK
level.
6.2.6 Refutation of influence and critiques of MTP data and the UK position
One Member State respondent felt that the UK position had at times focused too strongly
on technical points that, in their view and from their own analysis, were of marginal or
even negligible impact in terms of energy savings. They felt that the UK has argued too
strongly and for too long on some of these points and that the technical input from MTP
was partly responsible for this.
6.2.7 Summary of influence on MEPS and labelling
The table below summarises the extent to which MTP impacts claimed in phase 1 regarding
MEPS and labelling have been verified through the phase 2 research:
The extent to which Defra/MTP had influenced the outcome of the policy making process -
both generally and for specific claimed impacts - is rated as follows:
Strong influence - MTP (alone or through Defra) were critical in influencing the
outcome;
Some influence – MTP (alone or through Defra) made some contribution to the
outcome ;
Weak influence – MTP (alone or through Defra) made a slight difference to the
outcome;
No influence - MTP (alone or through Defra) made no difference to the outcome.
The level of confidence in that assessment of influence; established through a ‗traffic light‘
system:
Green – there is good verification of the Defra / MTP claim, with at least one
respondent is able to authoritatively verify the claim and there is a good match in
the detail for specific claims.
Amber – there is some verification of the claim; at least one respondent is able to
verify the claim but there is some uncertainty in the recall of the detail of specific
claims.
Red – there is no verification of the claim made by MTP / Defra or there is
disagreement between respondents or such a large degree of uncertainty that
verification of the actual influence on the outcome is difficult to discern.
Evaluation of the Market Transformation Programme
Final report
April 2012
37
Hypotheses
Product group Introduced More stringent/
earlier/ speedier Held EC line
Compliance – do
we need this here?
‘Policy vs. Reference’
Impact assigned to
the policy from latest
departmental figures
(average annual
impact)
‘Policy A vs. policy B’
impact assigned from
review of attribution
information and SEUP
% apportionments
(average annual
impact)13
Standby No influence claimed Some influence Some influence Some influence £288m / 1.80MtCO2 £28.8m / 0.18 MtCO2
EPSUs No influence claimed Weak influence No influence claimed Weak influence £11m / 0.1 MtCO2 £1.1m / 0.01 MtCO2
Electric motors No influence claimed Some influence Some influence Some influence £92m / 1.3 MtCO2 £23m / 0.325 MtCO2
Circulators No influence claimed Some influence Some influence No influence claimed £-1114m / 0.3 MtCO2 £-4.4m / 0.12MtCO2
Cold appliances
and labelling No influence claimed Some influence No influence claimed No influence claimed £44m / 0.6 MtCO2 £0.88m / 0.012MtCO2
TVs and
labelling Some influence Strong influence Some influence Strong influence £64m / 0.7 MtCO2 £16m / 0.175MtCO2
Wet appliances
and labelling Some influence Some influence No influence claimed No influence claimed £14m / 0.2 MtCO2 £1.05m / 0.015MtCO2
Domestic
lighting Some influence Some influence No influence claimed No influence claimed £124m / 0.5 MtCO2 £37.2m / 0.15MtCO2
Tertiary lighting Weak influence Weak influence Weak influence No influence claimed £214m / 1.3 MtCO2 £21.4m / 0.13MtCO2
Simple STBs Strong influence Some influence No influence claimed Strong influence £82m / 0.4 MtCO2 £16.4m / 0.08MtCO2
13 Figures for this column show average annual impact except for standby, where figures are fixed because the impact was timing rather than stringency. 14 Despite having a positive environmental impact, EU policies relating to circulators were calculated to have a slight negative financial impact in the UK.
Evaluation of the Market Transformation Programme
Final report
April 2012
38
The table shows the following:
That where MTP influence has been identified and verified, this influence has
usually been contributory – ‗some influence‘ - rather than critical. No phase 2
respondents were able to identify a critical role for MTP. This partly reflects the
underpinning role of the programme and the complex nature of the regulatory
process it is trying to influence i.e. a balance of political and technical arguments,
multiple stakeholders with varying degrees of involvement and a lengthy
timetable.
That where MTP influence is most often recognised within the consortium, this
tends to be either influencing the stringency of a policy or accelerating its
introduction rather than influencing the establishment of a policy where there
would have been none. Discussion with Defra and MTP in Phase 1 indicated that
there would be less activity in this area since the EC are fairly rigorous in their
listing of products for consideration.
It is also difficult to assess the extent to which MTP has also played a role in
supporting Defra and / or the Commission in holding a position in the face of
opposition from other Member States. This was not a situation generally
recognised by respondents, given the perception of the UK as proactive and
dominant in calling for more stringent or earlier requirements.
The clearest verification of claims from Defra/MTP for this influence was
established for standby, TVs, simple STBs and wet appliances.
6.2.8 Influences on non-priority products
The phase 2 interviews brought to light a number of examples of MTP influence upon
policies affecting products not included in the priority products focused upon in this
evaluation. These comprised the following:
As part of MTP‘s direct support to the Commission on the development of
standards for ICT products, gaming consoles were incorporated into the proposed
regulation. The original Commission proposal was to base the requirements on
Energy Star which only covers office equipment and would have excluded the sale
of higher energy consuming gaming consoles and gaming PCs and possibly led to
illegal imports or imports of components with no regulation on energy efficiency.
For network standby, the Commission desk officer noted the UK has provided a
strong contribution with a strong technical content but mainly around definition
and packaging rather than the level of ambition.
For vacuum cleaners, the UK has been particularly active in the discussions around
measurement of energy consumption. This was noted both by respondents from
the Commission and from industry. The Commission stated the UK input was very
useful and they contributed well.
For air conditioning, a respondent from Germany noted the UK led calls for stricter
requirements with support from other Member States, but was not sure if MTP data
or evidence was used.
6.3 Influence on Public (Green) Procurement
MTP was acknowledged to have played a significant role in the establishment of public
procurement standards in relation to ICT (as an MTP specialist on this product group was
consulted by Defra. For other energy using products, MTP provided some input, though
Evaluation of the Market Transformation Programme
Final report
April 2012
39
this was less holistic and mainly covered energy use rather than other considerations such
as recyclability. A Defra representative did acknowledge that Defra had not asked MTP for
input beyond energy use.
A Defra respondent reported that initially, the provision of information for public
procurement was an opportunistic move to use information already developed by MTP as
part of their work on MEPS and labelling. Therefore it originally only covered products
under the Ecodesign and Energy Labelling Directives. Provision of this information for
procurement was an ‗add on‘ to the main body of MTP‘s work.
Going forward, an opportunity was identified by Defra to develop a more effective and
structured green public procurement programme. The expertise and access to support
offered by MTP was noted as enabling Defra to set up a proper programme of support.
The respondent currently responsible for this area reported that MTP still provide support
to Defra for energy using products but that these have become smaller aspects of the
procurement standards. MTP now provide cost benefit analyses as required under a
framework agreement. MTP recommendations are generally accepted. These support Defra
in continuing to raise the energy performance requirements in procurement standards
across departments. The respondent felt that MTP are well respected and their input is
generally seen to be accurate and of a high standard.
Overall, MTP has been critical in enabling the introduction of new procurement standards
for ICT. MTP has a continuing role in contributing to more stringent standards on this and
other product groups. This reflects the extent of the evidence base developed for their
work on MEPS and labelling.
Policy Introduced More stringent/ earlier/
speedier
Public green procurement Strong influence Strong influence
6.4 Influence on voluntary agreements
For EU Codes of Conduct, the UK is again noted by Commission and Member State
respondents as one of a few active Member States contributing to the discussion at the
Commission level; Denmark, the Netherlands, Austria and Germany were also cited in this
regard. This has largely been through calling for stricter requirements and test standards
rather than through calling for new product areas to be covered.
The technical consultant to the Commission acknowledged the involvement of MTP in
external power supply units (EPSUs), simple set top boxes (STBs), and broadband
equipment. The relevant desk officer at the Commission recalled UK input to discussions
around upcoming agreements on complex set top boxes and imaging equipment.
Like the discussions with MEPS and labelling, respondents again found it difficult to identify
a specific influence of the UK on the outcome of negotiations in relation to voluntary
agreements. Again this reflected the large number of stakeholders and consensus driven
Evaluation of the Market Transformation Programme
Final report
April 2012
40
nature of the negotiations. However, the input from MTP was viewed positively, in
particular for its ability to provide an independent view: ―There is good consultative
dialogue amongst many parties, so it is difficult to say who has the share of the outcome.
It [MTP] is a serious partner, also trusted by industry. MTP has a good line to industry.‖
The Commission consultant in this area noted the UK contribution has often been
predominantly through representation by MTP; Defra participation declined from earlier
days although the UK participation overall is less than it was.
MTP is currently providing additional technical staff to work directly for the Commission on
ICT products, given the limited resources in the Commission (also working on MEPS and
Energy Star). Support is being provided on a voluntary agreement for imaging products
and MTP are pursuing for more stringent requirements although the Commission has not
adopted these. It is likely these savings will be further reduced given the arrangements
for managing the agreement. MTP are also writing a mandatory measure in case the
voluntary agreement fails.
The Codes of Conduct have preceded MEPS for several product areas. Influencing
voluntary agreements is also a way in which MTP has then gone on to influence the setting
of MEPS, though this has not always been visible to other stakeholders.
Policy More stringent/ earlier/
speedier
Compliance with the
agreement
Voluntary agreements (EU
codes of conduct) Some influence Some influence
6.5 Influence on Energy Star
There is good evidence that MTP has contributed to more stringent requirements for
Energy Star by providing an additional technical resource directly to the Commission,
enabling them to put together a robust and ambitious proposal.
MTP has provided input to the criteria for the voluntary ‗best in class‘ Energy Star label in
the past through its presence on working groups reviewing proposals from the US
Environmental Protection Agency (EPA) who own the Energy Star. An MTP representative is
currently working directly for the Commission to support them on the revision of Energy
Star standards for computers, displays and imaging equipment (also supporting the
Commission on MEPS and labelling and voluntary agreements for ICT products). MTP is
currently advising the Commission on a more ambitious proposal for displays than that
proposed by the US EPA. MTP claims that it has achieved some of its proposals for
displays (e.g. around sleep and off modes) but the process is not yet finished.
The technical support provided by MTP is welcomed by the Commission. According to the
desk officer at the Commission, the USA provides an estimated 80% of the technical input
to Energy Star, therefore it is considered important that Europe / the Commission is able
to provide technical input as well. The Commission Desk Officer reported that the input is
„taken on board and is useful‟ but does not personally get involved with the technical
detail. He commented that the UK is one of four to five Member States that provide input
Evaluation of the Market Transformation Programme
Final report
April 2012
41
but this is not a controversial area as there is not such a large European Manufacturing
base. The debate is more about what is achievable and cost effective from a technical point
of view.
The technical consultant to the Commission was more positive and felt that the input from
MTP has influenced at least part of the increased stringency of the standard: ―Most of the
influence will be by having good technical arguments and analysis that supports the
arguments. It is the product of many people discussing so it‟s difficult. But I believe they
[MTP] have supported the process and part of the outcome is because of MTP‟s input.
Compared with input from other member states I would say MTP has generally provided
the most useful analysis of the issues. Not just providing opinions but providing analysis in
spreadsheets.‖
Overall, MTP has contributed to more stringent requirements for Energy Star, which
started out as very much a standard with minimum requirements. This has been
particularly strong where MTP is working directly for the Commission.
Energy Star also has a link to the Minimum Energy Performance Standards in that it has
formed the basis of some MEPS for ICT.
Policy More stringent/ earlier/ speedier
Energy Star Some influence
6.6 Other UK-specific policy initiatives
Further policy initiatives, in particular UK policies, were discussed in phase 1 but have not
been explored specifically in phase 2 as they were considered out of scope or of minor
importance / impact. MTP and respondents in phase 2 did identify a limited number of
impacts on UK policy initiatives and these are presented here.
For motors, MTP claimed an influence on the Carbon Trust‘s updating of the Enhanced
Capital Allowance (ECA) scheme with the new IEC testing standards, the same as those
used for Ecodesign and Energy Labelling measures. This effectively makes the ECA more
ambitious than previously and the measurement methods used are more accurate. There
was no verification of this claim.
For circulators, MTP claim their work on circulators led to industry proposing to the
Department for Communities and Local Government (CLG) that they include in their 2010
building regulations a requirement that circulators are labelled with the industry developed
A-G label. This was included in the regulations published in March and is expected to
promote the sales of more efficient circulators until 2013, when the Ecodesign minimum
standard will require circulators to be at least A rated. The monitoring and reporting of
market share of products in this group was acknowledged.
For simple set top boxes, MTP contributed to the work on identification of best practice
standards through collaboration with EST and Digital Tick. This initiative gave recognition
Evaluation of the Market Transformation Programme
Final report
April 2012
42
to the best performing products in the market (top 10-15%). The standards were
developed in discussion with MTP consultants and criteria peer reviewed by Bob Harrison
and others. This was a demand side measure designed to help consumers choose the most
efficient products on the market. There was no verification of this claim.
The respondent from the Energy Saving Trust stated that MTP data is used for general
savings calculations for all household appliances, research, the Energy Saving Trust
Recommended scheme and the UK Voluntary Retailer Initiative on the TVs. EST use the
briefing notes, in particular the reference scenario and base case, and current market
average performance usage data. The data is useful partly because it is the main and most
complete source: “There is not much else out there.”
The respondent stated that part of the benefit of using MTP data is that it saves EST
calculating savings from first principles based on information from the energy label. If MTP
was not there, EST would probably have to commission testing of more products and
commission more research (for which they do not have funds). MTP provides a reliable and
government-backed official source that they felt can be trusted.
As stated, MTP data is used directly for Energy Saving Trust Recommended label to
provide the baseline saving against which the proposed ESTR standard is compared. MTP
also receive test data from ESTR.
MTP were also involved in the TV Voluntary Retailer Initiative. However, the main contact
for this at EST has since left and could not be contacted. The respondent at EST thought
MTP supported the project and may have provided some of the calculations.
MTP data was also a key input to their recent publication „The Elephant in the Living
Room‟; this was produced partly to access the wealth of evidence held by MTP and make it
more readable and accessible to a wider audience. This is in a similar vein to two earlier
publications - „Rise of the Machines‟ and ‗the ampere strikes back‟. These were joint
initiatives between MTP and EST and were developed in part to showcase the public data
that MTP held.
Evaluation of the Market Transformation Programme
Final report
April 2012
43
6.7 Assigning impacts to attribution bands
The table below takes all the impacts associated with the various policy initiatives and
assigns them to attribution bands. For each assignment, the traffic light system also shows
the level of confidence in this assignment, based upon the factors set out in the attribution
methodology section.
The figures in the table refer to the overall saving to the UK on each product arising from
introduction of Ecodesign regulations. They do not amount to the unique contribution of
MTP / Defra to the individual products i.e. over and above the contribution of other factors.
The reasons for this are explained in the methodology appendix.
Level of MTP
influence
Product group /
policy
‘Policy vs. reference’
impact (average annual
UK net benefit / net
MtCO2 savings in 2020)
‘Policy A vs. Policy B’
impact (average annual
UK net benefit / net
MtCO2 savings in 2020)
Strong
influence
upon outcome
Green Procurement £2.4m £2.4m
TVs and labelling £64m / 0.7 MtCO2 £16m / 0.175MtCO2
Simple STBs £82m / 0.4 MtCO2 £16.4m / 0.08MtCO2
TOTAL £148.4m / 1.1 MtCO2 £34.8m / 0.255MtCO2
Some
influence
upon outcome
Standby £288m / 1.80MtCO2 £28.8m / 0.18 MtCO2
Energy Star
Voluntary Agreements
Domestic Lighting £124m / 0.5 MtCO2 £37.2m / 0.15MtCO2
Cold appliances and
labelling £44m / 0.6 MtCO2 £0.88m / 0.012MtCO2
Wet appliances and
labelling £14m / 0.2 MtCO2 £1.05m / 0.015MtCO2
Electric Motors £92m / 1.3 MtCO2 £23m / 0.325 MtCO2
Circulators £-11m / 0.3 MtCO2 £-4.4m / 0.12MtCO2
TOTAL
£551m / 4.7 MtCO2, impact
of Energy Star and Voluntary
agreements
£86.53m / 0.802MtCO2
Weak
influence
upon outcome
ESPUs £11m / 0.1 MtCO2 £1.1m / 0.01 MtCO2
Tertiary Lighting £214m / 1.3 MtCO2 £21.4m / 0.13MtCO2
TOTAL £225m / 1.4 MtCO2 £22.5m / 0.14MtCO2
Key:
Green – High confidence in influence assessment
Amber – Medium confidence in influence assessment
Red – Low confidence in influence assessment
Evaluation of the Market Transformation Programme
Final report
April 2012
44
The charts below summarise the separation of annualised attributed impacts using both
the ‗policy vs. reference‘ and ‗policy A vs. policy B‘ impact figures:
Figure 1: Breakdown of ‘policy vs. reference’ impacts against attribution
categories established in phase 2
Figure 2: Breakdown of ‘policy A vs. policy B’ impacts against attribution
categories established in phase 2
From figure 2, overall, 88% of the CO2 impact influences and 85% of the £ savings
influences which MTP claimed in phase 1 have been verified i.e. the evaluation finds that
MTP had strong or some influence upon the policy outcome.
Evaluation of the Market Transformation Programme
Final report
April 2012
45
6.8 Value for money analysis
This section uses the data collated in sections 5 and 6 of this report to provide an
assessment of the value for money being delivered by the MTP.
6.8.1 Factors considered in the analysis
The costs and benefits that have been taken into account to enable the calculation are as
follows:
Value for money is calculated using both the ‗policy vs. reference‘ and ‗policy A vs.
policy B‘ figures.
The lifetime – rather than annual - benefits of the policies are used. The lifetime
factors work in the same way for both ‗policy vs. reference‘ and ‗policy A vs. policy
B‘ impacts, with the exception of standby. For standby, as the marginal additional
impact relates to quicker implementation rather than increased stringency, only
the additional impact - derived from the two years where the more stringent
standards might not have been in place without MTP – are used.
Lifetime £ benefits are derived from the annual attributed impacts table in section
6.7 and are assigned a lifetime factor of x10. Lifetime CO2 benefits are derived
from the table in section 5.1.2 of this report to ensure that the are consistent with
carbon budget figures.
As they are not quantifiable, additional benefits are not included.
Costs are derived from the total Defra spend on the programme for the 2010-11
budget year. Although there are costs to UK industry of implementing / complying
with new policies and initiatives, these are already accounted for within the ‗net‘
impacts. The table below shows the total programme spend for the Market
Transformation Programme in 2007-1115:
2007 - 8 £2.7m16
2008 - 9 £2.7m
2009 – 10 £2.7m
2010 – 11 £2.2m
2011 - 12 £1.9m
This makes a total contract spend – between 2007 and 2012 – of £12.2m.
In addition to the budgeted programme spend, for the last three years the person
cost for Defra of managing the programme has been approximately £260,000 per
annum17. However, as the Defra team also have other responsibilities, assessing
15 All £ figures are for the budget year April to March; spend for each year was roughly equally split
between six month periods. 16 Estimated (on the basis that the budget for 2007-8 also included production of SCP evidence) 17 Based upon Defra salary estimates.
Evaluation of the Market Transformation Programme
Final report
April 2012
46
the precise cost to Defra of managing MTP would rely upon assumptions. Therefore
this cost has been excluded from cost effectiveness calculations.
6.8.2 MTP Value for money
Using the programme cost information, the following table sets out MTP value for money
for a scenario in which:
a. Only outputs where MTP have had a strong influence are taken into account
b. Any outputs where MTP have had at least some influence are taken into account
‗Policy vs. reference‘
‗Policy A vs. Policy B‘
Strong
influence
only
Lifetime savings
£1,484m and 7.1MtCO2 £348m and 1.61MtCO2
Impacts per £ spent
£122 and 0.58tCO2 £28.50 and 0.13tCO2
Strong
and
some
influence
Lifetime savings
£6,994m and 36.5MtCO2 £954m and 5.38MtCO218
Impacts per £ spent
£573 and 3tCO2 £78.20 and 0.44tCO2
Although the ‗policy A vs. policy B‘ figures are comparably smaller than for the ‗policy vs.
reference‘ figures:
These figures are more cautious and carry less potential for misinterpretation
These figures still show exceptional value for money, especially when it is
considered that unquantified impacts from efforts on Energy Star and Voluntary
Agreements have not been incorporated.
18 Figures for this cell take account of the fact that, for standby, attributed impact figures should not
have a lifetime factor applied because the impact was timing rather than stringency.
Evaluation of the Market Transformation Programme
Final report
April 2012
47
7 Conclusions and recommendations
This section draws together the report findings around process, attributed impact and
wider benefits to inform our response to the evaluation objectives. This section seeks to
answer the following:
Is there value in the outputs being produced?
Are there opportunities to deliver these outputs – and their outcomes - in a more
efficient way?
Are there opportunities to deliver further benefits to the UK?
7.1 Is there value in the outputs?
At a UK, European and global level, MTP data and outputs have been used to influence
policy outcomes and programmes which carry substantial CO2 and consumer saving
impacts. The programme has also delivered good value for money when comparing the UK
benefits of policy outcomes influenced to programme budget.
The findings of this report provide strong evidence that – at a European policy making
level – the UK enjoys substantial influence upon the eventual policy outcome and that this
influence is underpinned by what the Commission and other Member States perceive to be
a robust evidence base.
The research suggests that development and possession of the types of output that MTP
produces have, amongst other benefits:
Given the UK a prominent role in pre-consultation scoping and a prominent
position at the European negotiating table;
Ensured a level of (almost automatic) support from some Member States for the
UK position;
Provided an important counter-balance to less progressive national / industry
interests;
Helped the UK to maintain some degree of control over the direction of policies
that could otherwise carry detrimental effects nationally;
Built good relations and reputation with stakeholders both domestically and
internationally;
Meant the UK does not have to use more political means of negotiating to a
desired outcome;
Helped to embed more effective monitoring and compliance.
For specific policy outcomes, there is a level of uncertainty as to the counterfactual – i.e.
what would have happened without MTP outputs – but overall it is clear that some policy
outcomes might have been very different without the UK taking an evidence-based
position.
A small number of stakeholders implied that other Member States can get by in
negotiations without the same support and that the UK – with a large number of votes –
could probably still enjoy some influence regardless of MTP. However, there are three
counter arguments to this view:
Evaluation of the Market Transformation Programme
Final report
April 2012
48
1. Most respondents felt that a Member State would struggle to play an active role in
shaping policy without technical evidence bases and data.
2. Without MTP work, the UK could be less clear on which policy outcome would be in
the national / environmental interest and therefore what position to support.
3. The benefits of MTP outputs are not only realised in Commission negotiations, but
across a range of national and international initiatives and programmes.
With a number of Ecodesign and Energy Labelling implementing measures still to be
negotiated, ongoing development of UK procurement standards and voluntary agreements,
and an important role for programmes such as Energy Star and 4E, there is also ongoing
potential for the benefits derived from 2007-11 activities to continue to be realised.
Overall, it is therefore clear that there has been – and continue to be – value in
the UK working to collate and analyse data on energy using products and forecast
benefits.
7.2 Opportunities for more efficient delivery?
7.2.1 Why MTP?
Whilst most respondents agreed that the type of outputs produced by MTP are beneficial,
most other Member States do not have a separate government-funded programme to
deliver these. The alternatives – as demonstrated by other Member States – were to have
advisors from within government (often energy / environmental departments or agencies)
or industry.
With so many industry stakeholders expressing willingness to engage in data collation and
providing advice to government, and an enhanced Defra team - does the UK need to
deliver these outputs through a separate programme?
As another alternative, one phase 1 respondent felt that the programme outputs would be
better if Defra were to directly manage groups of experts and delegate specific technical
tasks to this group. This would allow more ad hoc resource use and potentially more
efficient delivery.
Alternative delivery methods are options that Defra may wish to consider. However, it is
also important to highlight that the research identified a number of arguments for
continuing to source the required data through MTP:
1. Even by the admission of some industry associations, useful data held and
provided by members is hard to find and inconsistent in quality. It would therefore
be a risk for the UK to rely upon such sources.
2. MTP is impartial and without vested interest in a developing data which supports a
particular outcome.
3. MTP is a recognised and respected brand at the European level. If MTP – or its
outputs – were simply to become a Defra-managed subsidiary which is called upon
for evidence for a regulatory position, the data – and so the UK – could lose some
of its reputation for impartiality and integrity.
Evaluation of the Market Transformation Programme
Final report
April 2012
49
4. Through the contract, Defra have established a fairly consistent method of
specifying and obtaining the data required in the format required.
Overall, whilst this evaluation shows that there is value in both alternative
arrangements and retaining the current set up, it has certainly highlighted a
number of ways in which the current MTP process could work more efficiently
and effectively. These are discussed below.
7.2.2 Improving efficiency within the current framework
Although the value for money delivered by the programme is good, the research has
highlighted a number of ways in which the same benefits could be realised with less
resource:
Defra has invested heavily in the development and maintenance of the evidence
base and associated models which the consortium has developed. Now that the
architecture is set up, Defra should consider the extent to which it may wish to
further develop the models or alternatively consider setting contracts based on
outputs. In other words, Defra still need the outputs, but how much of the process
is now necessary?
The tendency in the current contract has been to develop outputs for any product
group under consideration. For some policy outcomes, it was felt by phase 2
respondents that the UK had effectively supported an outcome that would have
been very likely regardless of their input. In a similar vein, as indicated by the
exclusion of various policy outcomes for the phase 2 verification, there are
recognised to be occasions where the UK has not been able to influence policy
outcomes. This suggests a need for more strategic assessment of whether or not,
for a particular product group or policy:
a. The effects of changes to a product group upon the UK are of enough
significance to warrant full investigation and analysis of data.
b. The eventual policy outcome is likely to be affected by the UK position.
c. The eventual policy outcome is likely to be influenced by possession of robust
data.
When seeking to influence policies, Defra may wish to give more priority to those
outcomes and policies having the highest potential benefit for the UK. This in turn
could mean a more robust evidence base is required for some policy types within
MTP‘s modelling than others. It should be borne in mind that where there is at
least some impact of a policy / product group upon the UK, there will still be a
degree of analysis required for carbon budgeting purposes. However, the resources
devoted should be proportionate rather than uniform.
Whilst the current MTP structure focuses around the development of the evidence
and models, the Defra team have evolved to lead on particular product groups. It
is therefore worth considering whether the consortium could also be organised on
a product by product basis.
One of the key issues cited by MTP for turnaround of required outputs was the
management and availability of subcontractors. This is an issue common to
Evaluation of the Market Transformation Programme
Final report
April 2012
50
subcontracting that perhaps should have been anticipated. Straightforward
solutions include a requirement in the contracts of independent experts to commit
to delivering work or making themselves available, or an expansion of the panel in
order to minimise the risk of expertise being unavailable for a particular product
group or policy area.
Although one Defra respondent acknowledged that efforts had been made to
identify savings that could be made in the MTP contract, one of the key issues in
the 2007 contract renewal was that the only bids competing with AEA for the
management role were not high quality19. Defra should work to encourage more
bidders in any new commissioning process in order to maximise the potential for
new and potentially more efficient approaches to be identified.
7.3 Enhancing programme effectiveness
7.3.1 Retaining work in key areas of impact
All phase 1 respondents saw the following MTP work-streams as key, and thought they
should be preserved in future contracts:
The Eco Design work stream was seen as crucial for influencing EuP legislation and
labelling at present and for the potential to influence ERP in the future. This is also
borne out by the phase 2 verification work.
International work, seen as important because it entails working with other big
markets such as Australia, the USA, China and Brazil enables Defra to influence big
manufacturers and drive up their standards. Examples cited included the SEAD
work and the Energy Star work.
Future studies, seen as important in working out likely scenarios for products and
product development and also on technical areas that can improve the evidence
base.
7.3.2 Enhancing the impact of current outputs
Drawing together both respondent views and our experience of using the policy
outcomes impact data, it is clear that although there is currently no modelling
system better placed to form an assessment of the impacts of the policies under
consideration, there is a need for enhancements to be made to the quality of the
assumptions and modelling used in the MTP evidence base. Furthermore, there are
a number of policies for which the outcomes and benefits at a UK level are unclear;
there would be benefit in seeking to evaluate these policies and quantify their
impacts more accurately.
It could be that more frequent revisions to the IAs may be necessary to update the
evidence bases used for modelling, particularly in fast-changing markets such as
consumer electronics and ICT. Clear markings of the updates and the significant
changes that have been made is also recommended.
19 Defra source.
Evaluation of the Market Transformation Programme
Final report
April 2012
51
Something noted by several respondents in phase 1 was the lack of communication
MTP outputs and capabilities. It would be worthwhile for Defra to consider how
best to communicate MTP outputs and the audiences it wishes to communicate to
in order to improve this in future. For example, providing briefing notes to product
buyers might allow them to push for more efficient goods from their suppliers.
Some respondents suggested that the consortium should have an economist within
their team to help deliver Defra‘s expectations in this area. Closer working with
economics teams within Defra and maybe DECC was another useful
recommendation. Defra has recently seconded an economist to AEA.
Several industry and agency representatives in the UK suggested that more
information on where the MTP evidence base comes from would be useful to make
it easier to assess how robust and therefore how useful and reliable it is. They
welcomed the introduction of the risk matrices with published data and research.
MTP may wish to consider alternative formats in which the modelling and evidence
base information could be stored, or the potential for separate front and back-end
systems to allow the modelling results to be obtained quickly while retaining the
functionality of the modelling process. The ‗What-If‘ web modelling tool has been
removed from the MTP website. Whilst the number of regular users of this tool was
likely to have been small, the detail and capability that it provided may well have
led to positive impacts through use by other organisations or policy makers.
Three industry respondents suggested that more industry input could be useful to
MTP or Defra. They suggested that this could be a review by a small industry group
(either at the start of the study process or before publication) to ensure the
appropriateness of the scope or approach. They felt that this would provide a sense
check from the market view. Two of these respondents acknowledged that this
raised challenges in terms of resource and fairness that would have to be
considered and managed appropriately.
7.3.3 Future opportunities
Inevitably, if MTP resource is being focused upon policies where little or no influence is
being achieved, there may be opportunities being missed in other areas. The key
opportunities for the programme in 2012 and onwards are set out below:
Greater potential savings can be made through changes in EU policy; however,
Defra and MTP could be more likely to have an attributable influence on domestic
policy changes, including ‗beyond compliance‘ voluntary agreements and
procurement standards.
MTP should also consider where it may be best to achieve outcomes through
working with different parts of the product markets. MTP currently works on both
supply side (though minimum MEPS, encouraging innovation, and responsibility
deals) and demand side (through co-labelling, influencing consumer purchasing
behaviour). As part of considering outcomes, MTP should consider relative
priorities given to each type of work. Which has proven to be more/ less effective
through past activities?
Evaluation of the Market Transformation Programme
Final report
April 2012
52
Overall, it is clear that MTP has evolved significantly over the period of the last
contract and so it is important to future-proof delivery wherever possible in the
future.
Defra need to assess what the likely changes are which will occur in UK, EU and
international policy in future and what impact these may have on the desired
direction for MTP.
Evaluation of the Market Transformation Programme
Final report
April 2012
53
Evaluation of the Market Transformation Programme
Final report
April 2012
54
8 Appendix A: Full methodology statement
Sections 8.1– 8.4 describe the overall approach taken to completing the different research
elements.
Section 8.5 and 8.6 describe the approach taken to establishing policy impact and
attribution of these to MTP.
8.1 Phase 1: desk research
The desk research focused on three types of document:
MTP programme inputs, including work plans and reports outlining the activities
the MTP programme had undertaken;
MTP outputs, including Briefing Notes and the ―Saving Energy Through Better
Products and Appliances‖ report;
Policy documents for areas which MTP has the potential to provide influence
leading to carbon impacts.
The table below shows the full list of documents included in the review, along with the
rationale for including each i.e. what information pertaining to the evaluation was obtained
from them:
Document Title(s) Rationale for inclusion in review
MTP Work Plan 2010/11 – Issue 4
Outlined activities undertaken by MTP and the spend allocated to those activities
MTP Work Plan 2011/12 – Issue 3
MTP - Proposed budget for 2011/12
Provided detail on the resources allocated to different workstreams
MTP Organogram Set out the MTP management structure and lines of reporting
Carbon Plan Action Summary Set out MTP's role within the context of the Carbon Budget
UK Carbon Plan Provided understanding of the policy context for MTP priorities and activities
IEA 4E Mapping and Benchmarking Annex
Provided information on this exercise and MTP‘s input to it
Defra's structural reform plan Explained how wider Defra spending and reform has affected MTP
MTP Quarterly Reports (Quarter 1, 2010/11 - Quarter 1, 2011/12)
Outlined activities undertaken by MTP and the spend allocated to those activities
MTP Evidence Base Progress Update Brief November 2010
Stated the progress of key MTP Evidence Base workstream tasks up to the end of November 2010
Evaluation of the Market Transformation Programme
Final report
April 2012
55
Modelling the energy use of products - A review of approaches from practice
Identified potential options for AEA to improve the cost effectiveness of the MTP modelling approach
MTP Evidence Base - Future Strategy
MTP EuP Monthly report – July 2011
Showed progress against budgets and deliverables for the EuP workstream against each project's original aims Reports for Energy using Products
workstream June 2009 - January 2010
Net Benefits from MTP IAs and latest departmental figures
Provided Defra and MTP views on the extent to which Defra has influenced EU policy and outcomes. The ‗SEUP Benefits‘
Defra contribution estimates enabled the ‗Policy A vs. Policy B‘ comparison.
Internal MTP ‗SEUP Benefits‘ document used as a guide to Defra‘s individual contribution to benefits delivered by EU Regulations.
Energy and CO2 savings for EuP
8.2 Phase 1: Consortium interviews
Interviews were undertaken with Defra and MTP consortia staff, including individuals who
were previously closely involved in the MTP programme but are no longer. These
interviews were designed to gather information on:
Programme delivery structures;
MTP operational success and challenges;
Potential areas of impact for exploration in phase 2.
In total, fourteen qualitative interviews were undertaken; these were split as follows:
Defra – current – 6
Defra – previous – 2
AEA – current – 4
AEA – previous – 2
Each interview was tailored to explore issues pertinent to the respondent‘s interaction with
MTP. The full discussion guides used for the consortia interviews are shown in the
appendices to this report.
The key risk of the approach was that MTP and Defra may (though this is unlikely) have
been unaware of influencing a particular policy outcome e.g. where the evidence base has
been utilised without direct communication with MTP. Were this the case, such outcomes –
and their benefits - would not have been explored in phase 2 and would not be accounted
for in value for money analysis.
8.3 Consortia workshop
Following completion of the desk research and consortia interviews, a review workshop
was facilitated by Databuild at Defra offices. This was to enable discussion of:
Evaluation of the Market Transformation Programme
Final report
April 2012
56
Summary findings and areas for consideration in the specification of the next
contract, which Defra were in the process of developing.
Areas of MTP outcomes and impacts that should be explored in phase 2.
8.4 Phase 2: Stakeholder interviews
Phase 2 of the evaluation aimed to provide verification of – and further insight into – the
areas of impact identified in phase 1.
8.4.1 Impacts explored
The desk review, consortium partner interviews and Defra / AEA workshop identified a
number of areas where the MTP programme was likely to have had an impact:
EU Eco-Design of Energy Related Products Directive:
- Ensuring consideration of particular products within the directive programme
- Influencing an increase in the agreed minimum energy performance standard
(MEPS) required for a particular product group
- Influencing the quicker introduction of minimum performance standards
- Advising on testing and compliance to enforce the new standards
Influencing the design of labelling of energy related products as part of the
mandatory EU labelling scheme
Influencing voluntary agreements with industry; specifically the phase out of GLS
bulbs and introduction of lower standby voltages for appliances.
Green procurement: influencing the level of energy performance standards
required for products purchased through government contracts.
International programmes; influencing the requirements of Energy Star, the public
procurement standard for all US and EU countries.
8.4.2 Stakeholder interviews conducted
Based upon these areas where Defra and MTP believed the programme had influenced
outcomes, Databuild worked with Defra and MTP to build a database of individuals who
would be appropriately placed to provide verification that this was the case. The chart
below shows which stakeholder groups were contacted in phase 2 and what areas they
were expected to provide verification on:
Stakeholder groups Interviews conducted Impact areas covered
Defra product / policy leads 4
MEPS and labelling,
voluntary agreements,
compliance, green
procurement.
MTP consortia leads 3 MEPS and labelling,
voluntary agreements.
European Commission policy
leads 7 MEPS and labelling.
Individuals representing
other EU member states at
directive negotiations
5 MEPS and labelling.
Evaluation of the Market Transformation Programme
Final report
April 2012
57
Industry representatives 9 MEPS and labelling,
voluntary agreements.
Energy Star programme
representatives 2
International programmes,
compliance
NMO 2 Compliance
8.4.3 Challenges encountered in securing interviews in phase 2
1. Sample coverage of product groups and policies – some respondents were able to
comment on a number of product areas or on more than one policy instrument;
but for some policy instrument / product group areas, appropriate respondents at
the European level could not be identified. The product groups for which responses
were primarily lacking were EPSUs, electric motors and circulators.
2. Ability / willingness to participate – six potential interviewees identified in
consultation with Defra were either unwilling to participate or could not be
contacted. This was a product of the high levels of movement of staff identified
through this research. The consequence of this was that responses could not be
obtained from some of those best placed to provide verification.
8.5 Impact methodology
8.5.1 What impact data was required and why?
In order to ascertain the outcomes of MTP activities (in terms of £s or CO2), the impacts of
the policies and programmes they intend to influence also had to be determined.
The decision was taken to focus upon UK impact because:
a. This is the basis upon which most of the impact assessments and forecasting
produced by MTP have been calculated. Although indicative extrapolations of the
UK impacts to European level have been conducted, these were not intended to be
robust. Instead they were taken to be indicative of the potential level of influence
MTP could have at the European level.
b. This provides the appropriate basis to assess the value of spending UK public funds
i.e. impacts in Europe and / or beyond do not (at least directly) produce
environmental or financial benefits for the UK.
The sections below set out the approach taken in the evaluation to quantify the impacts of
the different policy instruments and programmes on the UK.
8.5.2 Sources of Impact Data
Latest carbon budget impact figures (derived from MTP IAs and the DECC Carbon Plan)
were used as the most robust monetary and carbon impact figures available for MTP‘s
work.
Evaluation of the Market Transformation Programme
Final report
April 2012
58
The document ‗Saving Energy Through Better Products and Appliances‘ gives some detail
on the way in which MTP‘s Impact Assessments use modelling to assess the likely impact
of new policies: “The modelling approach used is mainly based upon a stock turnover
model, where the number of products in use, expected growth or decline of that number in
future years and product lifetime form the basis for calculating how the installed stock in
any one year is composed of products sold in current and previous years. This data provide
the basis for analysis and scenario building to quantify the current and future
environmental performance of products, and the likely effects of various policies and
changes in the market. This analysis allows the Government to assess the expected impact
of different policy options over time and decide on adequate performance standards for
energy-using products.”
In transferring the Impact Assessment (IA) impacts into the carbon budget figures, some
alterations to the calculation mechanisms were made in order to ensure that they were up-
to-date and consistent.
1. Monetised benefits such as the ‗total value of absolute reduction in required
delivery of renewable energy‘ were removed. This was included in the analysis
used for the Impact Assessments as a requirement of the DECC Guidance on
Appraising Energy and GHG Policies, but was taken out of scope for the purposes
of the departmental figures. The change ensures that the benefits the UK enjoys as
a result of needing to generate less renewable energy are not over-stated in
different Impact Assessments.
2. Some assumptions used in the impact assessments were updated to reflect the
time that the departmental figures were produced e.g. the base price year. This
change ensures that up-to-date assumptions about product prices and costs were
used when generating the impact figures.
The list below shows which Impact Assessments were used to generate impact figures for
each product group in the departmental figures:
Implementing
Measure
MTP Impact Assessment(s) included in this product
group
Standby Impact Assessment of EuP Implementing Measures for
Standby and Off-Mode Losses Version 1.0
ESPUs Impact Assessment of EuP Implementing Measure for External
Power Supplies Version 1.0
Electric Motors Impact Assessment of EuP Implementing Measure for Motors
Version 2.0
Circulators Impact Assessment of EuP Implementing Measure for
Circulators Version 7.2
Cold appliances and
labelling
Impact Assessment of EuP Implementing Measures for cold
domestic appliances version 1.8
TVs and labelling
Impact Assessment of Implementing Measures for eco-design
requirements and energy labelling of Televisions (TVs) Version
4.7
Wet appliances and
labelling
Impact Assessment of EuP Implementing Measures for
washing machines Version 1.3
Impact Assessment of EuP Implementing Measures for
Evaluation of the Market Transformation Programme
Final report
April 2012
59
dishwashers Version 5.0
Domestic Lighting Impact Assessment of EuP Implementing Measures of
Domestic Lighting Version 2.4
Tertiary Lighting Impact Assessment of EuP Implementing Measures for Tertiary
Lighting Version 1.0
Simple STBs Impact Assessment of EuP Implementing Measures for Simple
Set Top Boxes Version 1.0
8.5.3 Overlap with other policies
In some product areas, more than one policy or initiative was identified as being
potentially supported by MTP. In these cases, impact figures should not be calculated as
the likely impact of policy A plus the likely impact of policy B, as the two policies are likely
to have influenced some of the same impacts (in other words, some impact achieved by
one of the policies is the same impact that would have been achieved by the other). The
impact assessments of EuP policies take account of this by including other policies which
may affect the product group in the reference scenario. This ensures that the impact
figures calculated relate only to the EuP policy in question.
Based upon review of the impact assessment documents, the table below shows, for each
product group considered in the evaluation, which policies have been accounted for when
calculating the overall impact:
Policies considered in IA analysis
EuP Directive
Energy Labelling
ENERGY STAR CERT
Building Regs ECA CRC Others
Pro
du
ct
Are
a
Standby
ESPUs
Electric Motors
20
Circulators
Cold appliances and labelling
TVs and labelling
Wet appliances and labelling
Domestic Lighting
21
Tertiary Lighting
Simple STBs
Key:
- Included in IA analysis
- Considered for inclusion in analysis but effect deemed negligible
20 Government Quick Wins Scheme 21 UK Voluntary Agreement
Evaluation of the Market Transformation Programme
Final report
April 2012
60
8.5.4 Impact data quality and limitations
The limitations of the IA data are described in detail in the main body of the report (section
4).
Of these, one of the most important was that the IA figures assess the impact of the MEPS
/ labelling policy compared to a reference scenario of there being no policy. In reality, this
reference scenario would be very unlikely and reporting MTP influence against these
impacts risks misinterpretation of the actual extent of impact MTP could reasonably have
influence.
For example, for standby, MTP data was predominantly used in order to secure the
introduction of more stringent standards 1-2 years earlier than would otherwise have been
the case. A MEPS policy would have been introduced regardless and more stringent
standards would almost certainly have been in place by the third year from the
introduction of the MEPS. It is therefore misleading to suggest that MTP influenced
anything more than the marginal impacts derived from two years of having more stringent
requirements, as the vast majority of the lifetime savings from the policy would have been
delivered anyway.
These limitations have necessitated – for completeness - exploring MTP influence upon two
different types of impact:
1. The impact of the policy compared to not having a policy at all (‗policy vs.
reference).
2. The impact of the MTP-influenced policy compared to the impact of the likely policy
outcome if MTP had not been involved (‗policy A vs. policy B‘)
It is important to understand the risks in using the larger ‗policy vs. reference‘ impacts
rather than the ‗policy outcome A vs. policy outcome B‘ impacts.
Evaluation of the Market Transformation Programme
Final report
April 2012
61
8.6 Attribution methodology
The attribution methodology in this section explains how MTP‘s influence upon particular
policy instruments and product groups were identified and verified. It also explains the
rationale behind the way in which attribution has been reported and integrated with impact
data in this report.
8.6.1 Stage 1: identifying potential areas of MTP influence
The activities of MTP are numerous and wide-ranging. It was therefore agreed that the
most efficient use of evaluation resource would be to use the desk research and consortia
interviews conducted in phase 1 to filter a set of outcomes where Defra and MTP were
confident that impact had been achieved.
Desk research and the phase 1 consortia interviews identified the main points at which
Defra, and the MTP consortium, have influenced policy making22. For each priority area
Defra and MTP identified a number of specific and general claims on what had been
achieved and how it had been influenced.
The desk review, consortium partner interviews and Defra / AEA workshop identified a
number of areas where the MTP programme was likely to have had an impact:
EU Eco-Design of Energy Related Products Directive:
- Ensuring consideration of particular products within the directive programme
- Influencing an increase in the agreed minimum energy performance standard
(MEPS) required for a particular product group
- Influencing the quicker introduction of minimum performance standards
- Advising on testing and compliance to enforce the new standards
Influencing the re-design and re-valorisation of labelling of energy related products
as part of the mandatory EU labelling scheme
Influencing voluntary agreements with industry; specifically the phase out of GLS
bulbs and introduction of lower standby voltages for appliances.
Green procurement: influencing the level of energy performance standard required
for products purchased through government contracts.
International programmes; influencing the requirements of Energy Star, the public
procurement standard for all US and EU countries.
8.6.2 Stage 2: verifying the claimed influences
Following identification of the outcomes that would be focused on, phase 2 stakeholder
interviews were undertaken with the aim of verifying, or otherwise, the general and
detailed list of specific MTP impacts.
22 Note this does not include some areas such as future studies and international, which
can take the form of different studies throughout the year.
Evaluation of the Market Transformation Programme
Final report
April 2012
62
8.6.3 Challenges in verifying MTP influence
There were a number of challenges faced when attempting to verify the hypothesised
influence of MTP; these included:
Movement of key personnel within the Commission; this meant that the desk
officers with first hand or most pertinent experience of MTP input to policies over
the past 3-4 years were not available (or in two cases not willing) to participate
and provide feedback. The other effect of this was that some of the recommended
respondents did not feel that they had been in their posts long enough to be able
to comment fully on the influence of Defra or MTP on the policies for which we
have estimated an impact.
No direct contact with MTP; although selected on the basis that they had some
experience of working with MTP and / or experience of the policies being
researched, some respondents had not had direct contact with MTP or were
unaware that MTP was responsible for a particular piece of analysis or technical
input to Defra. Similarly, some UK trade bodies input to the Commission legislative
process through a European level trade federation. Therefore some industry
respondents did not have sufficiently direct involvement in the legislative process
to be able to directly assess the effectiveness of Defra or MTP in influencing the
outcome in those negotiations.
Complexity of the decision making process; the legislative process at the European
level is complex with a large number of participants and multiple stages. The
process is one of building consensus and negotiation, involving multiple and often
differing opinions, taking into account both political and technical arguments.
Many respondents therefore found it difficult - or were unwilling - to attribute
outcomes to one particular stakeholder or one particular contribution. In a similar
vein, several respondents at the Commission felt they were unable to attribute
influence to an individual Member State or comment on specific events in the
legislative process on the basis of confidentiality linked to the commitology
procedure. There was the sense that some member states respondents were not
willing to go too far in crediting another individual Member State with influencing
an outcome.
8.6.4 Stage 3: Arriving at an assessment of attribution
On the basis of the challenges outlined above, the following were assessed in order to
provide categorisation of attribution for each policy outcome:
1. The extent to which Defra/MTP had influenced the outcome of the policy making
process, both generally and for specific claimed impacts. The level of influence was
rated as follows:
Strong influence - MTP (alone or through Defra) were critical in influencing
the outcome;
Some influence – MTP (alone or through Defra) made some contribution to
the outcome ;
Weak influence – MTP (alone or through Defra) made a slight difference to
the outcome;
No influence - MTP (alone or through Defra) made no difference to the
outcome;
Evaluation of the Market Transformation Programme
Final report
April 2012
63
2. The level of confidence in that assessment of influence; established through a
‗traffic light‘ system:
Green – there is good verification of the Defra / MTP claim, with at least
one respondent is able to authoritatively verify the claim and there is a
good match in the detail for specific claims.
Amber – there is some verification of the claim; at least one respondent is
able to verify the claim but there is some uncertainty in the recall of the
detail of specific claims.
Red – there is no verification of the claim made by MTP / Defra or there is
disagreement between respondents or such a large degree of uncertainty
that verification of the actual influence on the outcome is difficult to
discern.
In all cases of assessing the level and reliability of the attribution categorisation, the
following were taken into account across applicable responses:
The varying degrees of respondent involvement in the negotiation process;
The varying degrees of political interests / vested interests that might affect
responses.
8.6.5 Stage 4: Integrating impact data with attribution data
Regarding MEPS (as the focus of most MTP work), prior to the evaluation, consideration
had already been given to MTP‘s influence on the policy outcomes relating to the product
groups being considered under the directive. Defra had undertaken a brief review which
resulted in a rough estimate of the proportion of the SEUP benefits impacts that would not
have been achieved without MTP.
It was originally anticipated that these % attributed impact ranges could be used as a
starting point to test with phase 2 respondents to draw out estimations of % attribution.
However, this was not done as Defra expressed reservations about the potential for
stakeholders to misconstrue the figures as robust and evidence-based claims.
There were therefore two ways in which the attribution and impact data could have been
integrated:
1. Work to estimate other precise % influences to enable calculation of
impacts delivered by MTP activities
This would have entailed ascertaining a description of the influence MTP activities
had upon a particular policy outcome, and then ascribing a proportion of the
impact of that policy outcome to MTP i.e. assuming that precise amount of impact
would not have been achieved without MTP input.
The main flaw with this approach as regard the MTP evaluation is that the ways in
which MTP may have influenced a particular outcome could be both nuanced and
numerous. Therefore arbitrarily selecting a proportion of impact to claim for MTP
ignores the issues around what the true counterfactual would have been and issues
around how the SEUP benefits impact data might have been disaggregated across
different levels of stringency / time delay etc.
Evaluation of the Market Transformation Programme
Final report
April 2012
64
2. Stating the impacts which MTP has influenced to some degree
This approach simply takes the ‗policy vs. reference‘ and ‗policy A vs. policy B‘
impacts - which MTP has attempted to influence - and states the extent to which
MTP has had an influence upon that impact. The limitation of this approach is that
a precise impact is not ascribed to MTP influence.
Evaluation of the Market Transformation Programme
Final report
April 2012
65
Evaluation of the Market Transformation Programme
Final report
April 2012
66
Results