bristol water - 2015/16 annual performance report … water - 2015/16 annual performance report and...

Bristol Water - 2015/16 Annual Performance Report and GSS Payments Assurance Report

27 June 2016

Bristol Water - 2015/16 Annual Performance Report and GSS Payments Bristol Water

Atkins 27 June 2016 | Ref: 5145235/DG/KH/072 | Version 2.0

Notice

This document and its contents have been prepared and are intended solely for Bristol Water’s information and use in relation to the technical assurance services provided by Atkins on the Bristol Water - 2015/16 Annual Performance Report and GSS Payments.

Atkins Ltd assumes no responsibility to any other party in respect of or arising out of or in connection with this document and/or its contents.

This document has 20 pages including the cover.

Document history

Document ref: 5145235/DG/KH/072

Revision Purpose description Originated Checked Reviewed Authorised Date

Rev 1.0 Final Draft (internal review) JPA/KH SJI GJ JPA 20/06/16

Rev 2.0 Issue to Bristol Water Board JPA/KH SJI GJ JPA 27/06/16


Atkins 27 June 2016 | Ref: 5145235/DG/KH/072 | Version 2.0

Table of contents

Chapter Pages

Assurance Statement 4

Executive Summary 5

1. Introduction 7

1.1. Background 7 1.2. Scope 8 1.3. Structure of Assurance Report 9

2. Approach 10

3. Summary of Findings 12

3.1. General 12 3.2. 2015/16 Annual Performance Report 13 3.3. Supporting Data for 2015/16 18 3.4. GSS Payments for 2015/16 19

Tables Table 1-1 Areas for Atkins Assurance – Annual Performance Report ...................................................... 8 Table 2-1 Description of Deliverables ...................................................................................................... 10 Table 2-2 Descriptions for RAG categories ............................................................................................. 11 Table 3-1 Methodology and Data Categories for each PC ...................................................................... 13 Table 3-2 RAG Assessment: Supporting Data ........................................................................................ 18 Table 3-3 RAG Assessment: GSS Payments .......................................................................................... 19

Figures Figure 1-1 Bristol Water’s assurance approach for 2016/17 submissions .................................................. 7 Figure 2-1 Audit Process ........................................................................................................................... 10

Atkins 27 June 2016 | Ref: 5145235/DG/KH/072 | Version 2.0 4

Assurance Statement

Based upon our assessment of Bristol Water’s 2015/16 Annual Performance Report and GSS Payments and the supporting information we saw over a structured programme of audit meetings from November 2015 to June 2016, we conclude that for the reporting of areas we covered, other than where indicated otherwise in our report:

• at a component level the various teams compiling the documents and information had an understanding of and were meeting their reporting obligations;

• the Company has sufficient processes and internal systems of control to meet its reporting obligations;

• the Company’s explanations of where and why it cannot fulfil its reporting obligations are soundly based;

• the Company has sufficient processes and internal systems in place to identify, manage and review its reporting risks; and

• the Company’s explanations of how it will manage and/or mitigate material or potentially material reporting risks are soundly based.

Our overall impression from the audit meetings was that considerable effort had gone into compiling the Annual Performance Report 2015/16 and that the associated processes are evolving and improving. The sources of raw data are broadly the same as used in previous years. Formally documented checks and processes are improving and, as is normal for performance reporting across the water industry, the complexity, occasional subjectivity and scale of the data collation process precludes absolute accuracy across all areas.

While we observed a number of issues for which we provide comment within the main report, we believe these do not impact materially upon the potential to sign-off the Company submission. Each is an area we believe should be given further consideration as part on continuing improvement to performance reporting by Bristol Water.

Jonathan P Archer Regulation Director Reporter providing Technical Assurance Services to Bristol Water



Executive Summary

Introduction

This report summarises the external technical assurance (Reporter) services Atkins has provided in relation to Bristol Water’s 2015/16 Annual Performance Report and GSS Payments. This is our first year of providing these services to Bristol Water and the first year of AMP6 in which Bristol Water has reported against the measures defined in the 2014 Final Determination by Ofwat (as subsequently amended as a result of the deliberations of the CMA). Our approach has been shaped by the expectations of the assurance to be provided for a “prescribed” water company. Throughout, we have received the cooperation of the Company and have had the freedom to express our opinions. We have had access to, and have fed back to, the Board, Audit Risk and Assurance Committee, Senior Leadership Team and Bristol Water Challenge Panel.

The scope of our assurance was the technical audit of the performance commitments (PCs) reportable in the 2015/16 Annual Performance Report, as defined in Bristol Water’s Assurance Plan 2016/17 including key supporting data. We also reviewed the Guaranteed Standard Scheme payments for 2015/16. Our assurance did not include forecasts of future performance, the financial consequences of performance, the 2014/15 performance (actual) or data previously audited by Mott Macdonald (previous Reporter).

Approach

We carried out a series of structured audits, with separate process and data audits in line with Ofgem’s Data Assurance Guidance. Our focus on particular areas was risk based as highlighted in Bristol Waters own analysis and supplemented by our experience in identifying and quantifying the elements of the journey from raw to published data that introduce material errors.

After detailed planning of an audit schedule to ensure the appropriate people (Company and auditor) are present, we formally notify all parties of the expectation of the audits. We provide immediate verbal feedback and document our audit findings in both a rapid feedback e-mail and a detailed audit summary. These provide the Company with the opportunity to correct errors of fact and respond with explanations or further information to our observations. The essence of the summaries is captured in an Action Log which is used to manage the progress on matters arising. The supporting documentation is available for inspection.

Summary of Findings

We assigned each PC an overall rating of Red, Amber or Green to reflect their priority with separate ratings for the methodology and the data. Descriptions for each category are given in the table below.

Category Description

RED High Priority: Failure to comply with reporting requirements, major failure of process or data errors that may lead to misreporting.

AMBER Medium Priority: Shortfalls in methodology and/or methodology documentation. Methodology under development. Incomplete data set or minor errors identified that do not alter the performance reported relative to targets and threshold values.

GREEN Low Priority: Minor revisions to methodology and/or methodology documentation needed. Issue(s) not judged to be material or no issues.

In response to our challenges, issues were resolved or downgraded to a level that does not preclude the sign-off of the performance data. However, there remains need for improvement in the comprehensiveness of the methodology documentation to reduce the reliance upon the detailed knowledge of individuals responsible for preparing information.

Governance and Board ownership of the published information is clear and once data have reached the regulation team, the governance process is well established. There is also need to make clear at the incremental stages between raw to consolidated data the sign-off and hand-over between data providers.

The categories assigned to each PC are detailed in the table below. Where we have categorised a methodology as amber, this is as a result of:

Shortfalls in methodology and/or methodology documentation;

Because the methodology is under development; and/or



The data set is incomplete or there are minor errors that do not alter the performance reported relative to targets and threshold values.

Outcome Performance Commitment Methodology Data

A: Reliable Supply A1: Unplanned customer minutes lost AMBER GREEN

A2: Asset Reliability – Infrastructure (Total Bursts and DG2 Low Pressure)

AMBER GREEN

A3: Asset Reliability – Non-Infrastructure (Turbidity and Unplanned Maintenance)

AMBER GREEN

B: Resilient Supply B1: Population in centres >25,000 at risk from asset failure

AMBER GREEN

C: Sufficient Supply C1: Security of supply index (SOSI) AMBER GREEN

C2: Hosepipe ban frequency AMBER GREEN

D: Mean Zonal Compliance (MZC)

D1: Mean Zonal Compliance GREEN GREEN

E: Water is good to drink E1: Negative Water Quality Contacts GREEN GREEN

F: Efficient use of resources by Company

F1: Leakage AMBER GREEN

G: Efficient use of resources by customers

G1: Meter penetration (%) GREEN GREEN

G2: Per capita consumption AMBER GREEN

H: Sustainable Environmental Impact

H1: Total carbon emissions GREEN GREEN

H2: Raw water quality of sources AMBER GREEN

H3: Biodiversity index AMBER GREEN

H4: Waste disposal compliance GREEN GREEN

I: Affordable bills I1: Percentage of customers in water poverty GREEN GREEN

J: Satisfied customers J1: Service Incentive Mechanism (SIM) GREEN GREEN

J2: General satisfaction from surveys GREEN GREEN

J3: Value for money GREEN GREEN

K: Easy to contact K1: Ease of contact from surveys GREEN GREEN

L: Bills are accurate and easy to understand

L1: Negative billing contacts GREEN GREEN

Supporting Information

In common with much of the reported data, the supporting data (properties, volumes) are the result of a well-established and previously audited processes. The documentation needs to be improved. However, the staff responsible have the appropriate skills and experience and were able to demonstrate checks and controls in place.

GSS

These payments result from activities at both BWBSL and Bristol Water itself. We found that the Company identifies GSS events systematically and effectively in line with its legal obligations and makes payments to customers where due. At BWBSL, the methodology and processes for managing and reporting GSS are documented and appear to be fit for purpose. There are also robust checks and controls built into management of GSS and there were comprehensive audit trails to demonstrate compliance. We were satisfied that the Company is meeting its legal obligations with respect to GSS and making payments to customers where due.

Conclusions

Considerable progress has been made by Bristol Water in formalising the methodology for the reporting of its AMP6 PCs. Further improvements are needed to the methodology documentation, but this is a reflection of the coverage by comprehensive documentation rather than of the application of the processes themselves. Many of the PCs reported against are the same as or similar to previously reported information and the reporting teams have well established reporting processes, checks and controls. This, combined with what we saw at the more detailed data audits gave us the comfort that the reported figures represent a reasonable and accurate representation of the performance of Bristol Water during the report year 2015/16.



1. Introduction

1.1. Background Atkins Limited has been appointed by Bristol Water to provide external assurance on the regulatory submissions presented by Bristol Water (the Company) to the Water Services Regulation Authority (commonly known as Ofwat) under the conditions set out in its Licence with the Secretary of State.

Following the conclusion of the PR14 Business Plan process, Ofwat announced in December 2015 that it was placing Bristol Water in the ‘Prescribed’ assurance category, due to concerns over the quality of the evidence presented in support of its PR14 Business Plan. Bristol Water is required to incorporate independent external assurance as part of its assurance plans. Bristol Water’s Assurance Plan 2016/17

1 sets

out the Company’s proposed approach to assurance of the information that they will publish during 2016/17. This is shown in Figure 1-1 below.

Figure 1-1 Bristol Water’s assurance approach for 2016/17 submissions

The Company’s Assurance Plan states that it is intended to meet the requirements of Ofwat’s Company Monitoring Framework (CMF)

2. With respect to data assurance, the CMF states that data assurance plans

should follow a risk based approach. The Ofgem Data Assurance Guidance (DAG) is highlighted as standard that could be used and the Assurance Plan reflects this approach.

1 http://www.bristolwater.co.uk/wp/wp-content/uploads/2016/03/Final-Assurance-Plan-March-2016.pdf

2 http://www.ofwat.gov.uk/publication/company-monitoring-framework-final-position/

http://www.bristolwater.co.uk/wp/wp-content/uploads/2016/03/Final-Assurance-Plan-March-2016.pdf



This report presents Atkins’ external technical assurance for Bristol Water’s 2015/16 Annual Performance Report and GSS Payments. Our approach to providing technical assurance services and reporting herein has been tailored to and focused upon the reporting against the PCs by Bristol Water in its 2015/16 Annual Performance Report in accordance with the Assurance Plan. Bristol Water has 21 PCs, of which 10 have financial rewards and/or penalties attached to them as Outcome Delivery Incentives (ODIs). We considered the methodologies and processes by which data are produced; the material accuracy of the data produced and the conclusions drawn by Bristol Water.

Our assurance has been tailored with the aim of ensuring that customers and stakeholders can trust the data and information that Bristol Water provides on its performance and the broader requirements of the Ofwat Monitoring Framework. We anticipate that our approach will evolve to reflect and respond to the year on year improvements to the Company processes and the expectations of stakeholders and Ofwat’s oversight of the assurance process. We are taking an evidential approach to confirm the application of, rather than just the adequacy and appropriateness of procedures.

We note that under Ofwat’s Company Monitoring Framework, Bristol Water must share full reports with Ofwat if they have not been published in full. There is no duty of care to Ofwat from the assurer and Ofwat would not publish or share material provided that the Company had not published without agreement. The supporting documentation for this report (audit reports and issues log) is available if required.

1.2. Scope

This report relates to the external assurance of the 2015/16 Annual Performance Report and Guaranteed Standard Scheme payments for 2015/16. The Performance Commitments (PCs) and associated Outcome Delivery Incentives (ODIs) against which the Company reports are as defined in Ofwat’s 2014 Final Determination and the subsequent 2015 Price Determination from the Competition and Markets Authority.

The changed reporting introduced for AMP6 covers measures ranging from those which have long been in place (like PC F1: Leakage) to those that are new and specific to Bristol Water (like PC H3: Biodiversity Index). The areas of the Annual Performance Report in scope for our assurance are detailed in Table 1-1 below.

Table 1-1 Areas for Atkins Assurance – Annual Performance Report

Performance Measure Measure

Table 3A - Outcome performance table

A1: Unplanned customer minutes lost

A2: Asset reliability - infrastructure

A3: Asset reliability - non-infrastructure

B1: Population in centres >25,000 at risk from asset failure

C1: Security of supply index (SOSI)

C2: Hosepipe ban frequency

D1: Mean zonal compliance (MZC)

E1: Negative water quality contacts

F1: Leakage

G1: Meter penetration

G2: Per capita consumption (PCC)

H1: Total carbon emissions

H2: Raw water quality of sources

H3: Biodiversity index

H4: Waste disposal compliance

I1: Percentage of customers in water poverty



Performance Measure Measure

J1: Service incentive mechanism (SIM)

J2: General satisfaction from surveys

J3: Value for money

K1: Ease of contact from surveys

L1: Negative billing contacts

Table 4A - Non-financial information

Number of households billed

Wholesale volumes (Ml/d)

The supporting data (properties, population and volumes) used in the performance reporting are the result of a well-established and previously audited processes. Our review considered census documentation, occupancy information and reports from the Company billing system.

Our review of Guaranteed Standard Scheme payments for 2015/16 covered payments made in the following areas:

Making and keeping appointments (GSS Regulation 6) Account queries and requests about changes to payment arrangements (GSS Regulation 7)

Complaints (GSS Regulation 7)

Notice of interruption to supply (GSS Regulation 8)

Supply not restored (GSS Regulation 9)

Low pressure (GSS Regulation 10)

The technical aspects of the interruptions, availability and pressure in supply were also considered as part of the audit of the performance measures A1, A2, A3, C1 and C2.

1.2.1. Exclusions Our assurance services for this round of audits did not include forecasts of future performance, the financial consequences of performance, the 2014/15 performance (actual) or data previously audited by Mott Macdonald (previous Reporter). Our review of the Performance Commitments reportable in Table 3A covered the 2015-16 performance level only. We did not review the following areas reportable in Table 3A:

2014-15 performance Level – actual

2015-16 Committed Performance Level (CPL) met?

2015-16 reward or penalty (in period ODIs) [Not applicable as Bristol Water has no in period ODIs]

2015-16 reward or penalty (in period ODIs) (£m absolute value) [Not applicable as Bristol Water has no in period ODIs]

Notional reward or penalty accrued at 31 March 2016

Notional reward or penalty accrued at 31 March 2016 (£ absolute value)

Total AMP6 reward or penalty 31 March 2020 forecast

Total AMP6 reward or penalty 31 March 2020 forecast (£ absolute value)

1.3. Structure of Assurance Report This report has a hierarchal structure as follows:

Assurance Statement

Executive Summary

Section 1 – Introduction

Section 2 – Approach

Section 3 – Summary of Findings



2. Approach

The overall approach to assurance is based around two stage audits; Process and Data. The purpose of each audit type is as follows:

Process Audits: To assess whether the Company’s methodology aligns with appropriate guidance, reporting requirements, licence conditions or industry practice and whether appropriate checks, controls and explanatory documents exist.

Data Audits: To assess whether processes/procedures are applied as indicated including data trailing to source documents to ensure alignment/ consistency with the reported number and sampling to confirm rigour of process application, checks and controls and appropriateness of confidence grades assigned to reported information (where applicable)

This approach is consistent with Ofgem’s Data Assurance Guidance (DAG) which identifies external process audit and external data audit as potential ‘assurance responses’, described as follows:

External Process Audit: Not responsible for ensuring that returns are complete and accurate but to provide an independent challenge to the process to produce the submission. Review of the adequacy and effectiveness of the internal control systems to ensure returns are timely, complete and accurate. Formal report produced. Control gaps/areas for improvement identified and actions logged. External Data Audit Responsible for providing evidence of verification of Data; Done through a sampling approach; Intends to determine the level of confidence that can be placed on the entire return through testing a sample of the Data; Formal report produced. The process flow followed for each audit is summarized as follows: Figure 2-1 Audit Process

The deliverables for each stage of the process are summarised below.

Table 2-1 Description of Deliverables

Deliverable Description

Notification of Audit Form (NAF) Issued in advance of audit. Details audit arrangements, scope and agenda

Email summary Initial feedback including detail of any material issues.

Summary of Audit Form (SAF) Issued following the audit. Details findings and any actions for inclusion in the action log

Action log Spreadsheet to track and report on response to issues identified at audit. Includes Reference; Date Raised; Raised by; Line; Observation; Recommendation; Priority; Agreed (Y/N); Company response; Owner; By when; Status

Our assessment of the Company’s reporting against each performance measure has been assigned an overall rating of Red, Amber or Green to reflect their priority. Separate ratings have been given to the methodology and to the data.

Table 2-2 sets out the definitions for the different categories.

NAF Email

Summary Draft SAF Response

Action Log

Final SAF



Table 2-2 Descriptions for RAG categories

Category Description

RED High Priority: Failure to comply with reporting requirements, major failure of process or data errors that may lead to misreporting.

AMBER Medium Priority: Shortfalls in methodology and/or methodology documentation. Methodology under development. Incomplete data set or minor errors identified that do not alter the performance reported relative to targets and threshold values.

GREEN Low Priority: Minor revisions to methodology and/or methodology documentation needed. Issue(s) not judged to be material or no issues.



3. Summary of Findings

Our Stage 1 process audits started in November 2015 and we have noted significant progress in the development of a portfolio of methodology documentation for the reporting of outturn performance against the Performance Commitments made in the PR14 Business Plan and confirmed in the Final Determination. Further development work is needed on the process documentation and the formalising of an evidence base for the Company’s reporting.

At our Stage 1 audits we challenged the content, clarity and robustness of the process documentation as it was developing and note that in many cases, there remains a need to bring the documentation up to a level that reflects the processes that are being applied as business-as-usual by experienced and appropriately skilled staff. There is need for the formalisation and the maintenance of an evidence base for the checks and controls we have had explained to us and seen demonstrated. For measures that have historically been reported on, we formed the opinion that the Company staff responsible for reporting have the necessary processes in place, but the documentation do not yet reflect this. The Company is aware of this and the work on process documentation is ongoing and, we understand, will be fully in place for detailed assessment as part of the mid-year reporting assurance in October 2016.

Stage 2 data audits dealt with year-end information and focused on areas we believed at greatest risk of leading to errors which would materially affect the accuracy of the reported figures or ODI impact. Overall, errors we identified at audit were rectified at the time and we were able to see demonstration that data was being reported and checked consistent with industry practice and in a way that aligned with the approaches used in developing the target and threshold levels stipulated within the Final Determination, adjusted where appropriate as a result of the CMA referral.

3.1. General

We make the following general comments on the submission and the reporting as a whole:

Controls – The Company appears to have well established high level governance processes in place, with staff identified for producing the reported data and a hierarchy of ownership and sign-off up to the final sign-off on behalf of the Board. However, insofar as most of the reported figures are derived from widely spread raw data, which are developed across the reporting year, there is need to better evidence and document the existence of checks and controls from end to end of the reporting process, from raw data to reported information. Sign-off is late in process and we gained the impression that the Reporter team has historically been seen as the “final check” rather than providing technical assurance of the end-to-end process and reported information.

Data Hand-over – In the flow of data from its raw form to reported information, there are often stages at which responsibility for the data transfers to different parts of the organisation. We believe at these change points, inter-stage sign-off should be considered. We are aware that there is an ongoing review of the Company’s organisation and activities and that process diagrams are being drawn up which would assist in identifying the critical loci and data ownership groups.

Methodologies – The documentation of the methodologies Bristol Water applies to reporting against its PCs is still evolving. Considerable amounts of process documentation already exist, insofar as some of the measures or components thereof have been reported against since privatisation of the industry. The Company has made considerable progress since our initial review of documentation in November 2015. We noted at that time that the documentation was being assembled on the basis of what had been reported upon previously or had fed into the Business Plan submission. The current process documentation is now specifically aligned with the reporting against the PCs made for AMP6. We believe that there is further to go, but that the direction of travel is appropriate. The need for further improvement is indicated by the allocation of an Amber status in the detail below.

Compliance with Reporting Guidance – We have noted in certain measures that reporting does not strictly align with the old DG measures or other July/June Reporting or Annual KPI reporting requirements. However, we have noted that the reporting being done now is consistent with how it was done in the pasts and thereby with the data that supported the setting of Performance Commitment targets and dead-band /



penalty / reward thresholds. The discrepancies are generally minor and we believe that the maintaining an approach which is consistent with the bases of the PCs is appropriate.

Robustness of reported data – We have concluded that there are no material issues with the data reported. The adequacy of the reported data is indicated by the allocation of a Green status in the detail below.

3.2. 2015/16 Annual Performance Report

The table below summarises the categories assigned to each PC reported in the Annual Performance Report with further detail below.

Table 3-1 Methodology and Data Categories for each PC

Outcome Measure Methodology Data

A: Reliable Supply A1: Unplanned customer minutes lost AMBER GREEN

A2: Asset Reliability – Infrastructure

(Total Bursts and DG2 Low Pressure) AMBER GREEN

A3: Asset Reliability – Non-Infrastructure

(Turbidity and Unplanned Maintenance) AMBER GREEN

B: Resilient Supply B1: Population in centres >25,000 at risk from asset failure

AMBER GREEN

C: Sufficient Supply C1: Security of supply index (SOSI) AMBER GREEN

C2: Hosepipe ban frequency AMBER GREEN

D: Mean Zonal Compliance (MZC)

D1: Mean Zonal Compliance GREEN GREEN

E: Water is good to drink E1: Negative Water Quality Contacts GREEN GREEN

F: Efficient use of resources by Company

F1: Leakage AMBER GREEN

G: Efficient use of resources by customers

G1: Meter penetration (%) GREEN GREEN

G2: Per capita consumption AMBER GREEN

H: Sustainable Environmental Impact

H1: Total carbon emissions GREEN GREEN

H2: Raw water quality of sources AMBER GREEN

H3: Biodiversity index AMBER GREEN

H4: Waste disposal compliance GREEN GREEN

I: Affordable bills I1: Percentage of customers in water poverty GREEN GREEN

J: Satisfied customers

J1: Service Incentive Mechanism (SIM) GREEN GREEN

J2: General satisfaction from surveys GREEN GREEN

J3: Value for money GREEN GREEN

K: Easy to contact K1: Ease of contact from surveys GREEN GREEN



Outcome Measure Methodology Data

L: Bills are accurate and easy to understand

L1: Negative billing contacts GREEN GREEN

3.2.1. A: Reliable Supply

A1: Unplanned Customer Minutes Lost

Reporting appears to be robust and well established, but the methodology documentation needs to be clearly differentiated from that prepared for data reported in AMP5 and before. We note that the reporting is consistent with the requirements of the Final Determination, but differs from many other water companies, insofar as the time measurement threshold is zero rather than the more typical three hours. This measure seem likely to change after PR19 to facilitate comparability across the industry.

A2: Asset Reliability – Infrastructure

This sub-measure comprises Total Bursts and DG2 (Low Pressure). Both have a long history of reporting in previous AMP periods. The Company’s approach to reporting each is consistent with what was done previously and on the same basis by which Performance Commitment targets were set. We noted minor errors in the reporting definition; raw water mains and 3rd party bursts have always been and are still incorrectly included. We noted that common supply pipes have always been and are still reported for DG2 on a reactive basis. Further work is needed to migrate the existing methodology documentation to the specific AMP6 context and the data produced is well understood and reliably reported. The overall assessment of the sub-measure as stable is appropriately reported and has a full evidence base.

A3: Asset Reliability – Non-Infrastructure

This sub-measure comprises Turbidity and Unplanned Maintenance. Both have a long history of reporting in previous AMP periods. The Company’s approach to reporting each is consistent with what was done previously and the basis upon which Performance Commitment targets were set. Although further work is needed to migrate the existing methodology documentation to the specific AMP6 context, the data produced is well understood and reliably reported. The overall assessment of the sub-measure as stable is appropriately reported and has a full evidence base.

3.2.2. B: Resilient Supply

B1: Population in centres >25,000 at risk from asset failure

This measure exists as a means of attaching an Outcome Delivery Incentive to the Southern Resilience Scheme. However, in order to confirm satisfying the criteria of the measure, it is necessary to confirm the resilience of the wider supply area. We challenged Bristol Water to provide a body of evidence that this is the case. We have seen good progress in the gathering of operational records and network modelling which is still being carried out. The Company was able to provide evidence supporting the reported figure. Further improvements are needed to the methodology documentation. Though not significant for this year’s reporting, we have challenged the inclusion of a rump of properties fed in part by Cooks Corner.

3.2.3. C: Sufficient Supply

C1: Security of supply index (SOSI)

There remain improvements necessary to the methodology documentation although we have seen considerable progress in this respect. The reported figure is comfortably achieved against the set criteria and good progress is being made in improving the audit trails within the suite of spreadsheet that support the reported data. This has included the linking of spreadsheets and inclusion of formulae to minimise the potential for transcription error. Further work is needed to migrate the existing process documentation to the specific AMP6 context and the data produced is well understood and reliably reported.

C2: Hosepipe ban frequency

The methodology for reporting this measure is semi-empirical and complicated, with the reservoir control curves and distribution input data used stretching back to 1910. There remain improvements necessary to the process documentation. The spreadsheet used to model the implied hosepipe ban days is currently populated up to 2002. This should be brought up to date. In developing the methodology documentation,



care is needed to avoid “black boxes”. For assurance to be robust, the efficacy of reporting tools needs to be demonstrable. A suite of process diagrams will help understanding and show where checks and controls should be taking place. We noted that the auditee was successfully endeavouring to link the various source spreadsheets and ensure that all cells are populated with appropriate formulae to improve the repeatability of the calculation. The reported outturn performance is well within target and was not stretched by adverse climatic conditions during the report year.

3.2.4. D: Mean Zonal Compliance (MZC)

D1: Mean Zonal Compliance

Reporting uses established industry methodology (DWI) with quality assurance checks and controls built into reporting. This measure has a long history of reporting in previous AMP periods. The Company’s approach to reporting is consistent with what was done previously and from the same basis by which Performance Commitment targets were set. Only minor changes were needed to migrate the existing process documentation to the specific AMP6 context, the data produced is well understood and reliably reported.

3.2.5. E: Water is good to drink

E1: Negative Water Quality Contacts

This performance commitment relates to Bristol Water’s calendar year performance of customer contacts about the appearance, taste and odour of water. Our findings for this ODI were satisfactory. The Company reports in line with the established Drinking Water Inspectorate industry methodology and robust quality assurance checks and controls are built into reporting. There were some minor changes made to improve the Company’s documentation of processes and procedures as a result of the assurance review, which we consider fit for purpose. The Company has agreed a performance ratchet for this commitment to reach 2,221 consumer contacts for the number of negative water quality contacts by the end of AMP6. In 2015 calendar year, the target was set at 2,422 and Bristol Water is reporting well within this target with an outturn of 2,329 contacts.

3.2.6. F: Efficient use of resources by Company

F1: Leakage

The calculation of the reported leakage figure is complicated and requires the calculation of numerous component parts, underpinned by key assumptions. It is calculated “top down” and “bottom up”, with any discrepancy spread amongst the component elements on the basis of the likelihood of error. For Bristol Water, this is a reporting area that is very well established and reported by skilled experienced staff who have been subject to rigorous challenge and audit over many years. We consider the development of the reported figure below.

Bottom up leakage assessment: The Company’s methodology for calculating supply pipe leakage is based on industry best practice, which incorporates information gathered as part of a collaborative study undertaken with other water companies. Netbase is used to calculate distribution losses. The Netbase system receives automated downloads from a variety of Bristol Water’s systems: telemetry (flow and pressure), GIS (properties, mains and fittings), SAP (job management) and Billing. Some data validation is completed automatically by Netbase, whilst other validation is undertaken manually. An exercise is carried out to confirm that the property numbers in Netbase are the same as those being used elsewhere in performance reporting. At present only two people have administrative rights to Netbase and several have read-only access. Netbase also keeps an automatic edit log and it is easy to revert to the raw data. This was illustrated during the audit and provides confidence that there is a good audit trail for edits to the data in Netbase and that it is relatively ‘tamper-proof’. At the end of each month post-validated night flow data are extracted from Netbase. Legitimate night use values are subtracted from night flows and the remainder gives the leakage value. A company-wide household legitimate night use has been applied this year and is equal to the figure used last year, which is from a study carried in 2014/15. Work was carried out on non-household legitimate night use this year as there was previously reliance on an old survey from 2007. This resulted in an increase in the figure closer to a level typical for other companies. Reservoir losses are sampled directly and represent a small contribution to leakage. Trunk main losses are calculated based on the length of trunk main outside of DMAs. The Company undertakes TMA (trunk main metered area) monitoring, which appears to provide a robust basis for calculating this component of leakage.



Top down leakage assessment: Per capita consumption (PCC) has remained steady over the past three years as has distribution input (DI) and this is reflected in water delivered (potable). PCC is discussed against Performance Commitment G2. During the audit we looked at the spreadsheet used to calculate the meter error on (DI) which is based on work undertaken on validation of the meters. It is assumed that the meter error is symmetrical. We investigated the impact of changing the error on the largest treatment works (in terms of volume supplied) from 1% to 5%. This changed the mean error and had a minor effect on the estimated level of leakage. We looked at specific areas of detail in plumbing losses, which are currently assumed to be zero for void properties, the assumptions for illegal use by troughs and estimated consumption from illegal use of standpipes as based on consumption being the same as for legal standpipes. Although the studies on which these are based may be relatively dated, they do not appear to be unreasonable and there is a high adjustment error placed on this component, which is appropriate.

Water balance calculations: We sampled several of the spreadsheets that underpin the calculations and found no errors that would materially impact upon the reported figure. The whole process of calculating leakage has supporting documentation, but there is need to draw the many documents together and clearly identify the checks and controls in the methodology that we have seen to be carried out. It is clear that there are a number of checks being undertaken of different elements of the water balance. However, there is further work to do in documenting these checks. We would like to see the development of a formalised analysis to understand the level of uncertainty around the leakage number and the sensitivity to different components, and to document this. The procedures followed appear to align with relevant guidance (e.g. UKWIR supply pipe leakage guidance). We note that the people responsible for the collating, analysing and reporting against the measure are the same people as are producing the methodology document. This will mean that the methodology documents will be aligned with outturn practice. This is reasonable and will allow others to assist or assume the reporting role. We concluded that this well-established reporting results in a reported figure that is robust.

3.2.7. G: Efficient use of resources by customers

G1: Meter penetration (%)

The calculation of meter penetration is a well-established and understood process and draws upon the properties data. Meter Penetration equals Domestic Metered Properties divided by the sum of Domestic Metered Properties and Domestic Unmetered Properties. For the calculation, the yearend figures are used. There remain minor improvements necessary to the methodology documentation which provide a detailed explanation of the process of calculating the reported figure. The outturn figure 47.3% falls short of the Final Determination target of 50.4%. The reasons are well understood by the Company.

G2: Per capita consumption

Per capita consumption is a major element of the top down leakage assessment. Measured PCC is calculated based on property figures and assumed measured occupancy rates. Measured occupancy is based on an extrapolation from the 2001 census and from surveys undertaken in 2007. The team recognise that they need to undertake a more up-to-date study, but are confident in the 2.1 measured occupancy figure currently used and have no reason to believe that it is not the best central estimate available at the present time. We discussed whether the Company can be confident in the data provided to them by CACI and were surprised to see that there is relatively little information provided in the CACI spreadsheets in terms of caveats and level of accuracy. The team reported that they have had reason to query the CACI data in the past them when looking at trends in the data provided. However, over the last couple of years they have had no reason to believe that the data provided are wrong based on the checks undertaken by the team on trends. CACI works according to protocol of JICPOPS and have provided documentation that they have a level of data assurance. Overall it was agreed that there is further work to do on the sensitivity to different assumptions. There remain improvements necessary to the methodology documentation. The reported figure is correctly calculated based upon the available data.

3.2.8. H: Sustainable Environmental Impact

H1: Total Carbon Emissions

Total carbon emissions were previously reported as an Ofwat KPI and as part of the Ofwat June Return (JR). The Performance Commitment (PC) is calculated as total net emissions / total population. The methodology documents in place at the methodology audit were subsequently updated to reflect the change to the PC. The Company continues to use the UKWIR Greenhouse Gas (GHG) estimation workbook to calculate



emissions as for previous JR and KPI reporting. The population data was subject to a separate audit. The majority (87%) of emissions are accounted for by total grid electricity used by the Company. Reporting is based on supplier data, which is reconciled with Company records. A small proportion of the remaining data was based on extrapolated / prior year data but this is not considered material. The data presented at audit had not been subject to internal review but this was completed following the audit.

H2: Raw Water Quality of Sources

The methodology for reporting H2 was not agreed as part of the PR14 process. As highlighted by the definition included in FD14, the Company intends to further develop this measure in 2015-20. This means that the method for assessing performance will be different for 2015/16 in comparison with the remainder of AMP6. The Company intends to agree the approach for reporting 2016/17 onwards through a subgroup of the Challenge Panel. The Company is reporting ‘Deteriorating’ in line with the target for 2015/16. This is based on the % of samples from Blagdon, Cheddar and Chew reservoir with >10,000 algal cells per ml. An agreed level of performance to correspond to the overall assessment was not included in FD14. The Company is basing its assessment on the long term trend, which appears reasonable.

H3: Biodiversity Index

The assessment methodology is based on the Defra ‘Biodiversity Offsetting Pilots Guidance for Developers’, which is reflected in the assessment spreadsheets used to calculate a value for the index. The 2014 Final Determination describes the measure as being an ‘aggregate Biodiversity Index for its overall landholdings’. The Company has assessed 49 out of its c.250 sites, which it estimates as equating to between 80 and 90% of its overall landholdings. The Company appears to have followed its methodology to compile the data, although the data presented at audit had not been subject to internal review. Some errors were identified that corrected following the audit, but these did not change the overall assessment of ‘Improving’. The Company intends to further develop the approach for reporting through a subgroup of the Challenge Panel.

H4: Waste Disposal Compliance

The Company has previously reported performance as an internal KPI and is continuing to report using the same methodology, which is detailed in a comprehensive methodology document. Internal checks and controls are in place, designed to improve the accuracy of reported data. Performance is reported based on compliance with Environment Agency permits (Environmental Permitting Regulations). We identified some inconsistencies between the permit limits being used to assess performance and those in force. This was corrected following review. Compliance with flow conditions in not included in the assessment although flow compliance is part of the EA permits. This is consistent with historic reporting and not explicitly defined as part of FD14.

3.2.9. I: Affordable Bills

I1: Percentage of customers in water poverty

The calculation of the reported figure for this is carried out within a model supplied by third parties (CACI). Due to the continuing dialogue between Bristol Water and its supplier, our assurance is limited to a confirmation that the number reported is as obtained from the CACI model. However, we have no reason to doubt the reported figure. We propose to review the workings and calculations in more depth in future years.

3.2.10. J: Satisfied customers

J1: Service Incentive Mechanism

Customer contact occurs at both the Company itself and at BWBSL. The results from both are aggregated for reporting purposes. We audited each separately and comment as follows:

Bristol Water: Whilst we identified a number of potential ‘amber’ issues during the course of our audits, these did not materially impact on the overall SIM score and were all satisfactorily addressed prior to submission. These related to the unwanted calls methodology not being applied in line with guidance for a small number of codes and written complaints associated with compensation claims not being systemically reported. They have therefore being amended to ‘green’ following remedial action by the Company. There were no issues identified with the SIM survey.



BWBSL: The review highlighted potential under-reporting of unwanted calls due to a relatively high error rate in the classification by agents. This has subsequently been addressed by adjusting the reporting to take account of the error rate. We also identified that Depot and Back Office teams were not systematically logging customer calls. While this is not in line with the SIM guidance, the volume of calls handled by these agents was very low and it does not materially impact on the overall SIM score. BWBSL has subsequently undertaken refresher training and built in checks to ensure that all staff are following the established processes. There were no material issues identified with the other SIM components, relating to written complaints or SIM survey.

J2: General satisfaction from surveys and J3 Value for money

There were minor gaps and weaknesses identified in documenting of methodology by the Company, which have subsequently been addressed satisfactorily. The source of the reporting are the results from a survey commissioned from a third party provider carried out in line with established market research practices. We followed the audit trail to confirm the accuracy of the reporting.

3.2.11. K: Ease to contact

K1: Ease of contact from surveys

There were minor gaps and weaknesses identified in documenting of methodology by the Company, which have subsequently been addressed satisfactorily. The source of the reporting is the result from a survey commissioned from a third party provider carried out in line with established market research practices. We followed the audit trail to confirm the accuracy of the reporting.

3.2.12. L: Bills are accurate and easy to understand

L1: Negative billing contacts

There were minor gaps and weaknesses identified in documenting of methodology by the Company, which have subsequently been addressed satisfactorily. The reported number is a subset of the SIM reporting for which no issues with the data were identified.

3.3. Supporting Data for 2015/16

Table 3-2 RAG Assessment: Supporting Data

Methodology Data

AMBER GREEN

In common with much of the reported data, the supporting data (properties, volumes) are the result of a well-established and previously audited processes. The documentation needs to be improved. However, the staff responsible have the appropriate skills and experience and were able to demonstrate checks and controls in place.

Properties: Each month, a spreadsheet is automatically exported from Rapid (Bristol Water’s Billing System – shared with Wessex Water). A methodology document exists that was previously used to report against the Bristol Water internal annual return data (a legacy of the old June Return reporting requirement). The property data is reported at a component level and the methodology document needs to be improved. We noted that indirect properties, such as flats, (19,617 in April 2016 supplied through 3.737 meters) are treated as non-household properties. We were told that this was how the figures had been calculated in the past and accorded with Ofwat guidance. An adjustment is made by reducing the household figure and adding to the non-household figure. We saw the recalculation of the total properties in both the raw and calculated figures, using pivot tables.

Volumes: A methodology document exists that was previously used to report against the Bristol Water internal annual return data (a legacy of the old June Return reporting requirement). The volume data is reported at a component level and the methodology should be amended and updated to make clear the



linkages to other aspects of the Annual Performance Report, to avoid the use of an incorrect figures in data transposition.

3.4. GSS Payments for 2015/16

Table 3-3 RAG Assessment: GSS Payments

Methodology Data

AMBER GREEN

GSS payments are generated at both the Company itself and at BWBSL. The results from both are aggregated for reporting purposes. We audited each separately and comment as follows:

Bristol Water: The Company methodology and processes for managing and reporting GSS, particularly in relation to appointments, is either not documented or not fully documented. Notwithstanding this issue, we found that the Company identifies GSS events systematically and effectively in line with its legal obligations and makes payments to customers where due. There was one area of the business, appointments made by the Water Regulations team, where there was no robust audit trail to demonstrate compliance although we found no evidence of unreported failures.

BWBSL: The Company methodology and processes for managing and reporting GSS are documented and appear to be fit for purpose. There are also robust checks and controls built into management of GSS and there were comprehensive audit trails to demonstrate compliance. Overall, we were satisfied that the Company is meeting its legal obligations with respect to GSS and making payments to customers where due. The only exception related to the Meter Reads team’s practice of offering all day appointments, which is not a permitted timeslot, however the numbers involved were not material and there was no suggestion that the Company has been inconveniencing or failing customers. BWBSL has subsequently withdrawn this option.

© Atkins Ltd except where stated otherwise. The Atkins logo, ‘Carbon Critical Design’ and the strapline ‘Plan Design Enable’ are trademarks of Atkins Ltd.

Jonathan Archer Atkins Epsom Gateway 2 Ashley Avenue Epsom KT18 5AL

[email protected]

bristol water - 2015/16 annual performance report … water - 2015/16 annual performance report and...

Documents