HSE Health & Safety

Executive

Measuring the Health and Safety Executive’s Field Operations Division inspection

effectiveness

Prepared by Amey VECTRA Limited

for the Health and Safety Executive 2002

RESEARCH REPORT 046

HSE Health & Safety

Executive

Measuring the Health and Safety Executive’s Field Operations Division inspection

effectiveness

GC Simpson and V Scotney Amey VECTRA Limited

Europa House 310 Europa Boulevard Gemini Business Park

Westbrook Warrington

WA5 7YQ

Research was undertaken to identify a systematic approach to the evaluation of inspection effectiveness within the Field Operations Division (FOD) of the Health and Safety Executive (HSE). This was undertaken in response to increasing demands (e.g. the delivery of Best Value, Revitalising Health and Safety etc.) to identify outcome measures (in additional to traditional output measures) suitable across the range of HSE FOD inspection activities.

The initial focus of the research was based on a combination of interviews and workshops (with HSE staff) and a Duty Holder questionnaire to establish the objectives and expectations of inspection as perceived by the stakeholders in the delivery of improved health and safety. These expectations were used to evaluate the potential health and safety performance indicators which had been proposed in the literature to devise a suite of indicators suitable for use in a Regulatory context as there proved to be no immediately obvious choice of suitable indicators based on previous research, for example, Best Value.

This report and the work it describes were funded by the HSE. Its contents, including any opinions and/or conclusions expressed, are those of the authors alone and do not necessarily reflect HSE policy.

HSE BOOKS

© Crown copyright 2002

First published 2002

ISBN 0 7176 2375 0

All rights reserved. No part of this publication may bereproduced, stored in a retrieval system, or transmitted inany form or by any means (electronic, mechanical,photocopying, recording or otherwise) without the priorwritten permission of the copyright owner.

Applications for reproduction should be made in writing to: Licensing Division, Her Majesty's Stationery Office, St Clements House, 2-16 Colegate, Norwich NR3 1BQ or by e-mail to hmsolicensing@cabinet-office.x.gsi.gov.uk

ii

Table of Contents

1.0 Background 1

2.0 Scope of Measurement Objectives 1

3.0 Refining the Requirements 2

3.1 Summary of Findings 33.2 Implications for a Measurement Process 53.3 Linking Requirements to Potential Indicators 5

4.0 Toward a Soluti on 8

4.1 Identifying an Appropriate Indicator Suite 84.2 Development of a Common Measurement Approach 104.3 Tailoring the Measurement Proformas 124.4 Model Questionnaire 13

5.0 Benefits of a Tool Kit Approach 13

6.0 Toward Implementation 15

The full text of this report is available on the Internet at www.hse.gov.uk/research/rrhtm/index.htm

iii

iv

1.0 Background

The Director of Field Operations Division of the HSE commissioned Amey VECTRA to undertake a project to evaluate approaches to the measurement of the effectiveness of the various inspection activities of HM Inspectors of Health and Safety in the Field Operations Directorate (FOD) with particular emphasis on the development of outcome measures.

The issue of the effectiveness of inspection in ensuring Duty Holder compliance with Health and Safety Regulations has been discussed and debated for some time both within the Health and Safety Executive (HSE) and externally. However, there was a perception, within the HSE, that while previous work in this area may have helped to define the problem (at least in part), nothing is currently available which provides a systematic basis on which to evaluate inspection effectiveness.

Increasing emphasis in, for example, a number of recent Government initiatives for the public sector (including Best Value etc.) on the need for outcome measures of HSE activities in addition to the (traditional) use of output measures also reinforced the need to consider the inspection measurement issue.

2.0 Scope of Measurement Objectives

On the basis of the Tender document and subsequent discussions between the Amey VECTRA project team and the HSE, the overall measurement objectives to support the development of a process to assess the effectiveness of inspection was as follows:

♦ The study was to address the measurement of inspection effectiveness strictly within the terms of inspection activities within the HSE’s Field Operations Division (excluding therefore any of the inspection processes associated with permissioning regimes such as conducted elsewhere within the HSE).

♦ Identification of OUTCOME measures to supplement current OUTPUT measures.

♦ A measure (or measures) which could be used to assess the effectiveness of FOD inspection activities.

♦ A measure (or measures) which could form a baseline to enable (eventually) a basis for comparison across inspection approaches and/or strategies to devise a more effective/efficient mix (for example to assess the balance between pre-emptive/routine inspection and reactive/investigative inspection, assess the relative benefits of pre-warned versus unannounced routine inspections etc.).

♦ A measure (or measures) which would capture success (or otherwise) at Sector level to identify possible cross-fertilisation between Sectors.

1

In order to meet these objectives the project was planned in three phases. The first phase was the collection and collation of information on the objectives and expectations of inspection (both from within HSE and external bodies including Duty Holders). Phase 1 also including consideration of the literature on health and safety performance indicators. The second phase was to identify appropriate indicators (and their associated measurement techniques) which could be used to assess performance against the objectives and expectations identified in phase one. The final phase was to expose the assessment/measurement approaches developed in phase two to Peer Review by HSE FOD staff.

Phase 1 was, essentially, a process of refining the requirements for and boundaries of an inspection effectiveness measurement process and Phases 2 and 3 the process of developing a solution. Phase 3 also included identification of issues to be considered in the implementation of an inspection effectiveness evaluation process.

3.0 Refining the Requirements

It was immediately apparent that the effectiveness of inspection could be addressed on several levels. It could, for example, be assessed in terms of the impact of a single inspection visit to a single Duty Holder, at the other extreme it could be assessed in terms of the impact of the inspection process on the national improvement in health and safety performance.

Similarly differences in the interpretation of what is encompassed within inspection could influence the measurement of effectiveness. For example, the “definition” of inspection could be restricted to the element which makes it unique from all other aspects of HSE’s activity, i.e. the use of the Inspector’s Warrant Card to under-pin Enforcement Action. At the other extreme, inspection could be defined as all that an Inspector does including the advice offered, the involvement in Projects and campaigns etc.

In addition, there are numerous parties (e.g. Duty Holders, government etc.) who have an interest (either direct or indirect) in the effectiveness/impact of the inspection process in promoting health and safety.

Unless this range of views was addressed and decisions made on the boundaries to the measurement of inspection effectiveness, the position could well arise where the indicators and their associated measures were assumed to measure a suite of issues whereas in fact they were, more correctly, reflecting other considerations.

On this basis, it was decided that a fundamental consideration in the success of the project would be to develop an understanding of the objectives and expectations of inspection both from within the HSE and from a sample of those regulated by the

2

HSE. In addition, it was also considered fundamentally important to define which aspects of the inspection process were to be included in the indicators/measures derived, in order ensure clarity of understanding when considering the interpretation of the measures proposed.

The data collection in relation to the objectives and expectations of the HSE in general and inspection in particular was obtained from two primary sources, from within HSE/HSC (by interviews and workshops) and from Duty Holders (via a questionnaire).

The literature review for the study focused, exclusively, on the identification of potential indicators of health and safety performance which could be of value when applied in a Regulatory context. Three sources of information were used, published literature (including internal HSE documents), information on the web-sites of other Health and Safety Regulators and a number of personal contacts (of the VECTRA Project Team) in the international health and safety business.

3.1 Summary of Findings

The most important issues which arose from this initial exercise (based on information from the HSE interviews/workshops and the Duty Holder questionnaire) were the identification of a wide range of measurement constraints, difficulties and complexities arising from:

♦ The variety of inspection/Inspector influences on Duty Holders ♦ The variety of inspection formats/approaches and activities ♦ The variety inspection objectives and expectations ♦ The variety of Duty Holders ♦ Availability/nature of data to be measured

Figure 1 deals with one aspect of the findings in relation to expectations by showing the preferred styles of inspection across a variety of groups. Most of the group names are self-evident, however, it should perhaps be explained that “interested” public is used to describe those members of the public who had contact with HSE activities as a result of personal involvement in work related injuries or ill-health. The dotted line for Government expectations is to reflect that the relative priority will vary from time to time as will the relative emphasis in light of major events and/or changes in political imperatives.

The wider aspects of the Duty Holder questionnaire dealing with perceptions of HSE can be summarised as follows:

♦ There is, generally, a positive reaction to HSE

3

♦ The positive view is hardly influenced by enforcement action (whether in the form of written advice, INs, PNs, formal accident investigations or even prosecutions)

♦ Where there are negative views they tend to be more likely to come from companies employing 50 or less

INSPECTION PROCESS

PROACTIVE

REACTIVE

ENFORCEMENT

ADVICE & GUIDANCE

PRESSURE GROUPS

"INTERESTED" PUBLIC

BUSINESS DUTY

HOLDERS

GOVERNMENT

UNIONS

GENERAL PUBLIC

Key: The forms of Inspection “preferred” by Duty Holders The forms of Inspection “preferred” by Government (it should be noted that the choice and mix will vary with circumstances) The forms of Inspection “preferred” by Pressure Groups and “Interested” Public opinion The forms of Inspection “preferred” by Trade Unions and general Public opinion

Figure 1: Expectations in Terms of —Preferred“ Forms of Inspection

The following points summarise the important findings from the literature review:

♦ No evidence was identified that the issue of systematically assessing inspection effectiveness had been seriously addressed, let alone resolved, by other Health and Safety Regulatory Bodies.

♦ There is little of immediate value in the literature to identify specific indicators and/or measurement approaches which deal effectively with the complexity of the inspection process.

♦ Much of what has been published on health and safety performance indicators is focused more on advocating their value rather than reporting their use.

♦ Despite the above limitations, a suite of 300 plus potentially useful indicators was identified.

4

3.2 Implications for a Measurement Process

On the basis of the examination of objectives, expectations and the health and safety performance indicator literature, the goal of measuring the effectiveness of inspection has three fundamental problems:

♦ Neither HSE in general, nor inspection in particular, can be seen as having a direct or even straightforward effect on achieving its ultimate objective of increasing compliance with H&S Regulations or, more generally, improving H&S standards - at best the effect is analogous to that of a catalyst.

♦ Unfortunately, unlike measuring the effect of a catalyst in a chemical reaction, in this context, every time the catalyst is used each circumstance is unique.

♦ There is little of immediate value in the literature to identify specific indicators and/or measurement approaches which deal effectively with the complexity of the inspection process and/or its relationship with the ultimate objective of aiding in the improvement of health and safety standards. While there have been a considerable number of health and safety performance indicators suggested, few have been proven in anything other than a limited context and there is little order or structure among those which have been suggested.

The above conclusions make the measurement of the effectiveness of inspection:

In measurement terms:extremely complex with no possibility of a simple or even straightforward measurement approach

and

In statistical terms:a virtual impossibility in any way which would be remotely cost­effective other than as a series of high structured and constrainedinitiatives (such as the recent Single Issue Inspection Project).

3.3 Linking Requirements to Potential Indicators

In order to be able to link the information obtained on the objectives and expectations of inspection and the variety of “routes of influence” on Duty Holders to the health and safety performance indicators suggested in the

5

literature, it was decided that a framework linking inspection activities and Duty Holder action was essential. The information derived in Phase 1 was used to create the framework shown in Figure 2.

The principal considerations behind the development of the framework were as follows:

♦ It is action within individual companies which, collectively, produces the national outcome of improved health and safety.

♦ The principle elements within a company which create/predispose such improvements are, management and workforce commitment to create effective risk management within a positive safety culture.

♦ Inspection activity can promote positive behaviour (through advice/guidance) and/or avoid negative behaviour (via enforcement). The effect of any HSE action will be enhanced if the Duty Holder(s) have a positive perception of HSE and its role.

♦ HSE action is influenced by its own major targets (e.g. “Revitalising Health and Safety”) and by external pressure (both political and public; both of which tend to be transitory).

♦ HSE’s ability to deliver effective Inspection is dependent on its capability in terms of both resource and information/intelligence.

6

REVITALISING EXTERNAL PRESSURE REDUCED ACCIDENTS & WORK RELATED ILL HEALTHHEALTH & SAFETY POLITICAL & PUBLIC

AVOID NEGATIVES

PROMOTE POSITIVES

POSITIVE PERCEPTION

HSE ACTION

ROUTINE INSPECTION SINGLE ISSUE INSPECTION ACCIDENT INVESTIGATION

COMPLAINTS

VERBAL ADVICE (During Visit) WRITTEN ADVICE (Post Visit)

CAMPAIGNS SPECIFIC SEMINARS SECTOR INITIATIVES

PROJECTS

ENFORCEMENT

ADVICE

INSPECTION CAPABILITY

Training ó ó additional skills

Selection Retention

IT Support Admin Support

RESOURCE:

INFORMATION:

Better Intelligence Regulations/Guidance

Sector Specific Guidance General Seminars

Campaigns Projects Research

HSE Books Web Site Info Line

POTENTIAL INDICATOR SUITE

IN-COMPANY ACTION

POSITIVE SAFETY CULTURE

EFFECTIVE RISK MANAGEMENT

MANAGEMENT COMMITMENT

WORKFORCE COMMITMENT

initial

MEASUREMENT REQUIREMENTS

SELECTED SUITE FOR TESTING

Figure 2: Framework for the Development of Potential Indicators Suite

7

4.0 Toward a Solution

The most likely avenue for success in such a complex measurement environment, appeared to be the development of a suite of measures which would, collectively, provide a basis on which to say:

On the balance of probabilities, inspection is positively contributing to an environment of improvement.

“Hard” proof of a direct contribution to specific improvement was unlikely to be feasible.

The process of measuring the effectiveness of inspection, based on the complexities identified will be best pursued as a form of performance epidemiology – a process of refining relationships between inspection and a variety of outcomes. In this way by developing a data set that continues to build the relationship, the links are gradually strengthened to a point where a causal relationship can be reasonably inferred even if the exact nature of the link cannot be established.

It was apparent that no single indicator would encompass the issues (as identified in Figure 2) for which indicators were required. It was decided therefore that a suite of indicators would be sought which, collectively, covered the breadth of issues required. On a similar basis it was apparent that the variety of Duty Holders to be encompassed, the variety of hazards/risks and the range of influences to be addressed, would require a high flexible approach to indicator measurement. Such an approach would need, ideally, to be capable of being tailored to be meaningful on a wide range of particular issues while, at the same time, having sufficient commonality to allow comparisons.

The concept of a Inspection Effectiveness Measurement Tool Kit emerged as a potential solution. Such a Tool Kit was seen as consisting of a suite of indicators with a selection of measurement options. Common indicators together with a common measurement approach would provide consistency, while the ability to tailor the measurement to particular issues would ensure the assessment was both flexible meaningful.

4.1 Identifying an Appropriate Indicator Suite

It was apparent from Phase 1 that the there were no simple solutions to the inspection effectiveness measurement problem and that in order to reflect both the breadth and complexity of the inspection process and its influence it would be necessary to develop a specific suite of indicators. In addition, the measures associated with the indicators would, ideally, need to be relevant and applicable across the suite and sufficiently flexible to be meaningfully tailored to specific topics/initiatives.

8

As none of the 300 plus potential indicators identified in the literature survey had any direct links to the inspection process, a series of iterations were undertaken testing the suggested indicators against the framework and the inspection expectations. On the basis of this exercise the following suite of indicator topics was selected:

♦ Achievement Indicators ♦ Direct Indicators ♦ Indirect Indicators Inspection Activities { Campaigns & Projects

The link back to the framework was completed by the addition of two further indicator categories:

♦ Duty Holder Awareness ♦ Internal Enabling Indicators

The final indicator categories were defined as detailed below:

Achievement Indicators: These reflect actual (proven) achievements in H&S performance across UK plc - although the link between these and inspection is tenuous, at best, they are important in the interpretation of any claims made from “lower” level indicators.

Direct Indicators: These indicators reflect direct Enforcement Action on Duty Holders (e.g. Notices).

Indirect Indicators: These indicators reflect issues which can be influenced by inspection but were the effect is indirect (they affect factors which influence the pre-cursors of improved H&S performance). They are sub-divided as follows:

Inspection Activities: These addressed the issues which Inspectors would consider during the course of a normal inspection visit and were designed to create a scoring mechanism which could be used where no Enforcement Action was taken. The intention of these indicators is to enable measurement of inspection outcomes (i.e. improvement, or otherwise) where the only action taken by the Inspector was advice (whether written or verbal).

Campaigns & Projects: As Inspectors spend a part of their time dealing with a range of specific campaigns and/or projects it is only sensible that any attempt to consider an overall measure of inspection should include consideration of the benefits achieved from such activities.

9

The Duty Holder awareness category addressed the positive perception and (in part) the management commitment issues as raised on the framework. The Internal Enabling category represented, essentially, a rewording of the inspection capability issues as raised in the framework.

Duty Holder Awareness: Without Duty Holder awareness of his responsibilities in relation to H&S, it is unlikely that improvements can be achieved. A contribution from inspection activities to increasing Duty Holder awareness of any/all aspects of H&S is therefore a positive pre-cursor to improved H&S standards and should therefore be reflected in the measurement of inspection effectiveness.

Internal Enabling Indicators: HSE must be, structurally and organisationally, capable of delivering effective inspection on a continual improvement basis, therefore action to improve the delivery of inspection should be measured and tracked.

4.2 Development of a Common Measurement Approach

The principal guiding consideration in the development of the measurement approaches was that the measures should not require an Inspector to consider issues which were additional to those normally considered during an inspection visit.

Six measurement topics were selected (on the basis of the literature, the framework and their relevance in relation to both Regulation, normal Inspector considerations and models of Health and Safety management such as HS(G)65):

♦ Risk Assessment♦ Roles and Responsibilities♦ Training & Commitment♦ Communications & Involvement♦ Workplace Precautions♦ Housekeeping & Welfare

Four measurement approaches were developed all based on the use of structured subjective assessment by the Inspector (as had proved viable during the Single Issue Inspection Project). All four methods are also based on a four­point assessment scale.

Table 1 shows an example of one of the measurement approaches (the other three are described in the more detailed project reports).

10

Table 1: Example of one of the Proposed Measurement Approaches

Indirect Indicator (Inspection Activities)

Score Anchor Description √

Indirect Indicator 1 4 Comprehensive, documented, controls identified & in regular use, review procedure established

RISK ASSESSMENT/MANAGEMENT

3 Comprehensive, documented, controls identified & in regular use 2 RA limited in scope, adequacy of controls or control usage 1 RA not in existence or generally insubstantial

Indirect Indicator 1 4 Clearly defined and understood “formal” R&R or effective/clear “informal” R&R (in SMEs)

ROLES & RESPONSIBILITIES 3 Arrangements in place generally good but some oversights 2 Little serious consideration of establishing H&S R&R 1 No clear consideration of H&S R&R

Indirect Indicator 1 4 Training needs identified (formally or informally) training resourced well and effective & recorded

TRAINING & COMMITMENT 3 Arrangements in place good on some issues but some oversights 2 Little serious consideration given – training provided limited in scope and/or ineffective 1 No clear consideration of H&S Training or generally insubstantial

Indirect Indicator 1 4 Management and workforce open – established formal or informal channels (dependent of size)

COMMUNICATIONS & INVOLVEMENT

3 Good intention/principle with channels identified but not fully effective (bottom-up limited) 2 Little serious consideration of H&S information/communications 1 Little or no effort both management and workforce distrustful of each other

Indirect Indicator 1 4 High standard of control (both hard and soft) understood by workforce & regularly checked

WORKPLACE PRECAUTIONS 3 Good standard on one aspect (e.g. engineered controls) but not as good on procedural or vice-versa 2 Some good intent but not effectively/reliably used/maintained 1 Poor standards – broken guards, over-ridden interlocks, PTW etc. overlooked

Indirect Indicator 1 4 Clean – tidy obvious pride in workplace – systems in place and used (e.g. Tidy Friday etc.)

HOUSEKEEPING 3 Generally good with some oversights (e.g. inadequate temporary storage etc.) 2 Major limitations on either general housekeeping or welfare facilities 1 Poor standards – little evidence of any care or pride in workplace

11

4.3 Tailoring the Measurement Proformas

It was evident during the early stages in the selection of an indicator set for FOD inspection that both the variety of influences and the variety of Duty Holders (both in terms of industry and size) would make the selection of a single, universally applicable, measurement approach extremely difficult. Although a single measurement approach would be feasible, it would be at such a generic level as to make its value, as a measure of improvement, severely limited.

In order to work within the range of circumstances faced by FOD, it became apparent that the best approach would be to develop common measurement/scoring approaches which could be tailored to specific Sectors/SICs and/or objectives of specific initiatives (e.g. campaigns).

The measurement approaches developed were designed to accommodate such tailoring.

For example, consider an initiative dealing with smaller firms across Sectors (in say, foundries and light engineering/manufacturing). The Workplace Precautions Indicator in foundries could focus on PPE whereas in the light engineering establishments it could focus on machinery guarding. To achieve this only the evidential statements would require modification.

Tailoring allows meaningfulness, while the use of a common approach maintains the ability to aggregate results. Take, for example, an initiative on PPE usage. Metals and Minerals could decide to check PPE in foundries in relation to, for example, teeming/pouring and in quarries in relation to, say, crushing plants. You define required PPE - for teeming, say, visors, gloves, aprons, gaiters, boots etc., for crushing plants, say, ear protection, eye protection, gloves, boots. These are then assessed against a pre-determined % value representing “high compliance”. This then allows to meaningfully score the % with high compliance at SIC level, then combine at Sector level and then across Sectors.

Similarly a campaign on, say, safety training, could use the scoring proforma for the Training and Commitment Indicator as the basis for an evaluation tool (or part thereof) for the campaign.

Some issues, in particular, the Duty Holder Awareness/Recognition indicator category could also be successfully addressed by questionnaire. The detailed project report includes example questionnaires and an Appendix on questionnaire design principles.

12

4.4 Model Questionnaire

As mentioned earlier, some of the evaluation would be best conducted by questionnaire. In particular, assessment of the Duty Holders’ awareness and recognition of their health and safety needs/responsibilities and their perceptions of HSE. The main report on the project (VECTRA Report No. 300-226 R2) includes detailed consideration of the type of questionnaire which would be appropriate and (in order to allow for the tailoring which is central to the approach advocated) an Appendix detailing the basic principles of questionnaire design.

Table 2 shows an example of a model approach to a questionnaire designed to address Duty Holder awareness and recognition.

5.0 Benefits of a Tool Kit Approach

The Tool Kit approach to the Measurement of Inspection Effectiveness would provide a highly flexible approach allowing:

♦ A focus on the measurement of change (i.e. improvement or otherwise) rather than on “actual scores” and, therefore, the opportunity to use some statistical analysis (based on non-parametric techniques as was the case in the Single Issue Inspection Project).

♦ Use of the six primary indicator measures as a common assessment suite which are meaningful in both enforcement and advice roles.

♦ Development of an annual FOD Evaluation Programme where selected activities/initiatives could be defined for formal assessment based on the common approach provided by the Tool Kit.

♦ Development of a FOD Effectiveness Database. Such a database would allow both year on year comparisons of selected outcome measures and data which would better inform strategic decisions about the allocation of resources within the Work Plan and the provision of more structured intelligence to operational managers about, for example, which inspection strategies are most effective in various circumstances.

♦ The creation of a database would also allow for the activities/initiatives to be spread over the year while maintaining the data for overall analysis at any time.

The flexibility provided by the Tool Kit approach would allow an Evaluation Programme to include measures based on routine inspection visits, single issue inspection initiatives, Key National Objectives, campaigns, projects, etc. It could also accommodate initiatives taken at a national level, at a Sector level or even at a SIC level.

13

Table 2: Model Approach to Duty Holder Awareness/RecognitionQuestionnaire

Tick One Box

Which of these Regulations apply at your Premises? Apply Do not apply

Management of Health & Safety at Work Regulations £ £ Manual Handling Regulations £ £ Display Screen Equipment Regulations £ £ Electricity at Work Regulations £ £ Construction Design and Management Regulations £ £ First Aid at Work Regulations £ £ Which of these statements are reasonable/unreasonable? True False We only need risk assessments for major hazards £ £ If staff don’t bother with safety rules that’s up to them £ £ We can’t be expected to keep telling staff about safety if they don’t want to listen £ £ The workforce need to be fully informed on the outcome of risk assessments £ £ We don’t need any specific health and safety responsibilities £ £ There’s no need for a health and safety policy – it’s all just paper £ £ We don’t need Safety Representatives £ £ If they damage their protective equipment, that’s their problem £ £ We don’t need to do any risk assessments – everyone here is experienced in the job £ £ We don’t need to worry about eating and drinking in the workplace £ £ Risk assessment is a one-off exercise £ £ If we tell staff about health and safety risks they’ll just complain even more £ £ We don’t need warning signs, everyone knows the problem areas £ £ There’s no need for any health and safety induction training £ £ You don’t need any special health and safety knowledge to know our problems £ £ If new workers have done similar jobs before we don’t need to give them more health and safety training £ £ It’s the worker’s responsibility to keep the place clean £ £ You can’t wrap people in cotton-wool, we’ve a business to run £ £

14

6.0 Toward Implementation

The variety and complexity of both the activities and the influences which should be measured in an evaluation of Inspection Effectiveness is so great that the implementation of any approach would need to be carefully structured and based on an evolutionary approach (slowly building a more detailed process with detailed, regular, reviews of value, effectiveness and practicality). On the basis of the issues considered during the project, the following points were identified as essential considerations in the implementation of an Inspection Effectiveness Measurement Programme:

♦ The development of a FOD Evaluation Policy which establishes the boundaries of the required assessments and expected outcomes (this would include, for example, a decision whether collation across the whole of FOD was required, whether collation across Sectors was required, whether collation across campaigns was required etc.).

♦ Development of a Evaluation Strategy by combining the requirements established above with a measurement approach/framework (such as that developed in this project).

♦ From the Strategy, specification of those aspects of the annual FOD Work Programme where evaluation will be mandated.

♦ Development of an Evaluation Database. Without a database the evaluation will, effectively, dictate the work programme (or, at least, those aspects which are to be formally evaluated).

♦ The data base will need to draw on each of the above points in order to ensure it is sufficiently flexible in terms of input while remaining capable of delivering the analysis specified in the Evaluation Policy.

♦ Such a data base would almost certainly be better if designed by someone with good knowledge of evaluation and some knowledge of IT rather than by an IT specialist.

♦ Once the database has been designed it should be piloted on a limited number of evaluation exercises (covering the range of requirements established in the Policy), with reviews of the Policy, Strategy and measurement approaches following the pilot study.

♦ Once any refinements in light of the above have been made, the first full year of application should be limited in terms of the number of issues/campaigns etc. covered in order to test the organisational infrastructure (access to data base, roles and responsibilities review of measurements etc.).

♦ Where the Evaluation Strategy mandates evaluation, the design of the evaluation must be included from the outset of the activity, it is not possible to tag-on evaluation at the last minute and have any hope of meaningful results (especially where there is an intention to collate a range of evaluations to provide a high-level assessment of effectiveness).

15

♦ Where evaluation is mandated, the proposed measures etc. should be reviewed against the Policy prior to use to ensure that tailoring of the actual measurements proposed retains a level of comparability. The Evaluation Policy and Strategy, together with the measurement approaches should be formally reviewed on an annual basis.

Printed and published by the Health and Safety ExecutiveC1.25 12/02

ISBN 0-7176-2375-0

RR 046

780717623754£5.00 9