1

UN Road Safety Collaboration

Subgroup: Monitoring and Evaluation (M&E) of the Decade of Action (DoA) for Global Road Safety

Friday, 27 March 2015 10:45 am – 12:15 pm

Meeting Minutes

Chair: Adnan Hyder, Chairman Road Traffic Injuries Research Network and Director Johns Hopkins International Injury Research Unit

Co-Chair: Margie Peden, WHO

Meeting Aims: The meeting will:

• Review current status of Pillar indicators • Provide updates on the 3rd Global Status Report on Road Safety • Discuss approaches for developing complex indices for M&E of the DoA.

Discussion

The M&E Project Group meeting was well attended with 32 participants. After a welcome message from the Chair and introducing the purpose of the M&E Project Group, agenda items and the aim of the meeting were briefly described. The Chair reminded attendees about the process of selecting the current, reduced list of indicators from the initial, proposed list of indicators in the Global Plan of Action for the Decade of Action. He noted that the final set of indicators was selected based on the following criteria: indicators should be global level, measurable with regularly updated reliable data collected from valid and easily accessible data sources. While Pillar Working Groups may still develop additional indicators in the future, in order to be considered for the monitoring and evaluation of the Decade, the Working Groups should also suggest specific data sources for populating new, proposed indicators. The finalized list of indicators is outlined in the M&E Working Group’s Report which is published on the UNRSC website.

Work Plan

The Work Plan of the M&E Project Group for the next three years, 2015-2018 was presented. This Project Group was established to assist the UNRSC to monitor and evaluate the overall progress of the Decade of Action. The Project Group will continue to work closely with the WHO to review available data from GSRRS and UN regional commissions to inform UNRSC deliberations. The Project Group will regularly update data for the 5 Pillars and global outcomes; assist WHO with understanding and collating data from the 3rd and 4th GSRRS; explore and develop composite indices for global road safety progress measurement; and present findings and summary data reports to the UNRSC. Participants agreed that the Project Group continue its work based on the stated terms of reference, and the revised work plan for 2015-18.

Global Status Report on Road Safety (GSRRS)

Dr. Toroyan from WHO reminded attendees of the session that GSRRS is the main monitoring tool for the Decade and beyond. The 3rd report will be officially launched in Brazil during the Ministerial in November 2015. The main areas addressed by the GSRRS are:

2

institutional management (lead agency and national strategy); fatality data disaggregated by road user, age and sex, as well as trends over the last 10 years; infrastructure and exposure to risk; vehicle safety; behavioural risk factors; and post crash care. Dr. Toroyan noted that the methodology used in the 3rd Report is the same as in the previous one. For fatality data multiple data sources have been used: reported police data (through questionnaire); reported health data (vital registration data) via WHO; and the data will be modelled to generate comparative estimates for countries poor quality or incomplete data. In the 3rd GSRRS data on vehicle safety have been modified. The key vehicle safety indicators would be collected by the UNECE.

The 3rd GSRRS expanded the criteria for “comprehensive” legislation in order to have more detailed information on legislation. A legal team at WHO headquarters has been analysing legislative data. Further, the new GSRRS contains additional questions on child restraints; for example: are there restrictions within the law on children sitting in the front seat? Does law specify restraint use by age, weight, height? And do child restraints have to meet national or international standards?

Dr. Toroyan emphasized that though the model used for data analysis is the same as the 2nd GSRRS, there are changes in the key covariates in the current model (population and GNI), which means direct comparisons to GSRRS-2 will not be possible at country level on fatalities. Similarly, as the criteria have been refined and methodology has changed, comparisons on legislation should be done with caution.

At this moment preliminary analysis of fatality data and country consultations are ongoing. Analysis of other variables (infrastructure, legislation) will be done next and the proper coding/grading system for legislation data should be developed. The team is busy preparing country profiles and statistical annexes as well. The main report drafting will be taking place through April and May 2015 followed by review and drafting and final report production. An online platform will be developed for discussions from April through August. And summary reports will be ready in multiple languages by the end of October. The WHO team is working on developing one standard outline/format for country profiles. They need to decide both the amount of the information and content as well as the format for the country profiles.

Dr. Toroyan confirmed that data on permanently disabled persons were collected, though they have not been pulled out yet and analysed. She further commented that they are planning to apply the similar model to the data from the 2nd GSRRS, and will make sure that the same model is applied to data from future GSRRSs so that by the end of Decade 4 or more comparable data points are available for all indicators on global level.

Mr. Boulaajoul, CNPAC representative from Morocco admitted that quite often when government officials look at the country profiles and see different numbers for their country, they are curious about the methodology and the way it is currently described in the report is not sufficient. Dr. Peden responded that they try best to make the description of complex methodology clear for everyone, but it seems that this is not enough. And she asked, Mr. Boulaajoul, as a government official, to read the methodology again and give his feedback on how to make it more easily digestible for other officials. She also noted that the estimates are quite often based on the vital registration data. In many countries, such data are missing, or are of poor quality and they have to apply regression models to come up with the estimates. Dr. Kacem of WHO further noted, that for example, in the 2nd GSRRS Thailand government checked their data (from multiple data sources) based on the WHO estimates, and they were able to get to the same estimates themselves, and that now Thailand has been very active in reviewing and understanding the estimated numbers for 2013. Dr. Hyder suggested that creating a brief document with the examples form the countries and lessons learned and presenting them at the Ministerial Conference in Brazil.

3

Further Dr. Peden asked if presenting on data and GSRRS methodology at the coming Ministerial in Brazil would be also helpful for the countries as long as this Conference is a place where representatives of many countries will be present. Dr. Gomez of Spain said though this was a really good opportunity, from her experience, officials are rarely interested in data or data analysis. Such discussions and presentations could be included in side events and presentations.

Road Safety Indices

The Chair started discussion around the question of developing complex road safety indices. A composite index is a combination of individual indicators and it could be used to measure the multi-dimensional concept of road safety by combining information from several underlying dimensions; to attract attention of policy/decision makers and to improve public communication; to provide the overall road safety picture in order to track changes over time and effect of interventions; and to overcome the difficulty of making international comparisons of road safety performance.

Road safety indices have advantages and limitations. Advantages include for example, that a composite index can summarize complex, multi-dimensional realities with a view to supporting decision-making; they are easier to interpret and can assess progress of countries over time; can facilitate communication with general public and promote accountability and enable users to compare complex dimensions effectively. On the contrary side, an index may send misleading policy messages if poorly constructed or misinterpreted; may invite simplistic policy conclusions; and may disguise serious failings and increase the difficulty of identifying proper remedial action, if the construction process is not transparent.

Dr. Sleet of CDC suggested that before other teams start working on this, it was better that the M&E group lead the work on developing composite road safety indices. He also suggested that when constructing complex indices, choice of indicators could be different for high- versus low- and middle-income countries. Mr. Ekman from Sweden noted that these indices should be transparent and evidence based. Ms. Feypell from International Transport Forum (ITF) commented that though at ITF they had been discussing this topic for a long time, they had not been able to start the process till now. Mr. Chong from Millennium Challenge Corporation added that it was necessary to be careful why constructing these indices, so that countries with positive numbers do not manipulate and use these numbers to jump up in their country rankings. And Mr. Boulaajoul acknowledged the importance of having a good quality data for these composite indices.

Next Steps

In conclusion, the M&E Project Group identified the following agenda items for the 2015-2018 work:

• Regularly update data for indicators around the 5 Pillars and outcome indicators • Assist WHO with understanding and collecting data from the 3rd and 4th GSRRS • Prepare a briefing and key messages for Brazil Ministerial Conference in November

2015-03-28 • Explore and develop summary indicator (index) for global road safety progress

measurement • Present finding and summary report to the UNRSC.

Annexes

• Presentation of M&E group to UNRSC plenary

4

• Presentation of GSRRS-3 Update

Rapporteur:

• Dr Nino Paichadze, Johns Hopkins International Injury Research Unit, USA

Global status report on road

safety 2015Tami Toroyan

World Health Organization

Geneva

Monitoring change

• UN GA resolution 54/255 in

2010 called for a Decade of

Action for Road Safety

(2011–2020).

• Status reports to be used as

a monitoring tool for the

Decade and beyond

• 3rd report - November 2015

Areas addressed by GSRRS3

• Institutional management – lead agency,

national strategy

• Fatality data, by road user, age, sex. Trends

• Infrastructure, exposure to risk

• Vehicle safety

• Behavioural risk factors

• Post crash care

Methodology

• National data coordinator (NDC) appointed in each

country

• NDC identifies a multisectoral group of road safety

experts (Min of Health, Transport, Interior, etc)

• Each expert fills in a questionnaire

• The group then meet to discuss the data and finally

submit one final country questionnaire that best

represents their country’s road safety situation

Fatality data

Multiple sources –

Reported police data – through questionnaire

Reported health data (vital registration data) – via WHO

Modelled to generate comparative estimates

Vehicle safety

• Data collected by UNECE on key vehicle safety

indicators

• This data is not collected from national counterparts

Legislation

• Expanded the criteria for “comprehensive”

legislation in order to paint a more in depth

picture of legislation in GSRRS3.

• Data analysed by a legal team at WHO HQ and

validated with NDCs.

Additional criteria: child restraints

• Is there a national child restraint law?

• Are there restrictions within the law on

children sitting in the front seat?

• Does the law specify restraint use by: age,

weight, height?

• Do child restraints have to meet a national or

international standard?

Ex: legislative analysis, Cameroon

Points of caution

• While the model used is the same as GSRRS2,

changes to key covariates in the model

(population, GNI) mean we cannot make

direct comparisons at country level on

fatalities

• Legislation - comparisons should be made

cautiously (methodology altered, criteria

refined)

Current steps

• Preliminary analysis of fatality data and

country consultation

• Analysis of other variables – infrastructure,

legislation

• Coding /grading legislation

• Preparation of country profiles and statistical

annexes

Next steps and timeline

• Main report

– Drafting main messages (April, May)

– Review and redrafting

– Production

• Development of on line platform (for

discussion), April - Aug

• Summary reports in multiple languages

Report

Project Group on Monitoring and Evaluation

Adnan Hyder Chair

Geneva March 27, 2015

Highlights

•  Project Group report on current status of Pillar indicators and published it on the UNRSC webpage: Report from the M&E Working Group of the UNRSC

•  Continuing work on monitoring the decade •  Renewing work plan for 2015-18 •  Exploring a road safety index

Agenda for 2015-2018

•  Regularly update data for indicators for 5 Pillars and Outcome Indicators

•  Assist WHO with understanding and collating data from the 3rd and 4th GSRRS

•  Prepare a briefing and key messages for Brazil meeting Nov 2015

•  Explore and develop summary indicator (index) for global road safety progress measurement

•  Present findings and summary reports to the UNRSC

Discussion

•  GSRRS – main tool/source of data –  Will have 4 comparable data points by 2020 to evaluate the progress of

the Decade –  Apply revised modeling approach to the data from 2nd and 3rd GSRRS

to make the estimates comparable on a global level –  Update indicators with new 2013 data (GSRRS 2015) for spring 2016

•  Explore approaches for developing road safety indices –  Rationale –  Pros and Cons –  Possible approaches

•  Dr. Margaret Peden •  Dr. Tamitza Toroyan •  Dr. Nino Paichadze •  World Health Organization •  United Nations Road Safety Collaboration (UNRSC)

Acknowledgments