Indicators: Levels, Types, Existing and New

Ken Mease, University of FloridaCairo, June 2009

What is an Indicator?

• An indicator is a device for providing specific information on the state or condition of something. An indicator is also a measure, gauge, barometer, index, mark, sign, signal, guide to, standard, touchstone, yardstick, benchmark, criterion and point of reference. (Source: Oxford Dictionary)

 

Macro, Micro or Both Levels of Governance

• The differences between the macro and micro levels should be examined closely.

• At the macro level we find political institutions

• At the micro level we find the experiences, attitudes and beliefs of typical citizens. In particular the extent to which the political institutions are accepted as legitimate

Macro, Micro or Both

• Many claim that to have democratic governance, there must be a fit between these two levels.

• At the macro level there should be political institutions that conform to democratic procedures and the rule of law.

• At the micro level, an acceptance by the mass public, who in a democracy have the power to sanction or remove their leaders.

• A careful examination of the political institutions and how they are perceived by the public is needed in a comprehensive governance assessment.

De facto or De jure?• The de facto form of data refers to what

happens in practice. An example here might be to ask citizens the degree to which there is freedom of the press in a country

• The de jure form refers to the existence of formal rules. These formal written rules are often found in laws, regulations and/or the constitution. For instance, a law or constitutional provision protecting freedom of the press

De facto, De jure or Both?

• Some projects assess governance, democracy and human rights with only one form of data – either de jure or de facto.

• While other projects assess both the de jure and de facto states of governance, democracy and/or human rights.

• This is because sometimes rights or laws may exist on paper (de jure), but not in practice (de facto).

Triangulation• Governance operates on two levels, the macro

level of political institutions and the micro level of the citizens

• The ideal situation may be to combine a professional desk study that captures the de jure state of governance with de facto survey of typical citizens and other available data to provide both qualitative and quantitative data respectively

• Quantitative data provide numbers and statistics, while qualitative data can offer deeper context

An index or a scale?•  Many of the existing governance

assessment report their results as an index.

• An index is often made up of many types of data – surveys, archival data, desk studies, etc. - collected in different ways from different stakeholders .

• It is a numerical tool that represents a combination of indicators and information that can be compared over time.

Objective indicators

• Can be developed from archives or other data sources.

• De jure examples include the existence of an integrity commission, existence of a law protecting human rights, freedom of speech or some other civil liberty.

• De facto (practice) examples include the number of corruption cases prosecuted or the number of human rights violations reported.

Reported behavior or events based

• These indicators are often used in surveys of typical citizens

• They are also found from a variety of sources including government statistics, such as number of arrests, convictions, etc.

• Household surveys conducted by Central Statistical Offices often ask citizens if they have ever been asked to pay a bribe to a public official (de facto).

Subjective or Perception Based

• Often found in surveys of typical citizens or smaller surveys of key government and no-government stakeholders

• They rely on opinions or perceptions of how things are (de facto).

• Perception based data have proven to be very reliable over the years in many different contexts and cultures.

• Moreover, in many cases they are the only data available.

Proxy indicators

• Measure the subject of interest indirectly, rather that directly.

• Sometimes issues of time and money influence the need to use proxy indicators.

• Other times, researchers use proxies for subjects that are difficult to measure directly.

Good Measurement: An Overview

• Theory - when possible, broadly based theory should guide how we operationalize these concepts into testable hypotheses

• Validity - does the indicator measure what it purports to measure?

• Reliability - can the indicator be produced by different people using the same sampling, coding rules and source material?

• Clarity – make sure the questions can be understood by the population

Measurement Issues • Lack of Transparency - in the production of the

indicator • Representativeness - for survey data, what is

the nature of the sample of individuals? • Variance Truncation - the degree to which

scales force observations into indistinguishable groupings

• Aggregation Problems - for combined scores, to what degree are aggregation rules logically inconsistent or overcomplicated.

• Information Bias - what kinds of sources of information are being used?

New or Existing Indicator Checklist

– What sources of data are available for this indicator?

– What type of data will this indicator produce?

– Can the results be disaggregated?– At what level does it assess governance:

macro or micro?– What form of governance is examined: de

facto or de jure? 

New or Existing Indicator Checklist

– Will it be an objective, perception-based or a proxy indicator?

– How should it be measured: by text (qualitatively), numerically (quantitatively) or by both?

– Will it capture the local context and/or vulnerable groups?

Participatory approaches to developing indicators

• Participation in indicator identification by an expanded group of stakeholders will likely increase the amount of ownership people feel in the process and increase interest in the results.

• However, the larger the group, the longer and more costly the process is likely to become. When it comes to the more technical side of developing the indicators, a smaller skilled group is best.

 

Developing New Indicators or Not

• Developing new indicators requires skill and cost $$$.

• Many previously used indicators have been tested for reliability and validity, which can save time and money.

• Only develop new indicators where gaps exist. • Brainstorming in a meeting of stakeholders is the

easy part; meeting high professional standards for this type of work is much harder.

• Like most factors, it may come down to how much time, money and human resources are available.

Conclusions, costs and benefits concerning indicators

•  Best to try and strike a balance between “core” (existing) and “satellite” (new country- or culturally-specific) indicators

• In the recent Mongolian assessment only 11 percent of indicators used fell into the “satellite” category.

• It also is recommended that both de jure and de facto forms of governance be assessed, and the macro and micro levels of governance be measured as directly as possible. 

Exercise• Pick a governance issue in your country• Develop approximately five indicators• Identify the level – macro or micro• Identify the type of indicator

– Objective– Reported events– Perceptions– Proxy

• Identify the source of the data