five methods for measuring unobserved events

7/29/2019 Five Methods for Measuring Unobserved Events

1/38

John Whitley

Institute for Defense Analyses

Five Methods for Measuring Unobserved EventsA Case Study of Federal Law Enforcement

Improving Performance Series


2/38

John Whitley

Institute or Deense Analyses

Impi Pfmc Sis 2012

Five Methods or Measuring

Unobserved Events: A Case Study

of Federal Law Enforcement


3/38


4/38

3

FIve MethodS For MeaSurIng unobServed eventS: a CaSe Study oF Federal law enForCeMent

.sissfm.

Table of Contents

Foreword 4

Part I: Introduction 6

Purpose o this Report 6

Outline o this Report 7

Part II: Perormance Management Framework or Law Enorcement 8

Law Enorcement Outcomes 8

Supporting Perormance Measures 11

Part III: Five Methods or Estimating Data on Unobserved Events 14

The Need or a Statistical Framework 14

Five Data Estimating Methods 15

Selecting Between and Implementing the Methods 22

Part IV: Applying the Methods: A Case Study o Measuring Illegal Immigration 24

Part V: Challenges and Conclusions 28

Challenges 28

Conclusions 31

Reerences 32

Acknowledgments 33

About the Author 34

Key Contact Inormation 35


5/38

4


IbM C f t bsiss f gm

Foreword

On behal o the IBM Center or The Business o Government,

we are pleased to present this report, Five Methods for

Measuring Unobserved Events: A Case Study of Federal Law

Enforcement, by John Whitley

Measuring program perormance is relatively straightorward in

many areas o government, such as social services, visa pro-

cessing, and air traic control But in some instances, assessingperormance and success is much harder One particularly dii-

cult area involves law enorcement, where a key goal is to pre-

vent or deter bad outcomeswhich can oten happen without

the knowledge o law enorcement oicials

This report describes the challenges o measuring unobserved

events such as tax cheating, drug smuggling, or illegal immigra-

tion These dier rom inrequent events (another measurement

area where success is diicult to judge), such as terrorist attacks

The author, John Whitley, is a ormer executive in the Department

o Homeland Security, where he aced these challenges daily Hereviews practical methods already in use in some government

and law enorcement agencies to estimate unobserved ederal

crime rates and ways to measure them He notes that applying

these methods more broadly at the ederal level could enable

the types o perormance management reorms that have revolu-

tionized local law enorcement

Whitley describes ive data estimation methods: the use o

administrative records; surveys; inspections, investigations,

and audits; experimental methods; and technical measurement

He provides examples o how these are already being pioneered

in areas as diverse as determining the amount o countereitmoney in circulation, the underpayment o taxes, and credit card

raud He shows how having this kind o inormation helps law

enorcement managers develop appropriate data-driven strategies

and approaches or countering these kinds o crimes

A key insight oered by Dr Whitley is that there is oten insti-

tutional resistance to diverting scarce resources rom enorcement

to data collection and measurement But he says that when

law enorcement leaders do make the leap, they quickly become

Daniel J Chenok

Gregory J Greben


6/38

5


.sissfm.

champions o the use o data Local police departments across

the country have begun to use these kinds o techniques to

undertake what has become known as predictive policing

moving police oicers to places where crime is predicted to

occur, rather than only responding to crimes that have occurred

This is happening in cities including Los Angeles, Pasadena,

and Charleston

This report is one in a series by the IBM Center exploring more

sophisticated approaches to analyzing and using perormance

inormation Other reports on the use o analytics include:

Empirically Based Intelligence Management: Using

Operations Research to Improve Programmatic Decision-

Making by Frank Strickland and Chris Whitlock

From Data to Decisions: Building an Analytics Culture by

the Partnership or Public Service

From Data to Decisions: The Power of Analytics by the

Partnership or Public Service

Strategic Use of Analytics in Government by Sirkka

Jarvenpaa and Thomas Davenport

We hope this report helps ederal law enorcement leaders and

other government managers to tackle the tough job o using

sophisticated approaches to improve program perormance in

hard-to-measure subject areas

Daniel J Chenok

Executive Director

IBM Center or The Business o Government

chenokd@usibmcom

Gregory J Greben

Vice President, Public Sector

Business Analytics & Optimization

IBM Global Business Services

greggreben@usibmcom


7/38

6



Purpose of this Report

In recent decades the local law enorcement community has been a pioneer in measuring and

reporting perormance and in using these data to drive strategy development and manage exe-

cution New York Citys CompStat1revolution ocused commanders on crime trends with the

same hawk-like attention private corporations pa[y to] proits and losses Crime statistics have

become the [police] departments bottom line, the best indicator o how police are doing pre-

cinct by precinct and citywide (Smith and Bratton 2001) Although scholars are still debatingthe relative contribution o perormance-driven management reorm in the dramatic crime rate

decline over the last 20 years, these policing reorms have likely played a signiicant role and

have been emulated around the country in areas well beyond law enorcement

Law enorcement at the ederal level, however, has been slower to adopt these reorms, and

many areas o ederal law enorcement do not systematically collect, use, or report basic data

on crime rates within their jurisdictions One o the most diicult challenges contributing to

this lack o outcome-oriented, data-driven management is how to measure the level o many

ederal crimes2 Federal crimes like drug smuggling and income tax evasion oten go unde-

tectedailing to leave a trail o administrative records that produce perormance data on

crime rates The perpetrators o these crimes preer not to get caught, and no directly aected

victims exist with the incentive to notiy law enorcement oicials Federal law enorcement

organizations are let with the challenge o how to measure unobserved events

These measurement challenges are not unique to ederal crimes No criminal wants to be

apprehended State and local law enorcement organizations struggle with the accurate

measurement o crimes like rape, drug use, and prostitution within their jurisdictionseven

murder can be surprisingly hard to measure Similarly, other ederal organizations deal with

unobserved events in many aspects o their missions as well

The purpose o this report is to review practical techniques, many already in use in other gov-

ernment and law enorcement areas, that can be used to estimate unobserved ederal crime

rates and related perormance measures The intended audience includes ederal senior execu-

tives responsible or managing and allocating scarce resources within law enorcement, along

with perormance and program management oicials tasked with estimating and reportingthese measures Applied to ederal law enorcement, the techniques reviewed in this report

could enable the types o perormance management reorms that have revolutionized local law

enorcement The discussion o these techniques may also be useul in other settings where

unobserved events hinder perormance management

1. Crime Statistics, Website of the New York City Police Department.

http://www.nyc.gov/html/nypd/html/crime_prevention/crime_statistics.shtml.

2. A crime is an offense against a public law. In its most general sense, it includes all offenses. It can also be used in a more limited

sense to only include felony violations of public law. This report uses the word crime in its more general sense to include all types of

offenses, whether felonies or not.

Part I: Introduction


8/38

7


.sissfm.

Outline of this Report

Part II o this report reviews the basic perormance management ramework or law enorce-

ment activities It uses examples to:

Identiy specifc, key outcome perormance measures essential or sound management

The challenges in estimating perormance

The problems created when non-outcome measure proxies are used

Part III systematically reviews techniques and methods that can be used to estimate unob-

served variables This includes empirical analysis o administrative records, surveys, and

covert testing

Part IV applies these techniques to speciic examples in ederal law enorcement to demon-

strate how they can be used in practice

Part V outlines selected major challenges with data quality and interpretation that arise in esti-

mating outcome perormance measures based on unobserved events, and includes concluding

remarks


9/38

8



Understanding an organizations perormance measurement challenges and developing a plan

to overcome them begins with identiying appropriate perormance measures This process,

oten integral to strategic planning, begins with identiying the outcomes the organization is

trying to achieve or inluence For law enorcement organizations, the clear and authoritative

place to start is in the laws the organization is tasked with enorcing

This section deines the primary outcome perormance measures or law enorcement and then

briely examines the output, input, and eiciency measures that can be used to eectivelyimplement strategies or the achievement o those outcomes

Law Enforcement Outcomes

Crime rates, the rates at which laws are broken, are the primary outcomes and outcome per-

ormance measures or law enorcement organizations At the local level, eg, New York City,

the outcome measures or which the chie o police is accountable are the rates o murder,

rape, aggravated assault, burglary, etc City oicials above the police chie have broader objec-

tives; or example, in addition to crime, the mayor must ocus on the standard o living and

economic growth o the city Oicials subordinate to the police chie may have narrower objec-

tives; or example, the manager o the 911 call system may be ocused on the response time

to emergency calls

In New York City, the police department measures these outcomes and uses the data to or-

mulate policy objectives, develop and evaluate strategies, monitor execution, and ensure

accountability (see OConnell 2001 and Smith and Bratton 2001 or detailed descriptions o

the New York City Police Departments use o perormance data and its CompStat system)

Figure 1 illustrates a typical publicly available weekly report rom the New York City Police

Department This level o transparency with the public has a direct impact on the public

debate about crime and expenditures on policing in New York, ocusing the debate as much

as possible on actual crime rates and how they change as policies and resources are adjusted

The primary outcomes and, thereby, outcome perormance measures or ederal law enorce-

ment organizations are the rates at which the ederal laws aecting their jurisdictions are bro-

ken For example, the ederal government enorces US immigration law, and a primary

outcome measure is the rate at which individuals enter and reside in the country unlawully

Enorcement or most o these laws is the responsibility o the Department o Homeland

Security (DHS) and its components:

Immigration and Customs Enorcement (ICE)

Customs and Border Protection (CBP)

United States Coast Guard (USCG)

Part II: Performance ManagementFramework for Law Enforcement


10/38

9


.sissfm.

The senior oicials in these organizations are responsible or enorcing US immigration laws

Among the outcomes or which they should be held accountable are the rates at which US

immigration laws are broken The President, and members o White House policy councils,may also have broader objectives, such as economic growth and security, which are aected

by violations o immigration law Subordinate program oices within CBP, ICE, and USCG may

also have narrower objectives, such as situational awareness within a particular border

domain and the time it takes to process a detained illegal immigrant or deportation

Unlike robbery and assault, which leave aggrieved victims with an incentive to notiy the

police and can involve abundant physical evidence, violations o immigration laws are largely

unobservable to ederal law enorcement oicials unless the perpetrators are apprehended in

the act Estimating the rate o illegal immigration is essential or data-driven management in

0LFKDHO5%ORRPEHUJ0D\RU

3ROLFH'HSDUWPHQW

&LW\RI1HZ


11/38

10



immigration law enorcement Improving the availability o objective and transparent peror-

mance outcome data would improve management and decision-making in areas like:

Setting clearer policy objectives: Many areas o ederal law enorcement, like border

control and drug trafcking regulation, are politically contentious Providing objective data

on outcomes can help ocus debate on choices between act-based alternatives, encourag-

ing the establishment o actual objectives or, at least, the concrete identifcation o where

agreement has been reached on objectives and where it has not

Selecting the right strategies: Because o the political nature o the issues and the ailure

to arrive at clear objectives, choices among strategies oten become contentious in ederal

law enorcement For example, policy-makers may choose the number o border patrol

agents and the types o employment immigration enorcement based on a desire to

demonstrate strength, rather than rigorous empirical evidence o the strategys eective-

ness when compared to alternative enorcement tools Objective measurement o outcomes

and the perormance levels o alternative strategies help to ocus the selection o strategies

on their actual contributions to outcomes instead o on the political signals they send

Managing execution: With clear objectives and an assessment o how strategies are

working, adjustments in execution can be deended politically by providing evidence to

support those adjustments

Ensuring accountability: When agencies and agents are given clear objectives and there

are measurement strategies in place to determine i the objectives are achieved, execution

can be decentralized and accountability maintained by measuring results produced instead

o inputs expended, eg, budget spent or number o agents deployed Congress and the

Administration may be more willing to ocus on setting objectives and evaluating strategies,

rather than micromanaging execution, because they know that management ofcials can

be held accountable or the resources they have been provided

When outcomes are hard to measure, proxy variables are oten used, but they can be ineec-

tive or, worse, counterproductive A common proxy measure in law enorcement is the number

o arrests or various crimes But as Washington, DC Police Chie Cathy Lanier said recently,

Arresting people is not a measure o success Less crime is a measure o success (Goldberg

2012) Arrests are an important element in the production o law enorcement that should be

measured, but they are not the outcome, and they cause conusion when they are reported

and used as such In immigration law enorcement, one equivalent proxy measure is the num-

ber o apprehensions o illegal immigrants attempting to cross the border But, as reported in a

recent study, CBP [attributed] increases in apprehensions made at checkpoints in some bor-

der sectors to improved operations and decreases in apprehensions in other sectors to the

deterrent eects o improved CBP technologies and increased staing Clearly, a measure that

relects successul perormance whether it rises or alls has limited value as a management

tool (RAND 2011)

In addition to reducing accountability and ailing to provide management with useul tools,

measures that do not act as a true proxy or outcomes can ocus attention and resources onthe wrong things This becomes counterproductive to achieving the desired outcomes Former

Mayor Rudolph Giuliani has stated that, in New York City, he ound that ocusing on arrest

rates actually distracted police oicers rom a ocus on crime:

We were equating success with how many arrests were made A police oicer was

regarded as a productive police oicer i he made a lot o arrests He would get pro-

moted A police commander in a precinct would be regarded as a really good police

commander i his arrests were up this year This wasnt the only measure o success,

but it was the predominant one


12/38

11


.sissfm.

Arrests, however, are not the ultimate goal o police departments or what the public

really wants rom a police department What the public wants rom a police depart-

ment is less crime So it seemed to me that i we put our ocus on crime reduction

and measured it as clearly as we possibly could, everybody would start thinking about

how we could reduce crime And as a result, we started getting better solutions rom

precinct commanders (Giuliani 2000)

Producing, reporting, and explaining proxy measures also consume resources, making those

resources unavailable or use in measuring actual perormance outcomes In addition to

reporting apprehensions, DHS reported the measure Border Miles UnderEective Control or

several years In 2006, a multi-billion-dollar, ive-year investment plan was launched or the

southwest land border, and DHS intended to use the measure to evaluate the eects o the

plan; however, the measure was never objectively deined By not capturing meaningul out-

comes, this measure introduced conusion and unnecessary contention into the public debate

instead o grounding debate in acts and evidence

The measure was ultimately abandoned, leaving DHS with limited ability to evaluate the eec-

tiveness o the border buildup or make subsequent decisions about how to execute the initia-

tive The likely costs o the types o measurement proposed in this report would not exceed,and could even be less than, the amount DHS spent on the Eective Control measure DHS

has stated that it is now creating a new set o proxy measures that will be reported as an

index The resources being committed to this eort might be better spent trying to measure

actual outcomes o interest

Supporting Performance Measures

Although the ocus o this report is primarily on measuring unobserved law enorcement out-

comes, a complete perormance management ramework requires measurement o more than

just outcomes Law enorcement activities aect crime rates in a variety o ways To eectively

manage activities so as to inluence outcomes, law enorcement oicials need to understand

and measure how their activities aect outcomes In addition to outcome perormance mea-sures, a law enorcement perormance management ramework needs to include:

Measurement o outputs, inputs, and efciency

Empirical analysis on how these measures relate to each other and to outcomes

Law enorcement activities reduce crime in two primary ways:

Stopping crimes (during planning or in the act)

Deterring crimes rom occurring in the frst place

In much o law enorcement, the primary method o reducing crime is through deterrence, as

ew crimes are actually stopped in the act Federal law enorcement, however, oers some

examples where stopping a crime in the act is a primary role o law enorcement, eg, terror-

ism and drug smuggling Deterrence is oten urther broken down into two elements:

Specifc deterrence ocuses on an individual who is committing a crime; arresting and

punishing the individual may prevent a repeat o the oense

General or indirect deterrence reers to the general prevention o crime by deterring

potential criminals rom committing a crime in the frst place

For example, in border law enorcement, these elements have been called at-the-borderdeter-

rencepreventing an illegal immigrant rom repeating an attempted border crossingand


13/38

12



behind-the-borderdeterrencestopping a potential illegal immigrant rom making the initial

decision to attempt migration and traveling to the border This deterrence is produced most

directly by arresting and punishing oenders, and letting other potential oenders know it will

happen to them

This understanding helps to identiy the irst level o supporting measures and analyses

needed or data-driven management in law enorcement: The number o arrests and the subsequent arrest rate (arrests per crime committed)

The rates at which these individuals are charged and convicted3

The punishments administered

A key analysis at this level is the relationship between these variables (collectively identiied

as the expected consequence o committing the crime) and the level o crime, ie, the level o

deterrence created In areas o law enorcement with a high volume o observed crime that

responds quickly to changing circumstances, this analysis might be done in real time through

varying enorcement eorts In other situations, such as illegal immigration, it may require

detailed empirical studies conducted over long periods o time This analysis can be challeng-

ing and may involve disentangling interrelated eectseg, the eect o law enorcement andeconomic changes on illegal immigration ratesbut is necessary or eective management

At this level o supporting measures, there may be other important issues that also need to be

considered For example, in law enorcement related to drug use, rehabilitation may be con-

sidered a substitutable (or complementary) consequence and, in this case, would also have to

be measured Analysis would then be used to determine its eectiveness at preventing crime

as well as its eectiveness relative to punishment

The next level o supporting measures and analyses identiies how the key outputs are pro-

duced Arrests are produced by intelligence gathering, patrolling, investigating, and numerous

other activities Identiying the available tools and their relative contributions to making arrests

is necessary or optimally allocating resources to eiciently increase arrest rates Charges andconvictions are achieved through urther investigation, evidence gathering, and legal analysis

Punishment consists o ines, incarceration, and other legal sanctions

An eective perormance management ramework should be built around a detailed under-

standing o how law enorcement activities contribute to the outcomes the oicials are trying

3. For some areas of federal law enforcement, e.g., border control, the apprehension rate is equivalent to the rate for both arresting

and charging. In general law enforcement, this is sometimes called the clearance rate.

The Need or More Academic Research

Extensive academic literature in economics and criminology empirically measures the effects ofarrests, prosecutions, and punishments on crime rates. This literature helps law enforcement offi-cials and policy-makers prioritize scarce enforcement resources, but much of the available literatureis focused on crimes of state and local primary jurisdiction. The lack of analysis on crimes of pri-marily federal jurisdiction is driven in part by the lack of systematically reported data. Improved per-formance reporting in federal law enforcement could help create an academic literature that wouldassist federal decision-makers in effectively prioritizing investments and managing law enforcementfunctions.


14/38

13


.sissfm.

to achieve This requires both measuring these activities and analyzing their relationships to

each other and to outcomes Some o this analysis may be possible to do inormally during

execution o actual operations, and some may require more detailed empirical analysis, but

eective stewardship o taxpayer resources requires that it be done


15/38

14



Tax evasion or illegal drug smuggling are oten not observable events or ederal law enorce-

ment oicials But to eectively manage ederal law enorcement activities, oicials and pol-

icy-makers in charge must have an idea o what is happening Five methods that can assist

government perormance analysts in estimating basic inormation on unobserved events are

introduced and described here

The Need for a Statistical FrameworkLaw enorcement can ace tough measurement challenges, but the ields o statistics and

econometrics have developed a ramework or dealing with them and it is useul to begin this

part with a brie overview o that ramework All violations o a ederal law can be thought o

as elements o a prospective data population The scope o the population can be deined in

various wayseg, immigrants illegally entering the United States in a calendar year, or the

illegal drugs smuggled across the southwest land border between the United States and

Mexico To eectively manage their operations, ederal law enorcement oicials need insight

into these unobserved violations; ie, they need to know the properties or parameters o this

population o data, such as its size and distribution

Law enorcement oicials are generally able to observe subsets, or samples, o this popula-

tion The most obvious is the subset o violators apprehended or arrested Detailed documen-

tation o apprehensions or arrests is generally retained in administrative records In addition,

there may be other available sources o data, oten partial and incomplete, that shed light on

various aspects o the population, eg, survey data on drug usage or the ootprints in the

desert o illegal border-crossers Actions can also be taken to increase the available data,

such as increasing the size o the observable subset, drawing additional samples rom the

population, or generating a sample o new data that mimics the characteristics o the popula-

tion o interest The methods described here use such samples to make estimates o the total

population

When using a sample to estimate parameters o the underlying (unobserved) population, an

important statistical property is whether the estimate is biased Bias occurs when the estimate

systematically diverges rom the true value o the population parameter being estimated An

unbiased and thereore preerred estimate does not systematically diverge rom the true value

One primary cause o bias is a poor sample that is not randomly selected A sample is random

when every element o the population has an equal probability o being included Examples o

non-random samples may include:

Records on individuals apprehended smuggling drugs across the border These records may

not be representative o all individuals who attempt to smuggle drugs across the border i

slower, less prepared individuals are more likely to be caught

Part III: Five Methods for EstimatingData on Unobserved Events


16/38

15


.sissfm.

Survey data on the propensity to illegally migrate to the United States collected rom urban

Mexican households These data may not be representative o the propensity o all Mexicans,

both rural and urban, to illegally migrate

In these cases, estimates o population parameters made rom the sample data may be biased

and thus misleading It is important or government managers who develop perormance mea-

sures to be constantly vigilant or bias in their estimates It may not be possible to eliminateall potential biases in data, but the analyst must be aware o the major potential biases in

their data and their possible eects

A inal note on the need or a statistical ramework in the area o law enorcement: although

oten related, the challenges o measuring unobserved events are dierent rom the challenges

associated with infrequent events The Department o Deense prepares to ight wars, but or-

tunately these are very inrequent DHS prepares or a nuclear attack on a major US city, but

ortunately this has never happened Measuring the perormance o military capability in a war

ight when there is no war ight or the perormance o response and recovery capabilities or a

terrorist nuclear attack when there has never been one are very important, but are not the

ocus o this report

Five Data Estimating Methods

Method One: Administrative records. Once a perormance manager has identiied theout-

comes that need to be measured and is beginning the task o developing a measurement

strategy, the irst action is to identiy all relevant data currently captured by the agency or by

others In the best-case scenario, the perormance manager may discover relevant data at a

lower level in the organization (eg, at the ield oices) or in another organization (eg, in a

survey conducted by the Census Bureau that asks a pertinent question)

Or it could be that estimation o the outcome is possible, but that multiple sources o data have

to be combined and those sources are spread across organizations For example, Immigration

and Customs Enorcement is responsible or law enorcement concerning individuals who enterthe United States on visas, but violate the visa by overstaying the required departure date The

rate o visa overstay, however, is unobservable to ederal law enorcement The number o visas

issued and their required departure dates are known, but who actually departs and when is

not Everyone leaving the United States by commercial air or maritime transport is known

because they are identiied in passenger maniest documents maintained by the transportation

companies4 Thus, a major portion o the perormance measure can be estimated by combining

data rom government visa records with commercial transportation passenger maniests

Inventorying available data is a valuable exercise even when it does not identiy a previously

unrealized, accessible method o estimation, because it may identiy ways to acilitate estima-

tion by enhancing existing data collection Examples might include the addition o another

inormation ield into a case management IT system or the addition o a question to an exist-ing government survey In these cases, estimation is possible but the required data are simply

not being captured

4. These outbound passenger manifests are an incomplete data source, however, because they are not available for land border exits

and may not be complete for non-commercial air and maritime exit. Therefore, although a major set of data that can be used in comput-

ing a visa overstay rate is available from existing data spread across multiple sources (some of which are outside of the government), this

illustrates another important pointit may be necessary to combine methods to develop complete measures. In this case, the use of out-

bound passenger manifests may have to be combined with techniques identified below to develop a complete visa overstay performance

measure.


17/38

16



Another potentially useul result rom inventorying the available data is to identiy dataor

new data that could be generatedthat could, with the use o clever empirical methods, sup-

port analyses that would estimate the outcomes o interest In the Recidivism Analysis box, a

method or using apprehension data to iner an apprehension rate, and subsequently the rate

o illegal immigration, is presented

Method Two: Surveys. Surveys area commonly used data collection method in policy and

social science research Surveys involve asking a set o questions to a sample population

They can be conducted by telephone or mail, online, or in person The goal is to obtain a

sample o suicient quality, eg, size and representation, to enable inerences to be drawn

about the population rom analysis o the data Surveys may be conducted on a regular,

recurring basis to create estimates through time or can be conducted on a one-time basis to

answer speciic questions at a point in time

There are numerous surveys already being conducted by the government and private organiza-

tions that provide valuable inormation on ederal law enorcement issues The US Bureau o

the Census and its many supporting surveys provide some o the most comprehensive data

about the United States Other ederal agencies conduct a wide range o surveys that include

speciic emphasis on law enorcement issues, such as the National Crime Victimization Survey

(NCVS) discussed in the National Crime Victimization Survey box that ollows Surveys are

Example o Using Administrative RecordsRecidivism Analysis

The U.S. Border Patrol (USBP), within the DHS Customs andBorder Protection, is responsible for controlling the U.S. land

border with Mexico between the Ports of Entry (POEs). USBPmaintains a detailed database called ENFORCE on all appre-hensions of illegal border-crossers. When illegal border-cross-

ers from Mexico are apprehended and returned to Mexico,many try again within a relatively short period of time. In fact,if all who were returned attempted to cross again, the fractionapprehended a subsequent time would constitute an estimateof the apprehension rate. With an estimated apprehensionrate in hand, it is then possible to estimate the flow of illegalborder-crossers.

A challenge with this approach is that every individual who is apprehended and returned does notattempt a subsequent crossingthe first apprehension acts as a deterrence (referred to as at-the-border deterrence in the earlier discussion on supporting measures). This means that recidivism

analysis by itself does not solve measurement challenges, but it can provide an important part ofthe solution in situations where it is appropriate. If it can be combined with estimates from othersources on the deterrence effect of apprehensions, it can be used to create an estimate of appre-hension rate and, subsequently, the rate of illegal immigration.

In the absence of data from other sources on the deterrence effect of an apprehension, an additionaltechnique that can be used to increase the usefulness of recidivism analysis is sensitivity analysis.The performance manager could estimate the implied apprehension rate from the recidivism data,assuming different levels of deterrence. The analysis could start with zero percent deterrence (theassumption in the motivating example above) and then consider 10 percent, 20 percent, etc.,where 10-percent deterrence means that 10 percent of individuals apprehended at the border aredeterred from making a subsequent attempt. This analysis would provide a range of possible esti-mates depending on the level of deterrence.


18/38

17


.sissfm.

also conducted by academic researchers, think tanks, and private companies In some situa-

tions, there may already be a recurring survey conducted that is close to, but not exactly, what

the perormance manager needs; a cost-eective way to get started is to partner with the

organization conducting the existing survey to expand it in a way that would be useul or the

law enorcement perormance measurement

There are, o course, challenges and limitations to surveys It can be costly to implement a

new survey that contacts enough individuals to be statistically valid It can also be challengingto elicit truthul answers, particularly on issues o interest to law enorcement like criminal

activity There are generally ew consequences or responding untruthully on a survey, and on

questions related to criminal activity, ear o how the inormation will be used may be a strong

incentive to lie A great deal o work has been done to structure questions and surveys to elicit

truthul answers and test or inconsistencies within a survey response, and the results allow

this risk to be mitigated, but it remains an issue when considering the use o surveys

When the decision has been made to urther explore use o a survey to address a measure-

ment challenge, important actors that will have to be addressed include the survey methods

(eg, telephone, one-on-one), the sampling methodology (eg, random drawing o names, tar-

geting o particular groups), question design (must be unambiguous, must allow estimation o

the desired measure), question sequencing, starting with a pilot or the ull survey, and howthe analysis will be conducted once the survey is complete Numerous proessional irms that

conduct surveys and academic sources o inormation can be explored urther

Method Three: Inspections, Investigations, and Audits. Criminal or administrative investiga-

tionsoer another way to systematically collect an accurate data sample The important point

about using investigations in the context o measuring unobserved events is that the investiga-

tions must be in some way random In typical law enorcement operations, proactive investi-

gations are prioritized to ollow the most important clues or those that are most likely to lead

to a major arrest or disruption o crime Investigations prioritized in this manner may not provide

statistically valid estimates o the underlying level o criminal activity Conducting investigations

Example o Using SurveysNational Crime Victimization Survey (NCVS)

Many state and local crimes leave aggrieved victims andphysical evidence and are more likely to be reported to law

enforcement officials than some federal crimes are. But thisreporting is still not perfect, and the records of these reportedcrimes may not represent the full extent of the crime commit-

ted. The NCVS is one method law enforcement officials use tounderstand and measure potential undercounting.

According to its official website, the NCVS surveys a nation-

ally representative sample of about 40,000 households oncriminal victimization in the United States. Each householdis interviewed twice during the year. The data are then usedto estimate the likelihood of victimization by rape, sexual assault, robbery, assault, theft, householdburglary, and motor vehicle theft for the population as a whole as well as for segments of the popu-lation such as women, the elderly, members of various racial groups, city dwellers, or other groups.

More information is on the NCVS is available from the Department of Justice at http://bjs.ojp.usdoj.gov/index.cfm?ty=dcdetail&iid=245.


19/38

18



on a more random sample o potential illegal activity represents a major cultural shit or law

enorcement operations, but limited and systematic use o them can be a powerul way to col-

lect inormation about the outcomes the law enorcement organization is trying to eect See

the National Research Program and theAdministrative Site Visit and Verification Program

text boxes or examples o this method

Method Four: Experimental Methods. Another method involves actually adding or modiying

law enorcement activities in the ield in ways that may acilitate estimation o the crime rateIn controlled environments like Ports o Entry or airport security screening, this could involve

selecting a randomized subset o individuals who pass the primary screen or a secondary,

more rigorous screen The rate at which violations are identiied in the secondary screen can

be used to iner the ailure rate o the primary screen The Randomized Secondary Screening

text box describes how CBP conducts these randomized secondary inspections at Ports o

Entry This method is not restricted to physical screeningapplication processing and other

orms o inormation-based screening can also have randomized secondary evaluations con-

ducted to evaluate the accuracy o the primary screening process

Example o Using AuditsNational Research Program

The Internal Revenue Service (IRS) within the Departmentof the Treasury is the nations tax collection agency and

administers the Internal Revenue Code. The tax gap is theIRSs measure of tax liability that is not paid on time. The IRSNational Research Program (NRP) measures the tax gap using

randomized audits.

Most IRS tax audits are targeted to those tax returns for whichthere is suspicion of non-compliance. These audits cannot be

used to develop an estimate of overall compliance becausethey are not a representative sample of all tax returns. TheNRP, therefore, conducts audits on a random set of tax returnsto develop an unbiased estimate.

The NRP originally drew samples every few years of about 45,000 individuals. In 2007, itswitched to an annual sampling of 13,000 individuals. This allows the IRS to make more frequent

estimates and more accurately monitor trends. The majority of individuals selected will have theirtax returns confirmed through in-person audits with an IRS examiner. The IRS will also use match-ing and third-party data to confirm the accuracy of the tax returns.

In addition to measuring the tax gap, the IRS also uses the NRP to update and improve the accu-racy of its audit selection tools. By identifying the correlations between reported data and non-compliance in the randomized sample of tax returns, the IRS is able to refine the set of factors itlooks for in all tax returns to prioritize auditing. In other words, collecting data and measuring per-formance not only help in strategic analysis such as evaluating the level of investment the UnitedStates should make in tax enforcementthey also help optimize the range of tools the IRS employsto ensure tax law compliance.

Another important thing to note about the NRP is that randomized audits only measure compliance

for individuals who actually file a tax return. Another form of noncompliance is to not file a returnat all. The NRP uses surveys and other tools to measure this form of noncompliance. This providesanother example of multiple methods being combined to develop a complete estimate.


20/38

19


.sissfm.

Example o Using InspectionsAdministrative Site Visit and Veriication Program (ASVVP)

The U.S. Citizenship and Immigration Service (USCIS) withinDHS is the government organization that oversees lawful

immigration to the United States. Detecting and preventingthe fraudulent obtainment of an immigration or citizenshipbenefit is an important area of federal law enforcement.

An example of such fraud would be the establishment of aphony religious organization created to sponsor individuals forimmigration using a religious worker visa. This was a particu-lar concern following the terrorist attacks of September 11,2001.

The USCIS program ASVVP uses systematic site inspections to verify information contained in visapetitions. Site inspections are unannounced and can be conducted pre- and post-adjudication.When potential fraud is identified, the evidence is turned over to law enforcement officials for poten-tial criminal investigation.

Significantly, the site inspections are not performed only where fraud is suspected, but are randomlyselected so that a sample of all petitions is created and an unbiased estimate of fraud can be per-formed. USCIS uses this program to estimate performance measures for fraud and reports them inthe DHS Annual Performance Report.

Example o Using Field ExperimentsRandomized Secondary Screening

The Office of Field Operations within U.S. Customs and BorderProtection (CBP) is responsible for screening all individualsentering the United States at Ports of Entry. With 340 millionindividuals entering the United States at these ports per year,this is a high-volume process. All individuals are subjectedto a primary screening procedure to ensure compliance withU.S. entry law.

To empirically estimate the failure rate of the primary screen-ing process, CBP randomly selects a sample of the entrantsat both air and land ports to conduct a more thoroughexamination for major violations. Major violations involveserious criminal activity, such as possession of narcotics, smuggling of prohibited products, human

smuggling, weapons possession, fraudulent U.S. documents, and other offenses serious enough toresult in arrest. For the air domain, passengers are selected in a random sample that totals 12,000passengers annually (1,000 passengers per month) at each of the 19 largest international airports.Similarly, for the land domain, passengers are selected in a random sample that totals 12,000 pas-sengers annually (1,000 passengers per month) at each of the 25 largest land border ports. Thesesample sizes were selected to obtain an overall 95-percent confidence level in the estimates.


21/38

20



Another example might be the randomized surge o enorcement activity across dierent geo-

graphic regions The US Border Patrol could conduct an unannounced surge o agents into a

randomly selected station and measure the increase in drug apprehensions Conducting these

random surges across a range o locations and circumstances could be used to iner inorma-

tion about the population o oenders who are evading detection and apprehension during

non-surge periods In addition to perormance measurement, these random surges can also

provide a deterrent eect to crime and have been used throughout law enorcement Forexample, the Amtrak Police Department uses random and unpredictable surges as part o its

law enorcement strategy

Another category o experimental methods is to conduct controlled tests With this method,

the organization is not collecting data about the underlying population; it is generating a new

data population in a controlled way so that this new population has the same properties as

the population o interest Perhaps the most prominent method in this category is red-team-

ing, also called covert testing or penetration testing A red team is an independent group that

seeks to challenge an organization or the purpose o detecting vulnerabilities The Red-

Teaming text box describes the use o this method in aviation security, where penetration tes-

ters physically test security Its most obvious application to law enorcement perormance

measurement is in situations where there is a controlled environment such as inbound passen-ger screening at POEs In less controlled environments, eg, drug smuggling routes along the

border, there are obviously saety issues or the testers that would have to be seriously evalu-

ated beore considering this method

Red-teaming does not have to be physical penetration testing, however Many ederal law

enorcement organizations and their counterpart beneit-delivery organizations (eg, the

Example o Using Experimental MethodsRed-Teaming

Red-teaming, also known as covert testing, is done through-out the federal government; examples range from theGovernment Accountability Office to the National NuclearSecurity Administration. The Transportation SecurityAdministration (TSA), within DHS, conducts systematic red-teaming of passenger and baggage screening in aviationtravel. Some of this red-teaming is done to explore new tech-niques that terrorists might use to identify gaps and vulnera-bilities in current systems. More important for purposes of thisreport, however, is the other portion of TSA red-teaming thatis done explicitly for the measurement of screening perfor-mancethe failure rate in detecting threats such as firearms,knives, Improvised Explosive Devices, and emerging threats.

This performance testing is part of TSAs Aviation Screening Assessment Program (ASAP). Althoughthe actual operational details of this program are classified, TSA has released to the public somebasic information. Each airport receives a prescribed number of assessments that they are required

to conduct within a six-month cycle. The tests are unannounced and conducted surreptitiously; i.e.,they are covert tests. The number of tests conducted is large enough to ensure that the sample col-lected can yield statistically significant estimates of the failure rate. Test result data are standardizedand maintained in a centralized database to facilitate performance measure estimation and otherempirical analysis. Through this program, TSA is able to compute statistically valid estimates of itsfailure rates in both passenger and baggage screening.


22/38

21


.sissfm.

USCIS) engage in inormation-based screening as well as physical screening In an inormation-

based screening context, where data are validated to ensure compliance with the law (eg,

visa and citizenship application processing), red-teaming could entail the systematic iling o

phony applications to identiy ailure rates o the screening process These methods are similar

to mystery shopper testing done by the private sector to test service in retail outlets

Red-teaming aims to conduct controlled tests, but is perormed directly in the ield (covertly)with the operational activities as they execute their daily mission An even more controlled test

is one that takes place in an artiicial environment; eg, a laboratory or range test This is

done routinely across the government in acquisition programs where both development and

operational test and evaluation are standard procedures in the procurement o newly designed

complicated government hardware such as deense weapon systems It can also be used as

part o a systematic perormance measurement process

For example, the US Border Patrol uses sensors and radars along the border to detect illegal

border crossing Extensive testing is perormed on these items during procurement to under-

stand their alse negative and positive rates; that is, the rate at which they miss inding an item

o interest and the rate at which they report inding something that is not o interest Although

actual ield conditions will dier rom these controlled conditions, it may be possible to use theresults o these controlled tests to iner likely characteristics o the system in the ield

Method Five: Technical Measurement. Although there are many more methods that can be

used, the inal method described here is technical data collection Well-known examples at

Example o Using Technical MeasurementCountereit Detection

The original mission of the U.S. Secret Service, now withinDHS, was to investigate counterfeiting of U.S. currency.Although presidential protection was later added and is nowwhat the Secret Service may be best known for, the Secret

Service remains the primary law enforcement organization oncounterfeiting. Working with the Federal Reserve Board (FRB),the Secret Service is able to estimate the level of counterfeitcurrency in circulation in part through technical measurement.

Most currency in circulation gets deposited with financial insti-tutions relatively frequently throughout its circulation life. Forsafekeeping, space, and financial reasons, these financial insti-tutions send much of this currency to the Federal Reserve Bank for the district in which the financialinstitution resides. The Federal Reserve Banks then use high-speed currency processing machines toverify the deposits, for both authenticity (counterfeit) and fitness (worn or damaged). For example,

the Federal Reserve Bank of New York processes more than 19 million notes each business day.Unfit notes are destroyed (the Federal Reserve Bank of New York destroys approximately five millionnotes per business day) and counterfeit notes are turned over to the Secret Service.

From this, the FRB can compute a counterfeit rate per one million notesprocessed. This is nota complete measure, however, because counterfeiting, particularly lower quality counterfeiting, isoften detected before the currency makes it to the Federal Reserve Banks. To control for this, theSecret Service combines the counterfeit notes detected by the FRB with notes turned in from othersources and law enforcement activity. The Secret Service then uses this total amount of counterfeitpassed to compute a measure of the percentage of counterfeit as compared to genuine U.S. cur-rency in circulation.


23/38

22



the state and local level include red-light and speeding cameras and, more recently, gunire

detectors in some major cities Examples at the ederal level include the use o sensors,

radars, and unmanned aerial vehicles to detect illegal immigrants crossing the border, and

radiation detectors and X-ray screening o containerized cargo entering the United States See

the Counterfeit Detection and the Measuring Drug Production text boxes or examples o this

method

This method has a wide range o potential costs While a red-light camera may be relatively

inexpensive (and quickly pay or itsel in ine revenue), arraying sensors and radars across the

1,900 miles o southwest land border is very expensive When considering this method, the

perormance manager should consider i there are ways that the unobserved crimes can be

observed by technical means and, i so, whether the cost justiies the beneit

Selecting Between and Implementing the Methods

There is no simple ormula that identiies the right methods or perormance managers to use

Each measurement challenge is dierent, and the best methods vary with the circumstances o

the individual challenge There are some rules o thumb, however, that can add structure to theselection process

First, the perormance manager should identiy all relevant data currently captured by the

agency and elsewhere This may lead to a discovery o methods or direct estimation rom

available data or by expanding existing data sources

A second step is to thoroughly examine the context and setting within which the law

enorcement activities and perormance measure challenges occur Is there an entity,

whether governmental or not, with an incentive to report or measure the crime? I there

is an entity suering an economic loss rom the crime (such as credit card companies

Example o Using Technical MeasurementMeasuring Drug Production

A major area of federal law enforcement is combating thesmuggling of illegal drugs into and within the United States.

Although some drugs have significant domestic production,e.g., methamphetamine, many drugs are predominantly pro-duced internationally and smuggled into the United States,

e.g., cocaine. Identifying the flow of illegal drugs into theUnited States is an important measure, but unobserved tofederal law enforcement officials.

The U.S. Government and the United Nations both producesystematic estimates of drug flows and these estimates startwith technical measurement by satellite and aerial imagery.For cocaine, the U.S. estimates are produced by the Inter-Agency Assessment of Cocaine Movement(IACM). These estimates use Intelligence Community (IC) imagery of coca-producing countries toestimate the total level of cultivation. Subsequent analyses include likely harvest yields, refinedproduct yields, distribution destinations (i.e., how much goes to U.S. markets versus markets in

other countries), and flow across individual vectors or pathways (e.g., overland through Mexicoversus maritime transit through the Caribbean). The final estimates are thus produced by combiningmany of the different methods described in this report, but the estimation starts with technical col-lection by satellite and other imagery.


24/38

23


.sissfm.

and banks discussed in the Credit Card and Bank Fraud text box), they may already be

measuring it There may also be think tanks, interest groups, or academic organizations

that or various reasons have decided to measure the level o crime Can measurements

rom these organizations be validated and used?

When data that allow or direct measurement are not discovered, and no other organization

can be ound that is already measuring or can be persuaded to measure the oense rate, it istime to begin exploring the estimation methods identiied above, and any additional methods

that may be relevant All potentially relevant methods should be identiied and then systemati-

cally evaluated or their costs and beneits, eg, how precise the estimates will be, how dis-

ruptive to current operations collection will be, and whether they can be implemented in a

repeatable and transparent manner The relevant methods can then be compared to select the

best candidates or urther development This shorter list o the best candidates should then

be developed more thoroughly to decide which ones should be ully implemented and how It

may also take some trial and error or pilot projects to inalize selection It also may take the

combination o two or more methods to develop a complete estimate or a particular peror-

mance measure

Technical expertise may need to be obtained The perormance management oice may wantto hire a statistician, economist, or similar technical expert to conduct the work in-house or to

oversee supporting work being done by outside entities

Example o Implementing MethodsCredit Card and Bank Fraud

The United States Secret Service is responsible for investigat-ing violations of federal law with respect to credit card andbank fraud, e.g., obtaining and using a credit card under a

false identity without paying the bill. The Secret Service doesnot directly observe the level of credit card and bank fraud,and developing its own methods to estimate it would be chal-lenging and costly. But victims of these crimesthe creditcard companies and banks against which the fraud is con-

ductedsuffer significant losses from their occurrence. Notsurprisingly, therefore, credit card companies and banks do tryto estimate the levels of fraud, and they have direct access tothe data required for measurement because they own it.

Each company has its own data and estimates that could be systematically collected and combinedto create aggregate estimates. In many cases, the companies have already done this through tradeorganizations. For example, organizations such as the Association for Financial Professionals andthe American Bankers Association survey companies to measure loss due to fraud (credit card and

check fraud), the frequency and nature of fraud, and the environments in which fraud occurs onan annual or periodic basis. Using this information, these organizations make suggestions regard-ing best practices in fraud protection and identify the leading threats to banks and businesses inthe fraud market. Other organizations like Javelin Strategy & Research and the Consumer SentinelNetwork (a part of the Federal Trade Commission) also survey consumers and law enforcementagencies regarding fraud loss and reporting.

Utilizing these preexisting resources to estimate the amount of credit card and bank fraud may besubstantially cheaper than creating a new system to track and measure the amount of fraud peryear nationwide.


25/38

24



The methods described in Section III, and other methods not discussed, can be used to estimate

unobserved crime rates in a wide range o ederal law enorcement settings To illustrate how

the methods might be combined in an actual agency, this section develops a comprehensive

example using one o the largest law enorcement areas o the ederal governmentenorcement

o US immigration laws

Enorcement or most US immigration law is the responsibility o DHS and its components

ICE, CBP, and the USCG The primary violations are being in the United States unlawullyand, or the border enorcement components o DHS (CBP and USCG), the act o entering

unlawully Entry o illegal immigrants can be subdivided into whether it occurs at a US Port

o Entry (POE)or example, attempting to enter with alsiied documents or in the trunk o a

caror between POEs CBP is responsible or control o entry at the POEs, while responsibil-

ity or entry between the POEs is divided between CBP (generally the air and land domains)

and USCG (generally the maritime domain)

In addition to illegal entry, migrants can enter legally and then move to an illegal statusor

example, by overstaying or violating some other condition o their visa ICE is responsible or

law enorcement with respect to visa violations, and interior enorcement more generally

Figure 2 illustrates these violations and their relationships Figure 2 also includes the exit o

illegal immigrants, dividing it into those:

Who are removed by law enorcement

Who leave voluntarily

Whose status is adjusted or who die while in the United States

This understanding o illegal immigration identiies the key outcome perormance measures

or DHS and its components with respect to this area o law enorcement Note that the only

measures directly observable to the ederal government by counting law enorcement admin-

istrative records are on exit (law enorcement removal and a portion o deaths and adjust-

ments) All o the other perormance measures are based on events unobserved to ederal

law enorcement oicials Table 1 lists the perormance measures identiied in Figure 2 and

discusses ways in which they might be presented There are many supporting output, input,and eiciency measures that would also have to be assessed or a comprehensive peror-

mance management ramework or immigration law enorcement, but the ocus here will be

on the outcome measures

DHS currently only measures and reports the last outcome in Table 1, illegal immigrants

resident in the United States Estimates o this outcome have been made since 1980 The

primary method o computation is by taking US Census survey data to estimate the oreign-

born population in the United States and subtract rom it the number o lawul immigrants In

Part IV: Applying the Methods:

A Case Study of MeasuringIllegal Immigration


26/38

25


.sissfm.

other words, it combines survey data with administrative records to create an empirical esti-

mate o the illegal immigrant population in the United States

DHS does not currently systematically estimate or report the other outcomes in Table 1 Forillegal entry between Ports o Entry, the largest inlows are generally believed to be the south-

west land border (largest inlow) and the maritime border near the Caribbean (second largest

inlow)making these the highest priority or perormance measure development The north-

ern border, other maritime domains, and air domains could be let or uture development,

illustrating another important principleperormance managers should not try to solve every

perormance measurement challenge at once; start with the most important areas and then

move to lower priority areas ater progress has been made on the irst tier

DHS would have several options available or systematic collection o data or these measures,

including migrant surveys such as the Mexican Migration Project (MMP)5 household survey,

estimation o known low (low that law enorcement observes and either apprehends or does

not apprehend), sensor technology, analysis o recidivism data, and red-teaming Each optionhas strengths and weaknesses, and there is substantial variation in their costs The most cost-

eective options or the southwest land domain likely are migrant surveys (best way to esti-

mate the apprehension rate, generate data needed by other approaches, and are relatively

low-cost) and recidivism analysis (low-cost approach)

For the maritime domain, the problem is dierent From 1995 to 2009, the USCG reported

the probability o interdiction o illegal immigrants (rom which inlow can be computed) who

5. MMP is an academic data collection effort co-led by the University of Guadalajara (Mexico) and Princeton University.

Entry of Illegal Immigrants

Stock of Illegal Immigrants

Exit of Illegal Immigrants

Illegal Entry Between Ports Illegal Entry at Ports Visa Overstay

Voluntary Departure Law Enforcement Removal Visa Deaths/Adjustments

Figure 2: Illegal Immigration


27/38

26



attempted to enter the United States in the maritime domain between the POEs using a

known-low estimate Known low may be the most eicient solution in the Caribbean because

there are only a small number o source countries (eg, Haiti, Cuba, and the Dominican

Republic), the migrants must ollow relatively speciic pathways, and, or some countries,

such as Cuba, there are special legal provisions that make it in the migrants interest to

declare i they reach the United States The USCG estimates known low by using its and

other relevant agencies and governments data observed through interdiction activities, sur-

veillance, and intelligence

Unlike inlow between the POEs, where each domain has its own unique characteristics lead-

ing to selection o dierent methods, illegal entry at the POEs is roughly similar across the air,

land, and maritime domains Perhaps the most important dierence between domains or

estimating outcomes is that CBP usually receives inbound passenger maniests prior to pre-

sentation or both the air and maritime domains, but does not or the land domain Measuring

illegal entry at the POEs could be accomplished in a number o ways

One option would be to expand the existing measurement methods described previously in the

Randomized Secondary Screening text box to systematically cover all the potential ailure

points in the screening processor example, the accuracy o entry document issuance deter-

minations and the accuracy o screening processes at the POEs themselves Screening can be

divided into physical screening (or example, questioning individuals and use o magnetome-

ters) and inormation-based screening (or example, collecting biometrics and comparing these

Table 1: Key Outcome Perormance Measures or Immigration Law Enorcement

Outcomes Performance Measures Comments

Illegal EntryBetween Ports

Number o Individuals Entering theCountry Illegally Between the Ports oEntry in the Reporting [Year, Quarter,

Month]

Four primary variables: Number o individuals attempting entry Number o apprehensions

Apprehension rate Number o successul entriesThe measure may also be broken out bydomain (air, land, and maritime) and bysector within a domain

Illegal Entry AtPorts

Number o Individuals Enteringthe Country Illegally at the Ports oEntry in the Reporting [Year, Quarter,Month]

Four primary variables: Number o individuals attempting entry Number o apprehensions Apprehension rate Number o successul entriesThe measure may also be broken out bydomain (air, land, and maritime) and bysector within a domain

Visa Overstay Number o Individuals Remaining in

the Country Illegally ater Violatinga Condition o Their Visa in theReporting [Year, Quarter, Month]

This could be presented in the context o

total entries with a legal visa and totaldepartures by legal visa holders It couldalso be broken out by visa category

IllegalImmigrantsResident inUnited States

Number o Individuals in the CountryIllegally in the Reporting [Year,Quarter, Month]*

This could be presented in the context ohow it changed rom the previous reportingperiod and the causes o the change, ie,changes in entry rates versus changes inexit rates It could also be broken out bylocation, eg, state

* Although not reported in the DHS Annual Performance Report, an estimate of this measure is available from

other sources.


28/38

27


.sissfm.

to immigration, criminal, and security databases) Methods or measuring screening accuracy

include red-teaming and the already discussed randomized secondary screening ollowing a

primary screening encounter

Finally, the level o immigration that starts out legal but moves to an illegal status (or exam-

ple, visa overstay) has a ew complicating aspects Perhaps the most common violation is to

overstay a visa For the air and maritime domains, where outbound passenger maniests exist(or commercial air and maritime travel, at least), overstay rates could be estimated by com-

paring visa records with passenger maniest data The challenge with this method is that the

land domain (pedestrian and vehicle traic) does not have outbound passenger maniest data

Two options or the land domain would be to use surveys or randomized investigations o

expired visas or which land exit is expected (perhaps because entry was by the land domain)

For non-overstay orms o compliance (ie, it is within the period o the visa but the visa

holder may have violated another condition o the visa such as reraining rom criminal activ-

ity), surveys and randomized investigations could again be used to estimate violation rates

Table 2 summarizes the methods that may be used to estimate the outcomes or illegal immi-

gration law enorcement These are only meant to be suggestions, as urther analysis may be

required to inalize the exact approaches to be taken in each case The goal is to illustratehow the individual detail o speciic measurement situations drive the selection o the appro-

priate methods

Table 2: Possible Estimation Methods

Outcomes Estimation Methods

Illegal Entry Between Ports Southwest Land Domain: Survey Data and Analysis o AdministrativeRecords (Recidivism Analysis)

Caribbean Maritime Domain: Administrative Records (Known-FlowEstimates)

Illegal Entry At Ports Experimental Methods (Red-Teaming) and Inspections (Randomized

Secondary Inspections)

Visa Overstay Visa Overstay Air and Maritime Domains: Analysis on AdministrativeRecords (Government Records Combined with Commercial Records)

Visa Overstay Land Domain: Survey Data and Investigations

Other Visa Violations: Survey Data and Randomized Investigations

Illegal Immigrants Resident inUnited States

Analysis o Survey Data and Administrative Records

Implementing the entire program described above would not be easy Complete cost esti-

mates6 and implementation plans would have to be developed or each method in each area

Potentially ierce resistance would have to be overcome; some examples o the likely argu-

ments against better perormance measurement are provided in the next section It might take

several years beore the methods could be ully implemented and suiciently reined to pro-

duce truly reliable estimates But doing the analysis to develop the strategy will take one

major obstacle o the tablethe argument that unobserved events cannot be measured and

there is no point in trying

6. The cost estimates should also be put into perspective. Although many investments would have to be made to implement the iden-

tified measurement strategy, the cost would likely only be a very small fraction of total expenditure in the mission area and may not be

any more than the current expenditure level for all of the proxy measures that DHS currently uses in this mission area. Providing those

comparisons could be a valuable way to overcome opposition based on the strategys cost.


29/38

28



This report has addressed the most direct challenge to measuring law enorcement peror-

mance outcomeshow to estimate unobserved events But even with a sound measurement

methodology in hand, there are still obstacles to implementing outcome-oriented data-driven

management In addition, simply measuring the outcomes is not enoughthe perormance

manager must also build an empirical understanding o how law enorcement activities and

outside actors aect the outcomes This section briely reviews these challenges

Challenges

There are many challenges acing a perormance manager in increasing the use o data to

drive decision-making Five speciic challenges are particularly relevant to ederal law enorce-

ment While other challenges will certainly arise in improving the quantitative content o per-

ormance management, preparing or these ive will assist perormance managers in ighting

the major battles ahead o them

Challenge One: Institutional resistance to diverting scarce resources rom enorcement to

data collection and measurement. Although it may seem obvious that one o the irst things a

serious manager does when solving a large, complicated problem is to start measuring it,

incentives within government organizations are oten not aligned to promote this long-run,

problem-solving view In the short-run view oten taken by government managers, investing in

data collection and measurement takes resources rom current enorcement and is thus con-

sidered a lower priority (even i it would improve mission perormance over the long run)

An example comes rom Customs and Border Protections (CBP) major increase in technology

investment along the border during the 20062011 time period CBP greatly increased its use

o technology, including sensors and radars to monitor or drug smuggling and illegal border

crossings, during that period, but its operating concept was ocused on deployment o technol-

ogy to locations where there were suicient law enorcement assets, or concurrent increases in

law enorcement assets were made, or interdiction This limited the deployment o the tech-

nology and increased its costs CBP was reluctant to make technology investments where

increased situational awareness and data collection would be the primary ocus, ie, where

interdiction would not be assured However, obtaining this situational awarenessie, mea-

suring outcomescould lead to improved allocation o scarce resources and improved out-

comes over time, even i there were some sensor hits or which no Border Patrol agent was

available to respond in the short run Making this cultural change in a law enorcement orga-

nization can be challenging

The perormance manager trying to drive change through a law enorcement organization must

predict how this concern will maniest itsel in their organization and develop a strategy or

overcoming it at the start o the eort The basic response to the challenge is to explain the

Part V: Challenges and Conclusions


30/38

29


.sissfm.

dierence between obtaining short-run versus long-run results The reluctance o operators to

make investments or the uture is a standard problem and should be countered with a well-

reasoned argument or why long-run results will be improved by making investments today

Challenge Two: Data quality. Although concerns with using an analytic method to estimate an

outcome that cannot be directly measured can be exaggerated by opponents o perormance-

driven management, there are legitimate concerns that the perormance manager must takeseriously One set o legitimate concerns is technical; the analytical methods applied will have

basic statistical properties such as conidence levels, sensitivity to measurement error, and

reliance on assumptions that characterize the accuracy and reliability o the results The per-

ormance manager must explicitly identiy and explain these issues to the decision-makers

using the data The goal is to make decisions based on acts and empirical evidence; hiding or

minimizing limitations to those acts and empirical evidence harms the decision-making pro-

cess It can be tempting or a perormance manager to want to minimize these concerns when

acing signiicant organizational opposition to increased data-driven management, but the per-

ormance manager must resist this deensiveness and remain transparent and open about the

limitations o the data and analysis

Another set o legitimate issues related to data quality concerns maturing the methods throughtime No matter how expert the perormance manager is, new methods to solve complicated

estimation challenges will take time to get right Perormance managers and decision-makers

should be careul not to expect too much too ast rom the new attempts to measure their law

enorcement challenge Estimates should be created, socialized, criticized, revised, and

improved Perhaps targets should not be used in the initial years o developing and socializing

the data, both because o data quality and maturation concerns and because o the cultural

changes that introducing the estimates create

Challenge Three: Cultural change that comes with quantiying a problem or the irst time.

Attempting to measure a contentious outcome or the irst time is a big step or an organiza-

tion and exposes it to new areas o criticism and risk An example comes again rom border

control Although DHS does not systematically measure border low and apprehension rates,

there are some estimates available rom empirical studies, and these ind an apprehension

rate between 33 and 50 percent (which implies a low rate o approximately one million per

year) or the southwest land border between the POEs With some policy-makers arguing that

the only acceptable outcome is a sealed border (zero net low and 100-percent apprehension

rate), it is understandable why the less courageous in DHS are reluctant to develop a report-

able measure

But hiding the truth does not change it, and makes it harder both to improve outcomes and

have open, act-based discussions about what outcome levels are acceptable In act, the

apprehension rate on the border is not that dierent rom the closure rate or many crime cat-

egories across the country For example, the national average closure rate or all violent crimes

is about 45 percent and the closure rate or all property crimes is less than 20 percent When

viewed in this context, the DHS border perormance may not be considered that bad

Regardless o whether 3350 percent is a good or bad rate, it is more important that policy-

makers cannot have an open debate on whether it is acceptable, whether more resources

should be applied to increase it, or whether more eicient methods could increase it at current

expenditure levels The perormance manager must be sensitive to how the results will be

received and be prepared to discuss why the results, even i they may appear unavorable at

irst, need to be produced and presented Putting them into context as in the above example,

with comparisons to violent and property crime closure rates, can be an important way o

dealing with this challenge


31/38

30



Challenge Four: Perormance management sits at the intersection o operators and analysts.

While the tendency o operators may be to dismiss analysis and data to inorm decisions,

there may also be a tendency or analysts to over-rely on data and analyses All data and

analyses are imperect, contain measurement error, and rely on assumptions Although objec-

tive, quantitative perormance data are essential to sound management, no linear pathway

exists rom data to a model to an estimate to a decision Decisions should be inormed by the

estimates, but may also be inormed by short-run operational realities and constraints, politi-cal actors, and stakeholder concerns and interests Perormance measurement analysts must

have a seat at the table but must not overstate the useulness o their results

Challenge Five: Analyzing the relationship between outcomes, outputs, and inputs. This

report is ocused primarily on how to measure unobserved events to improve law enorcement,

but estimating and reporting crime rates are not the only responsibilities o a perormance

manager in supporting decision-making A complete law enorcement perormance manage-

ment ramework needs to include empirical analysis o how outcomes, outputs, and inputs are

related to each other It must also include the analytic ability to orecast these relationships

into the uture

Similar to the challenges created by the cultural change o measuring the outcomes an organi-zation is trying to impact when they were not previously measured, measuring the direct con-

tribution o the organizations activities on those outcomes also represents a signiicant cultural

change and potential challenge No program manager wants to learn that the program he or

she has been devoted to or years has not had much o a measurable impact on the outcome

the program is supposed to eect

Outcomes are aected by program outputs Numerous programs are already in place, with

other potential programs available, to work toward most outcomes that ederal law enorce-

ment organizations are trying to achieve Selecting among possible programs to achieve

desired outcomes most eiciently requires analysis For example, the DHS component ICE is

responsible or interior immigration law enorcement Some o the programs it uses include

worksite investigations, worksite audits, human smuggling investigations, visa compliance

investigations, and criminal alien revie

five methods for measuring unobserved events

Documents