using inspection information to identify and ameliorate risks - andrew robinson
TRANSCRIPT
Using Inspection Information to Identify andAmeliorate Risks,
and Monitor Performance of Risk Management
Andrew Robinson
CEBRA, University of Melbourne
September 12, 2013
Outline
Background
Measuring Performance
Reporting
Risk–Based Approach to Inspection
Lessons Learned
Background
Quarantine is Expensive
Program Annual Revenue1
Airports $79,810,000Import Clearance $112,862,000International Mail $22,100,000... ...
TOTAL $376,165,000
Why (and how) do we spend all this money?
1DAFF 2007–8 Figures, Table 4, Beale et al. (2008).
Terror on the High Seas
Picture: AP Source: AP
http://www.news.com.au/lifestyle/home/
panic-as-giant-african-land-snail-invades-us-state-of-florida/
story-fneuz6rh-1226620722434
Giant African Snail
I Achatina fulica
I Up to 30 cm long
I Up to 1 kg in weight
I Omnivorous (eats 500 species of plants)
I Up to 9 years lifespan
I Hermaphroditic reproduction
I Up to 1200 eggs p.a. after matingI In the 1960s, 3 snails accidentally released in FL.
I 10 years and a million dollars to catch 18,000 snails.
I First recorded in American Samoa in the mid–1970sI a million snails were collected by hand in 1977I more than 26 million snails were collected over the following
three years.
ECIR & RDI
International Vessels
ULD
Plant–Product Pathways
Measuring Performance
Good Questions.
1. How risky is pathway X ?
2. How much effort are we investing in our monitoring of X ?
3. How appropriately are we focusing our effort?
4. How useful is our intervention?
5. How can we (better) reduce the risk?
Performance Indicator Qualities.
A Performance Indicator is simply a Statistic.
A good set of performance indicators . . .
1. is easy to interpret;
2. does not mislead the decision-maker;
3. can be computed using readily-available data; and
4. provides an appropriate measure of uncertainty.
Pathway
Examine? ExamineNot
Compliant
Compliant
LeakageSurvey?
ExamineNot
Compliant
Released Compliant
v
i
Rectification
bv − i
v
n
v − n
i− b
b
n− y
y
Rectification
y
Figure 1: Flow chart for sampled intervention of pathway with leakage survey. Rectifica-tion means that the BRM is captured, and the mail item is then assumed to be compliant.The leakage survey records whether the unit was released or inspected after screening.
3
Performance Indicator Examples.
I Leakage (Slippage)
I Before-intervention compliance rate (BIC),
I Post-intervention compliance rate (PIC),
I Effectiveness (E, for intervention), and
I Hit Rate (HR, or Odds Ratio, for screening / profiling).
Needs to Know
In order to measure passenger cohort risk, we need to know
I v , the volume, which is the number of units on the pathway;
I i , the number of units inspected after screening;
I b, the number of units that were non-compliant;
I n, the number of units processed in the leakage survey; and
I y , the number of units that were found to be non-compliantin the leakage survey.
NB: n and y may have subscripts i for inspected and r for released.
Intervention Leakage Count l : Point Estimate
l = i × yini
+ (v − i)× yrnr
(1)
A Worked Example With Completely Fabricated Data
> v = 10000
> i = 3000
> b = 30
> n_i = 100
> y_i = 5
> n_r = 300
> y_r = 5
The estimated leakage count is
> (l.hat = i * (y_i / n_i) + (v - i) * (y_r / n_r))
[1] 266.6667
Intervention Leakage Count l : Interval Estimate
Let pi = (yi + 1)/(ni + 2) and pr = (yr + 1)/(nr + 2), and
sl =
√i2 × pi × (1− pi)
ni + 2+ (v − i)2 × pr × (1− pr )
nr + 2(2)
Then the interval estimate for the leakage count is
lI = i × pi + (v − i)× pr ± 1.96× sl (3)
Estimated Leakage Count
The standard error of this estimated leakage count is computed by
> p_i = (y_i + 1) / (n_i + 2)
> p_r = (y_r + 1) / (n_r + 2)
> (s_l = sqrt(i^2 * p_i * (1 - p_i) / (n_i + 2) +
+ (v - i)^2 * p_r * (1 - p_r) / (n_r + 2)))
[1] 89.69113
The 95% confidence interval estimate for the leakage count is
> (l.int = i * p_i + (v - i) * p_r +
+ c(-1,1) * 1.96 * s_l)
[1] 139.7488 491.3380
Estimated Approach Count a: Point and Interval
a = b + l (4)
aI = b + lI (5)
The estimated approach count is
> (a.hat = b + l.hat)
[1] 296.6667
and the interval estimate of the approach count is
> (a.int = b + l.int)
[1] 169.7488 521.3380
Before-Intervention Compliance
BIC =v − a
v(6)
BICI =v − aI
v(7)
Then BIC is
> (BIC = (v - a.hat) / v)
[1] 0.9703333
and the interval estimate of BIC is
> (BIC.int = (v - a.int) / v)[2:1]
[1] 0.9478662 0.9830251
Post-Intervention Compliance
PIC =v − l + yi + yr
v(8)
PICI =v − lI + yi + yr
v(9)
Then PIC is
> (PIC.hat = (v - l.hat + y_r + y_i) / v)
[1] 0.9743333
and its interval estimate is
> (PIC.int = (v - l.int + y_r + y_i) / v)[2:1]
[1] 0.9518662 0.9870251
NCE (Inspection)
NCE =b
a(10)
NCEI =b
aI(11)
The E of the inspection, expressed as a percentage, is
> (NCE.insp.hat = b / a.hat) * 100
[1] 10.11236
with interval estimate
> (NCE.insp.hat = b / a.int)[2:1] * 100
[1] 5.754424 17.673170
Hit Rate
HR =b + lii
(12)
Let
li = i × yini
(13)
> (HR.hat = (b + i * (y_i / n_i)) / i)
[1] 0.06
Hit Rate
An estimate of the SE is
sli = i ×√
(yi + 2)× (ni − yi + 2)
(ni + 4)3(14)
then
liI = i × yi + 2
ni + 4± 1.96× sli (15)
following Agresti and Coull (1998), and
HRI =b + liI
i(16)
Hit Rate
> (s_hat_li = i * sqrt((y_i + 2) * (n_i - y_i + 2) /
+ (n_i + 4)^3))
[1] 73.70656
> (l_i.hat = i * (y_i + 2) / (n_i + 4) +
+ c(-1,1) * 1.96 * s_hat_li)
[1] 57.45822 346.38793
> (HR_I = (b + l_i.hat) / i)
[1] 0.02915274 0.12546264
Challenge 1
In some systems, we don’t know i so we estimate it using n,∑
i ,and raking.
Cohort A & R X-Ray Manual Total
A a1 a2 a3∑
N AB b1 b2 b3
∑N B
C c1 c2 c3∑
N C
Total∑
N AR∑
N X∑
N M∑
Raked up to 4 dimensions — cohort, port, declaration, screeningmethod.
Challenge 2
n is small so p = yn
is highly variable, and sometimes i < b!
I Smooth i using a ridge.
I Smooth p using Empirical Bayes methods.
Bias–variance trade-off (Could also create larger groups).
Empirical Bayes
L(α, β; x ,n) =
k∑i=1
− ln Γ(α+ xi)− ln Γ(β + ni − xi)
+ ln Γ(α+ β + ni)
+ ln Γ(α) + ln Γ(β)− ln Γ(α+ β) (17)
pi =xi + α
ni + α+ β(18)
Comparing Profiles: Leakage curves and ROC
DAFF Biosecurity: International Passenger Profiling
●
●
●
●
0 500 1000 1500 2000
0.0
0.5
1.0
1.5
Risk
Effort (passengers)
Leak
age
(%)
●
●
●
●
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
AUC: 0.65
False Positives
True
Pos
itive
s
Figure 3.1: Example graphical assessment of profiling strategy for three categories of passengers,as reported in Table 3.1. The left panel shows the leakage curve, and the right panel shows theROC curve, and reports the AUC in the panel title. The solid line is the 1:1 line, representing arandom allocation of passengers to risk categories.
the overall operational outcome. In particular, the higher volume, lower risk segments are goingto be difficult to profile. Profiling has its operational limits, and is a complement to the other190
interventions.2
3.3.2 Balance
Here we refer to balance in the informal sense that the profiling will be more likely to providepredictions with desirable statistical properties if the categories into which the passengers aredivided are, more or less, the same size. The counts of citizenships of international passengers are195
quite variable; the best-represented is Australia with nearly 6 million arrivals, and a number ofcategories are very small, for example, it appears that there was only one arrival from AmericanSamoa during the data collection period. It would be straightforward to set a lower limit forcategory sizes, and merge all categories that are smaller than that lower limit, however in doingso it is possible that some small but useful efficiencies could be missed. For our purposes we200
retained as many of the categories as we had data for, and used smoothing statistics (specifically,the empirical Bayes procedure, see Appendix C) to provide some robustness for the estimatesof the smaller categories.
3.3.3 Monitoring
As noted in earlier ACERA reports ( e.g. Robinson et al., 2008), all pathways should be inspected205
at some level, even if that level of inspection is purely for the purposes of monitoring the pathwayto ensure that it remains within operationally acceptable limits. This monitoring does not referspecifically to the use of a leakage survey, but rather to a random infrequent inspection ofpathways that are deemed likely to be compliant. It may be convenient for this monitoring tobe performed as part of the leakage survey, as is done presently.210
The consequence of this monitoring is that the leakage curve will not quite reflect the trueamount of effort required to manage the pathways, because the monitoring component is not
2Comment 16: Text from CW.
12
Reporting
Reporting: PIC by Port, Channel, and Declaration
A/R K9 Manual Xray●
●
● ● ●
●
● ●
● ● ● ● ●●
● ●
● ● ● ● ●
●
●●
● ●●
● ●● ● ●
● ●●
● ●● ● ●
●
●
●
● ● ● ●●
●●
●
● ● ●●
●
● ● ●●
●
● ● ●
92.5
95.0
97.5
100.0
92.5
95.0
97.5
100.0
Declarant
Non−
Declarant
Pt 1 Pt 2 Pt 3 Pt 4 Pt 5 Pt 6 Pt 7 Pt 8 Pt 1 Pt 2 Pt 3 Pt 4 Pt 5 Pt 6 Pt 7 Pt 8 Pt 1 Pt 2 Pt 3 Pt 4 Pt 5 Pt 6 Pt 7 Pt 8 Pt 1 Pt 2 Pt 3 Pt 4 Pt 5 Pt 6 Pt 7 Pt 8
Port
PIC
(%
)
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
BALAOFKFNJJABFFEMHCCNFQCMG
PIKANBCHKDKCLEGEHAGAOCBJAJ
MBMI
DDOBCIDJJEBBQGKGABLHRI
KHQHGH
FIREOABEHJ
90.0 92.5 95.0 97.5 100.0BIC (%)
Citi
zens
hip
Lessons Learned
I Start small — solve case studies.
I Visit.
I Operationalise–Light.
I Sustain engagement.
I Be patient.
I Deliver useful outcomes.
I Build bridges inside and outside.
Discussion and Questions
Background
Measuring Performance
Reporting
Risk–Based Approach to Inspection
Lessons Learned