joint research centre - european commission · joint research centre changes to lpis qa: analysis...
TRANSCRIPT
The European Commission’s
science and knowledge service
Joint Research Centre
Changes to LPIS QA:
Analysis of the
sampling
representativeness
and impact
on 2017
W. DEVOS and D. FASBENDER
2
PART1: Sampling --- Trigger
ECA Special report (25/2016) to “assess the reliability, effectiveness
and impact of LPIS/GIS across the EU”
Recommendation 5:
“The Commission should, before the start of the QA exercise 2017, carry
out a cost-benefit analysis to determine whether the representativeness
of QA samples could be improved so that a better coverage of the
population of parcels in
the LPIS can be achieved.”
EC (DG JRC) acknowledged the recommendation since
“The monitoring of sample representativeness is part of a continuous
process.”
2017: Experience of 2 years dedicated LPIS QA image provision
Image allocations unchanged in 2016
update with latest population data
4
What is representativeness ?
Short definition:A representative sample is a sample that shows
similar characteristics compared to the population
from which it has been issued.
Representative of what ?In LPIS QA context: representative of the system’s
quality (i.e. presence of non-conformities, MEA correctly recorded…)
How to proceed with analysis?
For this analysis, reference area is the only
observable parameter (i.e. proxy parameter) !
Note: larger RP may behave different than smaller RP
Use simulations for verifying that the whole sampling
procedure tends to produce samples in which
no particular portion of the population is over-/under-
represented.
Population
Representativesample
Unrepresentativesample
5
Population statistics:Min, max, mean, var, range, intervals of 95%,…
Each sample statistics:Min, max, mean, var, range, intervals of 95%,…
With 1, 2, 3,… zone(s) until acceptable match
Simulation (Improving representativeness of LPIS samples)
100 x sampledeach time random zones
6
Which characteristics (stats) to compare?
Z-test Chi-square test Kolmogorov-Smirnov test
PCPI (% of pop. in
central prob. interval)
Comparison of sample and pop. means
Comparison of sample and pop. variances
Full comparisonof distribution’sshapes
Comparison of « 95% imprints »
•Detection of bias
•Fails to assessthe variability
•Fails to assessthe bias
•Detection of abnormalvariance
•Tests all aspectsof the sample
•So idealistic thatit becomes toorestrictive
•Detection of shifts•Detection of abnormalvariance…while keeping the objective realistic!
7
Tests’ results on the past LPIS samples.
Proposed tests were
applied to the actual
samples of previous
years (2010-2016)
No particular trend in
the acceptance rates (all
systems together)
Two distinct periods
2010-2014 and 2015-
2016
Impact of “at least 3
images” could be
identified (green and red
lines)
Old rule-of-thumb
reintroduced from
20.000+ km2
Accepta
nce r
ate
of th
e t
ests
[%
]
PCPI ideal
95%=Producer error
8
Simulation to pass KS: test too restrictive ?
Hypothesis:
What if we increased
the current (2015-16)
image allocation:
threefold
fivefold
The KS test still
fails 43 systems!
PS: nonsense to
calculate the largest
systems.
Number of images 2015-2016
1 2 3 4 5
9
PCPI simulation: steps
The working definition of representativeness: « The interval with 95% of the sample shouldcontain 95% of the population ». Statistical tolerance / margin :±2%
1: zones
2: sample
3: interval
4: PCPI
10
B. A new density mask
• Threshold of 2 RPs/km² for low density
• Using 2016 population
• 2 km resolution regular grid (instead of 10 km)
• Improved coastline delineation
2015-2016 2017
11
C. Low RP density map modifications
Low RP density = “less than
2 RPs/km²”
Simulated campaigns to
establish the “maximum”
percentage of low RP
density mask in the control
zones
MSs with maximum
percentage < 20% are
regarded as homogeneous (not considered for the PanEU zone)
N.B. Size of MS also taken into
account (FR in PanEU, BE-WA and DE-
ST not)
Total of 15 LPIS systems
in the PanEU zone
Values are the number of images per LPIS system
12
D. A new image allocation map
LPIS name 2015-2016 2017
BE-WA 1 + 1
CY 1 + 1
DE-BB 1 + 2
DE-MV 2 + 1
DE-NW 2 + 1
DE-SN 1 + 1
DE-ST 1 + 2
DE-TH 1 + 1
DK 2 + 1
EE 1 + 2
HR 3 + 1
LU 1 + 1
LV 2 + 1
PL 4 + 2
PT 4 + 1
SK 2 + 1
UK-SC 2 + 1
TOTAL 121 +21
1 image2 images3 imagesPanEU (2 images)
13
E. A new RP sampling procedure
Simple random sampling (2016) Stratified random sampling (2017)
Example:
Total of 10,000 RPs covered
spread over 4 zones
Need to select 3750 RPs(i.e. 3 times 1250)
The 3750 RPs selected
proportionally in each zone
… and then pooled together
The 10,000 RPs are pooled
together
… and then the 3750 RPs are
selected at once
14
Why stratified random sampling ?
Sampling design better in line with the
spatial component of the control
zones:
Each individual sample is generated per zone
before to be pooled together
No “missed control zone”:Each zone is covered with at least one inspection (Particularly noticeable if the density of RPs is drastically
different between the zones)
Independent samples between the
zones:Each sample will generally be representative of
its own zone (not always true for zones with very low
density) so it allows to analyze the difference
between control zones
15
F. … adapted inspection loop (?)
Proposal 1 « scientific »
Proposal 2 « pragmatic »
Proposal 3 « lowest burden »
DG JRC generates one sample pre-selection list per control zone
MSs make sure that the required number of inspections per control zone is reached
DG JRC generates one uniquesample pre-selection list
MSs make sure that the required number of inspections per control zone is reached
DG JRC generates one uniquesample pre-selection list with a particular structure that “maximizes the chance to reach the required number of inspections per control zone”
• Fully in line with the samplingprocedure
• Instructions are clear (work
organized per zone)
• There is up to 6 lists for someMS (update of software?)
• Required number of inspections is garanteed
• Instructions are more complicated (manual jumping until
required number of inspections)
• There is one list (No need to
update the software)
• Required number of inspections is not garanteed
• Instructions are clear(=Business As Usual)
• There is one list (No need to
update the software)
In all the cases, no increase of inspections !
16
Example:
3 zones
10000 RPs covered (10% in control zone 1, 30% in control zone 2, 60% in control zone 3)
1250 inspections (pre-select 3750 RPs)
125 for control zone 1, 375 for control zone 2, 750 for control zone 3
Proposal 1: “Scientific”
Control zone 1: 125 inspections
RP1.1
RP1.2
RP1.375
Control zone 2: 375 inspections
RP2.1
RP2.2
RP2.1125
Control zone 3: 750 inspections
RP3.1
RP3.2
RP3.2250
Once 125 inspections reached,
start with the 2nd list
Once 375 inspections reached,
start with the 3rd list
Once 750 inspections reached,
full stop
17
Proposal 2: “Pragmatic”
Example:
3 zones
10000 RPs covered (10% in control zone 1, 30% in control zone 2, 60% in control zone 3)
1250 inspections (pre-select 3750 RPs)
125 for control zone 1, 375 for control zone 2, 750 for control zone 3
RP1.1
RP1.2
RP1.375
RP2.376
RP2.377
RP2.1500
RP3.1501
RP3.1502
RP3.3750
Once 125 inspections reached, jump
manually to the 2nd “block”
Once 375 extra (sub-total of 500)
inspections reached, jump manually to
the 3rd “block”
Once 750 extra (sub-total of 1250)
inspections reached, full stop
18
Proposal 3: “Lowest burden”
Example:
3 zones
10000 RPs covered (10% in control zone 1, 30% in control zone 2, 60% in control zone
3)
1250 inspections (pre-select 3750 RPs) - known skipping rate = 10%
125 for control zone 1, 375 for control zone 2, 750 for control zone 3
RP1.1
…
RP1.125
RP2.126
…
RP2.500
RP3.501
…
RP3.1250
RP1.1251
…
RP1.1263
RP2.1264
…
RP2.1300
RP3.1301
…
RP3.1375
RP1.1376
RP2.1377
RP3.1378
RP1.1379
….
Required number without skipping
Once 1250 inspections reached, full stop
Expected end of loop end from skipping in the past
19
2017: Improved representativeness
The objective of 2% margin
for the PCPI is now
generally met (except for CY and
LU)
Score system (with classes of
PCPI) shows good cost-
benefit improvements
Classes of PCPI2015-2016 2017
Good(less than 1% difference) 26 31
Intermediate (between 1-
2% difference) 11 11
Poor(more than 2% difference) 7 2
Score (Good=2, Intermediate=1,
Poor=0) 63 73 (+16%)
Total images123 144 (+17%)
20
Conclusions (on part 1)
• Better establishment of the location of the agricultural areas (i.e.
improved spatial resolution of density masks, improved coastline delineation…)
• Representativeness of the samples assessed and remedial actions
were taken where necessary
• Biggest changes are:
- addition of 21 images spread over 17 LPIS systems
- change of RP sampling procedure (internal for DG JRC)
- repartition of the inspections imposed across the control zones
(ETS inspections by MS) through adapted inspection loop (?)
• Required (?) number of inspections is provided per control zone
but total number of inspections (500/800/1250) is unchanged !
21
PART 2: Clarifications on some previous
communications
1. Parcels in scope (“active parcel”)
2. Counting multiple non-conformities in a single non-
conforming item
3. Mapping / reporting small artificial surfaces
THESE ARE NO CHANGES!!! Merely clarification / typo
corrections to avoid erroneous inspection recycled slides
22
reference parcel
Agricultural parcel 1declared
Agric
ultu
ral
parc
el 2
decla
red
n years
Agric
ultu
ral
parc
el 2
decla
red
Agricultural parcel 1declared
Agric
ultu
ral
parc
el 2
n years
Agric
ultu
ral
parc
el 2
reference parcel
Agricultural parcel 1
Agric
ultu
ral
parc
el 2
Agric
ultu
ral
parc
el 2
reference parcel
Active Active Not active
Belief
The LPIS QA scope
contains all RPs on
agricultural land
Fact
Only the “active” RPs are
in the scope
n ≥ 2
23
P1
P2
P3
P5
P4
Declared parcels
1. 5,00 ha
2. 2,00 ha
3. 2,00 ha
4. 4,00 ha
5. 2,00 ha
Claim year 2014
RP1-1. 5,00 ha
RP2-2. 2,00 ha
RP2-3. 2,00 ha
RP2-4. 4,00 ha
RP2-5. 2,00 ha
Claim year 2016
RP1-1. 5,00 ha
RP2-2. 2,00 ha
RP2-3. -
RP2-4. 4,00 ha
RP2-5. -
Claim year 2015
RP1-1. 5,00 ha
RP2-2. 2,00 ha
RP2-3. -
RP2-4. 4,00 ha
RP2-5. 2,00 ha
ALERT
RP 2RP 1
IN SCOPE
RP2 remains completely active
24
Non-conformities (cf. EU 640/2014 art 6.2)
Conformance class 1 “assess the quality of LPIS”, counts non-
conforming items (RP or crop aggregate)
•QE1, QE2, QE3 ≈ factual assessment
•Straightforward link with RP upkeep processes
•expectation = 5% (QE2) or 1% (QE3) non-conforming items
NO CHANGE FROM ETSv5.3
Conformance class 2 “identify possible weaknesses”, requires a
broader system wide analysis,
•QE4 ≈ analysis on the LPIS processes and design
•Example: a single, large parcel is contaminated, includes ineligible
land and its land is wrongly classified
this represents 1 NC RP but 3 different weaknesses!
QE4: expectation = <5% non-conformities per 100 items
where item ≡ RP/aggregate24
25
In practice 1
25
Illustration of observation change(fictitious)
ETS v6.0: If area-conforming then locate contaminating road and building [x,y]
26
Illustration of processing change
26
test observation ETS v5.32014
ETS v6.02015+
Area conformance
120198 m2
><
94513 m2
Fail Fail
Contamination 1 road, 1 shed n/a n/a *
Area correctness
PG:120198 m2
><8925 m2
--- Fail
QE2 Any fail above Fail Fail
QE4 (ex-QE3) Count QE2/QE3 fails
1 non con-forming item
2* non conformities
*: any unrelated road and shed would be individually counted i.e. 2 counts
27
Mapping and reporting of small non-
agriculture features
Reporting of the artificial sealed features in ETS is regardless
their size
•for accounting: any potential triggers for contamination
•following the LPIS guidance of AGRI, stating that “man-
made constructions …. should be excluded from the RP by
delineation”
•However, ETS does not require a delineation of all non-
agriculture features
• only those larger than or equal to 0.03 ha
• or 0.01 ha depending on the orthoimage and nature of
land feature
• other smaller features are reported as points only
27