statistica 1: neyman vs bayes. frequenze in esp. di ...rotondi/statistica_1.pdf · •statistica 2:...
TRANSCRIPT
• Statistica 1: Neyman vs Bayes.
frequenze in esp. di conteggio
• Statistica 2: likelihood ratio e test di ipotesi
segnale su fondo
• Track fitting: tracking in GEANT3 &GEANT4
filtri di Kalman e global fitting
1
Alberto Rotondi Università di Pavia - Scuola di Alghero 2010
2
3
2
4
5
PHYSTAT 05 - Oxford 12th - 15th September 2005
Statistical problems in Particle Physics, Astrophysics and Cosmology
6
7
8
9
Frequentist confidenceintervals
x
10
x1x2
x
x= x1 < < x= x2 < < 2
x
1< < 2 when x1 < x <x2
True value
Possibleinterval
CL
P 1< < 2) = P(x1 < x<x2) = CL
11
NEYMAN INTEGRALS
x
x
Elementary statisticsmay be
WRONG!!
12
Because P{Q} does not
contain the parameter!
13
Estimation of the sample mean
since
Due to the Central Limit theorem we have a pivot quantity when N>>1
Hence:
14
Hence, we have three methods to find confidenceintervals with the Neyman (frequentist) technique:
• Graphical method (Neyman band)
•Analytic with the Neyman Integrals (Clopper Pearson method)
• Inversion method (pivot variable)
x
x
Counting experiments
CLtx
xP
][
||
CL is the asymptotic probability the interval will contain the true value
COVERAGE is the probability that the specific experiment does contain the true value irrespective of what the true value is
On the infinite ensemle of experiments, for a continuous variable Coverageand CL tend to coincide
In counting experiments the variables are discrete and CL and Coverage do not coincide
What is requested is the minimum overcoverage
16
Wald
1
)1(
4
1
2||)1(
||2
2
2
2
2
n
tn
ff
n
tt
n
tn
tf
pt
n
pp
pf
n
fftfpn )1(1
t is the quantile of the normal distribution
t
t=1, area 84%Quantile =0.84P[|f-p|<t ]= 68%
CLt
n
pp
pFPt
p
pFP
)1(
||
][
||
Counting experiments: Binomial case
Wilson interval(1934)
Wald (1950)Standard in Physics
17
xtxx
Counting experiments: Poisson case
Wilson interval (1934)
Wald (1950)Standard in Physics
42
|| 22 txt
txCLt
xP
Why to complicate all this?
18
Why to complicate all this?
19
n
fftfn )1(1
n=20
20
p1 p2
p
21
The 90% CL gaussian upper limit
Observedvalue
90% area
10% area
Meaning II: a larger upper limit should give values less than the observed one in less than 10% of the experiments
1.28
Meaning III: the probability to be wrong is 10%
Meaning I: with this upper limit, values less than theobserved one are possible with a probability <10%
22
23
muon a selects trigger the that prob.
pion a selects trigger the that prob.
piona betoprob.
muona betoprob.
trigger a give to pion a for prob.
trigger a give to muon a for prob.
)|(
)|(
90.0)(
10.0)(
05.0)|(
95.0)|(
TP
TP
P
P
TP
TP
The probability to be a muon after the trigger P( |T):
678.090.005.010.095.0
10.095.0
)()|()()|(
)()|()|(
PTPPTP
PTPTP
The trigger problem
10.000 particles
9000 1.000
950450
8550 50
enrichment 950/(950+450) = 68%
Efficiency (950+450)/10.000 = 14%
triggertrigger
prior
26
27
28
29
30
31
Bayesian credible interval
32
2n
33
34
35
BayesiansvsFrequentists
+
36
Why ML does work?
hypothesis observation
37
38
)
39
)();(ln2)(
2
1ln 2
2
2)(
2
1 2
XLX
ex
Gaussian variables: ML corresponds to Minimum 2
40
The weighted average
22
21
22
2
21
1
2
22
22
21
212
110
)(,
)()()(
xx
xx
Consider the well-known weighted mean:
122
2
2
1
2
112
2
2
1
2
12
2
21
2
2
2
2
1
1
2
2
2
1
2
2
2
1
2
2
2
1
2
2
2
2
1
1
11xxx
xxxx
xx
A simple algebraic manipulation gives the recursive form(Kalman filter):
measured point weight matrix predictionKalman= the measurement is weightedwith a model prediction (track following)
Gaussian variables:
weighted average = Bayes (uniform) = Likelihood
41
42
20 events have been generated and 5 passed the cut What is the estimation of the efficiency with CL=90%?
Frequentist result:
Bayesian result:
=[0.104, 0.455]
=[0.122, 0.423]
x=5, n=20, CL=90%
90.0
)1(
)1(
1
0
2
1
d
d
xnx
p
p
xnx
1, 2
1, 2
What meaning??
05.0)1( 22
0
knk
x
k k
n
05.0)1( 11knk
n
xk k
n
Elementary example
x
43
n
fftfn )1(1
Efficiency calculation: an OPEN PROBLEM!!
1
)1(
4
1
22
2
2
2
2
n
tn
ff
n
tt
n
tn
tf
2/)1( 11knk
n
xk k
n
?
)1(
)1(
1
0
2
1 CL
xnx
p
p
xnx
d
d
2/)1( 22
0
knk
x
k k
n
Wilson interval (1934)
Bayes.This is not frequentistbut can be testedin a frequentist way
Wald (1950)Standard in Physics
Exact frequentistClopper Pearson (1934) (PDG)
Coverage simulation
44
x = gRandom → Binomial(p,N) → x
2/)1( 11knk
n
xk
ppk
n
2/)1( 22
0
knk
x
k
ppk
n
1-CL =
p
p1p2
Tmath:: BinomialI(p,N,x)
k++=k/n
p2p1
0ne expects ~ CL
45
Simulate many x with a true pand check when the intervals contain the true value p . Compare this frequency with the stated CL
CL=0.95, n=50
Simulate many x with a true p and check when the intervals contain the true value p . Compare this frequency with the stated CL
46
CL=0.90, n=20
47
In the estimation of the efficiency (probability) the coverage is “chaotic”
The new standard (not yet for physicists)is to use the exact frequentist or the formula
The standard formula
should be abandoned
111,/
)1(1],,0[,0
)1(],1,[,,
1
)1(
4
1
2
2/
/122
/111
2
2/
2
2
2/2/
2
2/
2
2/
is gaussian, t,α-CLtnxf
CLppx
CLppnx
n
t
n
ff
n
tt
n
t
n
tf
n
n
n
fftf
)1(BYE-BYE
48
A further improvement:The continuity correction is equivalent toThe Clopper-Pearson formula
1is1 1 gaussian,
,/)5.0(,/)5.0(
)1(1],,0[,0
)1(],1,[,,
1
)1(
4
1
2
2/
/122
/111
2
2/
2
2
2/2/
2
2/
2
2/
t,α-CLt
nxfnxf
CLppx
CLppnx
n
t
n
ff
n
tt
n
t
n
tf
n
n
This should become the standard formula also for physicists
Correzione di continuità
6.5 7 7.5
Quando una distribuzione discreta (come la binomiale) è approssimata con una continua(come la gaussiana), l’area del rettangolo centrato su un valore discreto x= della variabile, che rappresenta la probabilità P( ), può essere stimata con l’area sottesa dalla curva continua nell’intervallo [ 0.5≤x≤ +0.5]
Ciò equivale a considerare il valore discreto della variabile come il punto medio dell’intervallo [ 0.5 , +0.5]
La probabilità binomiale per x= =7è stimata mediante l’area sottesa dalla curva normale approssimantenell’intervallo [6.5 ≤x≤ 7.5]
Binomiale (istogramma a rettangoli)Gaussiana (curva continua)
Se la probabilità si riferisce a una sequenza di valori discreti - cioè è richiesta la
probabilità P( 1)+P( 2)+…+ P( k) - questa si può approssimare con la probabilità
gaussiana nell’intervallo [ 1 0.5 ≤x≤ k+0.5]
Più in dettaglio, ecco come operare la correzione di continuità per diversi valori cercati:
Valore della probabilità Intervallo su cui stimare
binomiale la probabilità gaussianaP(x= ) [ 0.5≤x≤ +0.5]
P(x≤ ) [x≤ +0.5]
P(x< ) [x≤ 0.5]
P(x≥ ) [x≥ 0.5]
P(x> ) [x≥ 0.5]
P( 1≤x≤ k) [ 1 0.5 ≤x≤ k+0.5]
P( 1<x< k) [ 1 0.5 ≤x≤ k 0.5]
L’approssimazione gaussiana alla binomiale si rivela particolarmente utile per n»1(fattoriali grandi nei coefficienti binomiali), e se si sommano le probabilità per una lunga sequenza di valori. La correzione di continuità consente di migliorare l’approssimazione
Esempio
a) Trovare la probabilità che esca Testa 23 volte in 36 lanci di una moneta
La distribuzione richiesta è una binomiale con n=36, p=q=1/2. La variabile è =23
Approssimazione di Gauss con =np=36 1/2=18 e =√npq=√36 1/2 1/2=3
Tenendo conto della correzione di continuità, i due valori della variabile standardizzatacorrispondenti agli estremi dell’intervallo di integrazione [22.5, 23.5] sono:
La probabilità binomiale è: P ( ) n!
!(n !)p q
n 36!
23!13!
1
2
231
2
13
3.36%
z122.5 18
31.50 z2
23.5 18
31.83
P 1( ≤z≤1.50 )=43.32%
P 2( ≤z≤1.83 )=46.64%
Pertanto la probabilità binomiale che esca Testa 23 volte, con l’approssimazionedi Gauss, può essere stimata in questo modo:
P ( ≤z≤1.83 )= P 2 P 1 =3.32%
Il valore è abbastanza vicino a quello calcolato esattamente con la binomiale
Nota: per un dato valore della variabile , la probabilità con l’approssimazione di Gauss si può anche stimare dal valore assunto dalla densità normale per x=
Il valore di probabilità di una distribuzione continua per uno specifico valore della variabile è zero, come abbiamo visto. In questo caso però stiamo semplicemente approssimando il valore di una distribuzione discreta (la binomiale) con il valoreassunto per quel valore dalla curva normale.
b) Trovare la probabilità che esca Testa almeno 23 volte in 36 lanci
Con l’approssimazione di Gauss, dobbiamo trovare la probabilità normale nell’intervallo [22.5, ∞], che stima la probabilità binomiale che Testa esca 23 o più volte
P (z≥1.5)=50% P ( ≤z <1.5)=50% 43.32%=6.68%
Pbin( ) fG(x )
1
2e(x )
2/ 2
2
Pertanto possiamo porre:
Pbin( )
1
3 2e(23 18)
2/ 2 3
2
0.1330 0.2494 0.0332 3.32%
Nel caso dell’esempio:
che coincide con il valore trovato valutando l’area nell’intervallo [22.5, 23.5]
53
N=50 CL=0.90
54
N=50 CL=0.95
The likelihood ratio method
55
),(
),(ln2),(ln2
),(
),(),(
xpL
xpLxp
xpL
xpLxp
best
best
Maximize
Minimize
56
Binomial Coverage simulationmax likelihood constraint
nk k
Feldman & Cousins, Phys. Rev. D 57(1998)3873
UNIFIED method
p
fxN
p
fxNp
1
1ln)(ln2),,(ln2
),,(ln2),,(ln2and,0|
,)1()!(!
!
xNpkNpkkA
CLppkNk
NkNk
Ak
57
N=50 CL=0.90
58
N=50 CL=0.95
59
N=20 CL=090
The problem persists alsowith large samples!
60
0.90
0.95
0.86
61
N=20 CL=0.90
62
From coin tossing to physics:the efficiency measurement
Valid also for
k=0 and k=n
ArXiv:physics/0701199v1
63
N=20 CL=0.90 Interval amplitude
likelihood
frequentist
Wilson cc
Wilson
x
64
N=20 CL=0.90 Interval limits
x
65
(2001) n
fftfn )1(1
2/)1( 22
0
knk
x
k k
n
2/)1( 22
0
knk
x
k k
n1
)1(
4
1
22
2
2
2
2
n
tn
ff
n
tt
n
tn
tf
66
xtxx
Counting experiments: Poisson case
?
!
!
0
2
1 CL
x
e
x
e
x
x
d
d
Wilson interval (1934)
Bayes.This is not frequentistbut can be testedin a frequentist way
Wald (1950)Standard in Physics
Exact frequentistClopper Pearson (1934) (PDG)
42
)( 22 txt
txt
x
2/!
2
0
2 ek
x
k
k
2/!
11 e
kxk
k
67
Poissonian Coverage simulation
CL=68%
68
Poissonian Coverage simulation
CL=90%
The Neyman-FC integrals
70
xq2q1
Neyman integrals
),(
),(ln2),(ln2
xL
xLx best minimize
),(ln2),(ln2and,0|
,);(
xkkkA
CLxp
Ak
71
Poissonian Coverage simulationmax likelihood constraint
CLekAk
k
!
nk k
Feldman & Cousins, Phys. Rev. D 57(1998)3873
nnn
nen
nen
nn
n
ln)(!/
!/ln2),(ln2
72
Poissonian Coverage simulation
en
ek
kACLek
nk
Ak
k
!!:
!
CL=68%
73
Poissonian Coverage simulation
CL=90%
74
Counting experiments: new formula for the Poisson case
Wilson interval with Continuity correctiongives the same results as …
Exact frequentistClopper Pearson (1934) (PDG)
5.042
)( 22
xxt
xtt
xtx
2/!
2
0
2 ex
x
k
x
2/!
11 exxk
x
75
76
77
The Unitarity Triangle
d s b
u
c
t
V V V
V V V
V V V
ud us ub
cd cs cb
td ts tb
0***tbtdcbcdubud VVVVVV
*
*
td tb
cd cb
V V
V V
*
*
ud ub
cd cb
V V
V V
1
B=(1,0)
C=(0,0)
A=( , )
• Quark mixing is described
by the CKM matrix
• Unitarity relations on matrix
elements lead to a triangle
in the complex plane
IVVt
CP Violation
Luca Lista Statistical Methods for Data
Analysis
78
79
A Bayesian application: UTFit
• UTFit: Bayesian determination of the CKM unitarity triangle– Many experimental and theoretical inputs
combined as product of PDF– Resulting likelihood interpreted as Bayesian
PDF in the UT plane
• Inputs:
– Standard Model experimental measurements and parameters
– Experimental constraints
Luca Lista Statistical Methods for Data
Analysis
80
Combine the constraints
• Given {xi} parameters and {ci} constraints
• Define the combined PDF– ƒ( ρ, η, x1, x2 , ..., xN | c1, c2 , ..., cM ) ∝
∏j=1,M ƒj(cj | ρ, η, x1, x2 , ..., xN) ∏i=1,N ƒi(xi)⋅ ƒo (ρ, η)
– PDF taken from experiments, wherever it is possible
• Determine the PDF of (ρ, η) integrating over the remaining parameters– ƒ(ρ, η) ∝
∫ ∏j=1,M ƒj(cj | ρ, η, x1, x2 , ..., xN) ∏i=1,N ƒi(xi)⋅ ƒo (ρ, η)
A priori PDF
=
=
Statistical Methods for Data
Analysis
81
Unitarity Triangle fit
68%, 95%
contours
Luca Lista Statistical Methods for Data
Analysis
82
PDFs for and
Luca Lista Statistical Methods for Data
Analysis
83
Projections on other observables
84
A Frequentist application: RFit
• RFit: to choose a point in the plane, and ask for the best set of the parameters for this points. The 2
values give the requested confidence region.
• No a priori distribution of parameters is requested
85
)();(ln2)(
2
1ln 2
2
2)(
2
1 2
xLx
ex
86
87
88
89
Conclusions• The usual formulae used by physicists in counting
experimets should be abandoned
• By adopting a practical attitude, also bayesianformulae can be tested in a frequentist way
• frequentism is the best way to give the resultsof an experiment in the form
x but other forms are also possible
• physicists should use Bayes formulae to parametrizethe previous (th or exp)
knowledge, not the ignorance
90
Quantum Mechanics: frequentist or bayesian?Born or Bohr?
dx2||
The standard interpretation is frequentist
END
91
92
x
Neyman integrals
Bootstrap );(1);( xFxF
Search for pivotal variables
This method avoids the graphic procedure and the resolution of the Neyman integrals
93