g. cowan rhul physics page 1 status of search procedures for atlas atlas-cms joint statistics...

G. CowanRHUL Physics page 1

Status of search procedures for ATLAS

ATLAS-CMS Joint Statistics Meeting

CERN, 15 October, 2009

Glen CowanPhysics DepartmentRoyal Holloway, University of Londong.cowan@rhul.ac.ukwww.pp.rhul.ac.uk/~cowan

G. Cowan / RHUL page 2

IntroductionThis talk describes (part of) the view from ATLAS with emphasis on searches using profile likelihood-based techniques;using as an example the combination of Higgs channels described in

Expected Performance of the ATLAS Experiment: Detector, Trigger and Physics, arXiv:0901.0512, CERN-OPEN-2008-20.

Other methods are also being explored and the view presentedhere should not be taken as a final word on methodology.

Some issues are:Basic formalism, definition of significanceMethod used for setting limitsLook-elsewhere effect

G. Cowan / RHUL page 3

Search formalism

Define a test variable whose distribution is sensitive to whetherhypothesis is background-only or signal + background.

E.g. count n events in signal region:

events found

expected signal expected background

strength parameter= s/ s,nominal

G. Cowan / RHUL page 4

Search formalism with multiple bins (channels)

Bin i of a given channel has ni events, expectation value is

Expected signal and background are:

is global strength parameter, common to all channels.= 0 means background only, = 1 is nominal signal hypothesis.

btot, s, b arenuisance parameters

G. Cowan / RHUL page 5

Subsidiary measurements for background

One may have a subsidiary measurement to constrain the background based on a control region where one expects no signal.

In bin i of control histogram find mi events; expectation value is

where the ui can be found from MC and includes parametersrelated to the background (mainly rate, sometimes also shape).

In some measurements there may be no explicit subsidiarymeasurement but the sidebands around a signal peak effectivelyplay the same role in constraining the background.

G. Cowan / RHUL page 6

Likelihood functionFor an individual search channel, ni ~ Poisson(si+bi), mi ~ Poisson(ui). The likelihood is:

Parameter of interest

Here represents allnuisance parameters

For multiple independent channels there is a likelihood Li(,i) for each. The full likelihood function is

G. Cowan / RHUL page 7

Systematics "built in" as long as some point in -space = "truth". Presence of nuisance parameters leads to broadening of theprofile likelihood, reflecting the loss of information, and givesappropriately reduced discovery significance, weaker limits.

G. Cowan / RHUL page 8

Modified test statistic for exclusion limits

For e.g. data generated with , -2 ln () can come out large for

If , then data more compatible with a higher value of . sodo not include this in critical region.

For upper limit, test hypothesis that strength parameter is ≥ .

Upper limit is smallest value of where this hypothesis can be rejected at significance level less than 1CL.

Critical region of test is region with less compatibility withthe hypothesis than the observed

G. Cowan / RHUL page 9

Test statistic for exclusion limitsTherefore for exclusion limits, define the test statistic to be

critical region

Thus distribution of modified q corresponds to lower branchonly of U-shaped plot above.

For low , this distribution falls off more quickly than theasymptotic chi-square form and thus gives conservative limit.

G. Cowan / RHUL page 10

p-value / significance of hypothesized

Test hypothesized by givingp-value, probability to see data with ≤ compatibility with compared to data observed:

Equivalently use significance,Z, defined as equivalent numberof sigmas for a Gaussian fluctuation in one direction:

G. Cowan / RHUL page 11

Distribution of qSo to find the p-value we need f(q|) .

Method 1: generate toy MC experiments with hypothesis , obtain at distribution of q.

OK for e.g. ~103 or 104 experiments, 95% CL limits.

But for discovery usually want 5, p-value = , so needto generate ~108 toy experiments (for every point in param. space).

Method 2: Wilk's theorem says that for large enough sample,

f(q|) ~ chi-square(1 dof)

This is the approach used in the ATLAS Higgs Combination exercise; not yet validated to 5 level.

If/when we are fortunate enough to see a signal, then focus MC resources on that point in parameter space.

G. Cowan / RHUL page 12

Example from validation exercise: ZZ(*) → 4l

5 level

(One minus)cumulativedistributions.Band gives 68%CL limits.

½2

½2

½2

½2

Distributions of q0 for 2, 10 fb from MC compared to ½2

G. Cowan / RHUL page 13

Significance from q

If we take f(q|) ~ 2 for 1dof, then the significance is simply:

For n ~ Poisson (s+b) with b known, testing = 0 gives

To quantify sensitivity give e.g. expected Z under s+b hypothesis

G. Cowan / RHUL page 14

SensitivityDiscovery:

Generate data under s+b (= 1) hypothesis;Test hypothesis = 0 → p-value → Z.

Exclusion:Generate data under background-only (= 0) hypothesis;Test hypothesis .If = 1 has p-value < 0.05 exclude mH at 95% CL.

Estimate median significance (sensitivity) either from MC or by using a single data set with observed numbers set equal tothe expectation values ("Asimov" data set).

G. Cowan / RHUL page 15

Example of ATLAS Higgs searchCombination of Higgs search channels (ATLAS) Expected Performance of the ATLAS Experiment: Detector, Trigger and Physics, arXiv:0901.0512, CERN-OPEN-2008-20.

Standard Model Higgs channels considered: H → H → WW (*)→ eH → ZZ(*) → 4l (l = e, ) H → → ll, lh

Not all channels included for now; final sensitivity will improve.

Used profile likelihood method for systematic uncertainties:nuisance parameers for: background rates, signal &

background shapes.

G. Cowan / RHUL page 16

Combined discovery significance

Discovery signficance (in colour) vs. L, mH:

Approximations used here not always accurate for L < 2 fb but in most cases conservative.

G. Cowan / RHUL page 17

Combined 95% CL exclusion limits

p-value of mH (in colour) vs. L, mH:

G. Cowan / RHUL page 18

Comment on combination softwareCurrent ATLAS Higgs combination shows median significances

Obtained using median significances from each channel

What we will need is the significance one would have from asingle (e.g. real) data sample.

Requires full likelihood function, global fit → software.

Since summer 2008 ATLAS/CMS decision to focus joint statisticssoftware effort in RooStats (based on RooFit, ROOT).

Provides facility to construct global likelihood forcombination of channels/experiments

Emphasis on retaining modularity for validation byswapping in/out different components.

G. Cowan / RHUL page 19

The "look-elsewhere effect"Look for Higgs at many mH values -- probability of seeing a large fluctuation for some mH increased.

Combined significance shown here relates to fixed mH.

False discovery prob enhanced by ~ mass region explored / m

For H→ and H→WW, studied by allowing mH to float in fit:

→

G. Cowan / RHUL page 20

Summary / conclusions

Current philosophy (ATLAS/CMS) is to encourage a variety of methods, e.g., for limits: classical (PL ratio), CLs, Bayesian,...

If the results agree, it's an important check of robustness.If the results disagree, we learn something (~ Cousins)

Discussion on look-elsewhere effect still ongoing

Derive trials factor from e.g. MC or other considerationsFloating mass search

D0, CDF, CMS, ATLAS need to compare like with like.

Need discussions on e.g. formalism for discovery,limits, combination, treatment of common systematics,…

G. Cowan / RHUL page 21

Extra slides

G. Cowan / RHUL page 22

Fast Fourier Transform method to find distribution; derivesn-event distribution from that of single event with FFT.

Hu and Nielson, physics/9906010

Solves "5-sigma problem".

Used at LEP -- systematics treated by averaging the likelihoodsby sampling new values of nuisance parameters for each simulated experiment (integrated rather than profile likelihood).

An alternative (in simple cases equivalent) test variable is

Comment on "LEP"-style methods

G. Cowan / RHUL page 23

Setting limits: CLs

Alternative method (from Alex Read at LEP); exclude if

where

This cures the problematic case where the one excludes parameterpoint where one has no sensitivity (e.g. large mass scale)because of a downwards fluctuation of the background.

But there are perhaps other ways to get around this problem,e.g., only exclude if both observed and expected p-value .

G. Cowan / RHUL page 24

G. Cowan / RHUL page 25

Some issuesThe profile likelihood method "includes" systematics to theextent that for some point in the model's parameter space, thedifference from the "truth" is negligible.

Q: What if the model is not good enough?

A: Improve the model, i.e., include additional flexibility (nuisance parameters).

Increased flexibility → decrease in sensitivity.How to achieve optimal balance in a general way is not obvious.

Corresponding exercise in Bayesian approach:

Include nuisance parameters in model with prior probabilities -- also not obvious in many important cases,

e.g., uncertainties in correlations.