intervention priority management this talk will show the cern priority list, the corresponding check...
TRANSCRIPT
Intervention Priority
Management This talk will show the CERN priority list, the
corresponding check list and the tools used by operators to diagnose a situation and to manage
the co-ordination of activities.
P. Sollander - ARW Cape Town 2
Intervention Priorities• Technical infrastructure operation• Big trouble -- Major power outage• Problems• Solutions• Experience, current situation and outlook
14/4/11
P. Sollander - ARW Cape Town 3
OperationCER
N
BE
OP
LHC
SPS
TI
…
EN
EL
OP
CV
OP
TE
CRG
OA
OD
GS
…
IT
…
14/4/11
P. Sollander - ARW Cape Town 4
• It is the mission of the technical infrastructure section to minimize the impact of technical breakdowns on accelerators and other important installations at CERN. The role of the section is to manage corrective maintenance activities and to co-ordinate interventions during major breakdowns, …
TI mandate
14/4/11
P. Sollander - ARW Cape Town 5
• It is the mission of the technical infrastructure section to minimize the impact of technical breakdowns on accelerators and other important installations at CERN. The role of the section is to manage corrective maintenance activities and to co-ordinate interventions during major breakdowns, …
TI mandate
14/4/11
P. Sollander - ARW Cape Town 6
• It is the mission of the technical infrastructure section to minimize the impact of technical breakdowns on accelerators and other important installations at CERN. The role of the section is to manage corrective maintenance activities and to co-ordinate interventions during major breakdowns, …
TI mandate
14/4/11
P. Sollander - ARW Cape Town 7
• It is the mission of the technical infrastructure section to minimize the impact of technical breakdowns on accelerators and other important installations at CERN. The role of the section is to manage corrective maintenance activities and to co-ordinate interventions during major breakdowns, …
TI mandate
14/4/11
P. Sollander - ARW Cape Town 8
• Operation 24*365• One operator on shift (+1 during shutdown)• Technician, 2 year university or equivalent• On average, 3 years of experience• Supervision of a large number of systems
o Electricity, cooling, ventilation, safety detection systems, lifts, computers, …
o For all accelerators, experiments and important facilities
TI operation
14/4/11
P. Sollander - ARW Cape Town 9
CERN accelerators
14/4/11
Alarms1’080’087 in 2010
Phone Calls150-300/day
TI First Line 4%
Contractor & CERNInterventions
Work Orders6960 in 2010
14/4/11 10P. Sollander - ARW Cape Town
Corrective maintenance
P. Sollander - ARW Cape Town 11
Technical Infrastructure desk in
the CCC
14/4/11
P. Sollander - ARW Cape Town 12
Big Bad Breakdowns
11/4/11
P. Sollander - ARW Cape Town 13
Thunderstorm
14/4/11
P. Sollander - ARW Cape Town 1414/4/11
P. Sollander - ARW Cape Town 15
Panic!
14/4/11
P. Sollander - ARW Cape Town 16
• CERN panel for intervention prioritieso Set up in 2008o Representatives from different departmentso Mandate to decide on priorities
• Provide a logically structured list of the facilities at CERN and examine the criticality of the equipment and systems in each facility.
• Arrange the list in order of priority of intervention as a function of time and as a function of the operational cycle of the accelerator complex and computing facilities.
• Provide new versions of the priority list as needed to the technical infrastructure operations team.
We need a plan!
14/4/11
P. Sollander - ARW Cape Town 17
Improve communication
• Installation of new Call Centre to handle callso Queues incoming callso Voice message explaining situationo Allows VIP list
• Technical Infrastructure Operation information feedo Up-to-date information on ongoing
interventionso Works if networks and servers are up!
• Information meeting same dayo Exchange information between
operation, equipment groups and users
14/4/11
P. Sollander - ARW Cape Town 18
• Agreed by panel• Can change as often as needed• Contains high level priorities to check, today:
1. Safety systems2. Environmental Protection3. Computer Centre4. Equipment protection
• LAr cryo, LKr cryo, RF electronics, …5. LHC cryogenics6. ATLAS7. CMS8. Accelerator backbone9. CNGS target areas10. ALICE11. LHCb12. …
Priority list
14/4/11
P. Sollander - ARW Cape Town 19
• Help; single operator not enough• Priority list• For each item on the priority list
o A means to know current stateo Information on what to do in case of problem
Need for a comprehensible CHECK LIST
TI operator needs
14/4/11
P. Sollander - ARW Cape Town 20
Checklist
14/4/11
P. Sollander - ARW Cape Town 21
• For each item on listo Verificationo Actionso Contactso Help Information
Checklist
14/4/11
P. Sollander - ARW Cape Town 22
High level monitoring
14/4/11
P. Sollander - ARW Cape Town 23
High level monitoring
14/4/11
P. Sollander - ARW Cape Town 24
Experience and Outlook
• Lists in use since 2008• Priorities change very little• (Fortunately) not often used, but very few opportunities for
training• Priority list useful also for smaller incidents
o Avoids overlooking anything, should be used every time
• Learning by checklist creationo Verification, action, contact: many clarifications obtained.
• Supervision and Monitoring o Still being implementedo Distills data AND ….. errors
• Perfectible but necessary, even partly complete• Escalation, management, outside communication not included
here, dealt with elsewhere.• Future vision of an “animated check-list”
o Very High Level Monitoring
14/4/11
P. Sollander - ARW Cape Town 25
VHLMElectrical networks
Safety systems
Environmental Protection
Computer Centre
14/4/11
P. Sollander - ARW Cape Town 26
Summary• TI operation: 1 technician does it all• Big trouble: major power outage, 1/year• Problems: communication, priorities, tools• Solutions: call centre, checklist, high level
monitoring• Experience: useful, priorities easy to find,
checklist difficult to put in place, not enough training possible for the time being
14/4/11
Questions?
P. Sollander - ARW Cape Town 28
Alarms by year
14/4/11
2003 2004 2005 2006 2007 2008 2009 20100
500000
1000000
1500000
2000000
2500000
961650
678073 674847 733969
1027944
1930289
1080087
Yearly alarm progression
Alarms over the years
The alarms are not including vacuum alarms, nor alarms from the electrical alarm system (ENS)
P. Sollander - ARW Cape Town 29
Phone calls 2010
14/4/11
1/1/
10
1/14
/10
1/27
/10
2/9/
10
2/22
/10
3/7/
10
3/20
/10
4/2/
10
4/15
/10
4/28
/10
5/11
/10
5/24
/10
6/6/
10
6/19
/10
7/2/
10
7/15
/10
7/28
/10
8/10
/10
8/23
/10
9/5/
10
9/18
/10
10/1
/10
10/1
4/10
10/2
7/10
11/9
/10
11/2
2/10
12/5
/10
12/1
8/10
12/3
1/10
0
100
200
300
400
500
600
700
Christmas closureTechnical stopRun