team members description of the te-mpe stand-by service statistics 2010 + 2011 seu
DESCRIPTION
TE-MPE Stand-By Service 2010 – 2011 lessons learnt Gert Jan Coelingh on behalf of TE-MPE Stand-By team,. Team members Description of the TE-MPE Stand-By Service Statistics 2010 + 2011 SEU DQHDS main switch “IN THE SHADOW” Personal touch Extrapolate 2011 – Improvements? - PowerPoint PPT PresentationTRANSCRIPT
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
1
TE-MPE Stand-By Service 2010 – 2011 lessons learnt
Gert Jan Coelingh on behalf of TE-MPE Stand-By team,
Team members
Description of the TE-MPE Stand-By Service
Statistics 2010 + 2011
SEU
DQHDS main switch
“IN THE SHADOW”
Personal touch
Extrapolate 2011 – Improvements?
Conclusion & Outlook
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
2
Team Members
– Giorgio D'Angelo EI ELQA & Heating Syst. C.L
– Richard Mompo EI ELQA & Heating Syst. C.L
– Grzegorz Seweryn EI ELQA & QPS Heating Syst. C.L
– Steven Pemberton EI ELQA & Heating Syst. C.L
– Kevin Priestnall CP QPS – Retirement 2011
– Vincent Froidbise CP QPS
– Joaquim Mourao CP QPS
– Gert-Jan Coelingh CP 600A EE, 13 kA & QPS
– Help of Mathieu Favre, Noel Fournier and Bozhidar Panev (CP)
Reiner is responsible of the service, and Gert-Jan is the linkman
Stand-By members:
In 2011, 8 members, with specific knowledge:
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
3
Team Members: Experts
Specialists: support for Stand-By members
Very limited specialists (in number ):
– Knud Dahlerup-Petersen (EE & QPS)
– Reiner Denz (QPS)
– Zinur Charifoulline (nQPS informatics).
– Jens Steckert (nQPS).
• CP section: Other specialists on Stand-By
– Current leads specialist all specialists on Stand-By
Now under EI responsibility;.
– DFB specialists: are part of Stand-By members, Mateusz Bednarek since November 2010!!
– MP3 members Stand-By
• All Experts are on BEST-EFFORT
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
4
MPE Stand-By mandate:
1. Mission:
– To put back in service, as rapid as possible, a faulty installation within the framework (QPS, Energy Extraction Systems and electrical components of the DFBs) during LHC run.
2. Actions:
– Intervene after a call from the Control Centre CERN (CCC),
– Diagnostics and if possible solve the problem remotely
– Organize the intervention with the operator/EIC CCC (RP)
– If needed ask the “Expert” (if reachable) for support
– Archive the intervention in the Logbook (QPS Logbook).
3. Responsibility:
– He must assure his own security as well as those he asked helping him and assure the correct way of intervening in the tunnel
– The technician is responsible for the intervention. If he cannot find a solution he must inform the group hierarchy.
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
5
MPE Expert mandate (1/2)
The Expert (based on best effort) :
1. Mission:
– Support the first line intervention in case of complex problems
2. Actions:
– If judged intervene to help the Stand-By technician
– Ask another Stand-By team member to join for intervention
3. Indirectly related actions:
Decrease LHC machine down-time where possible
– Implement solutions for problems
Firmware upgrades
Preventive maintenance
DQHDS main switch exchange campaign
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
6
MPE Expert mandate (2/2)
3. Indirectly related actions (cont.):
– Not to be underestimated in hours and workload
– Example:
Arkadiusz Gorzawski developed an interface
logging database – 600A EE signals
- permits preventive maintenance on breakers with too many closing failures
- Currently extended with intervention history per system
Direct result = no breaker exchange so far this year
HUGE EFFORT not visible in Stand-By statistics
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
7
MPE Stand-By organization:
Documentation:
– Each equipment specialist prepared a documentation on how to solve a typical problem.
– An updated copy is always available in two Stand-By cars
Regular meeting and training:
– Weekly meeting on Monday morning to transmit the “Piquet role” and share experience over the past week.
– Training are organized in order to be as multidisciplinary as possible.
– Two Stand-By cars available, equipped with tooling, docs and spare parts.
– Storage place in Build.30, 281 and 936 (in front of CCC) Yearly planning:
On our repository, we have a yearly planning with foreseen vacations, official holidays, night and day “Stand-By” turn-over.
– Day calls: transparent to CCC (same number 16.8801), assures intervention during normal hours. This allow night piquet to recover.
– Night calls: outside normal hours and w-e, official holidays included.
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
8
Intervention Flowchart
Emergency call from CCC to Stand-By duty:Stand-By: 16.8801
Emergency call from CCC to Stand-By duty:Stand-By: 16.8801
End of intervention
End of intervention
N
Stand-By person to analyze
max. delay (1hour)
Stand-By person to analyze
max. delay (1hour)
Information exchange via telephone
Stand-By / CCC operatorRemote Analysis
Information exchange via telephone
Stand-By / CCC operatorRemote Analysis
Need more than onePerson?
Assistance via:Stand-By support
Operator CCC*
Assistance via:Stand-By support
Operator CCC*
N
Y
Y
Repair and restart the equipment or,
feedback situation with CCC.
Repair and restart the equipment or,
feedback situation with CCC.
Call equipment Specialist or Responsible TE/MPE
Call equipment Specialist or Responsible TE/MPE
Problem Solved remotely?Stand-By / CCC
Difficult problem or
Time needed > 3 hours ?
N
Repair and
restart the equipment
Repair and
restart the equipment
Y
*(en dehors des heures ouvrables uniquement )
Fill in QPS Logbook
Fill in QPS Logbook
End of intervention
Fill in QPS Logbook
Fill in QPS Logbook
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
9
TE-MPE Stand-By service for the LHC
Gert Jan Coelingh on behalf of TE-MPE Stand-By team,with inputs from R. Denz
Team members
Description of the TE-MPE Stand-By Service
Statistics 2010 + 2011 SEU DQHDS “IN THE SHADOW” Personal touch Improvements Conclusion & Outlook
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
10
66
151.5
41
50
33
18
0
20
40
60
80
100
120
140
160
180
200
220
240
Total number of interventions Total duration [hours]
Interventions from MPE Piquet in 2010 (39 Weeks)
Outside normal working hours Inside normal working hours Remote (day & night)
Some statistics from 2010/2011: (1/..)
Total number of interventions (Remote / Local) and Total duration
140 Interventions
219 hours
87 Interventions87 hours
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
11
Some statistics from 2010/2011: (2/..)
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
12
Some statistics from 2010/2011: (3/..)
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
13
Some statistics from 2010/2011: (4/..)
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
14
Some statistics from 2010: (5/..)
26
19
12
8
20
65
43
37
6076 10438 1624 2532 2532 202 32 32 1298
Quench heater power supplies
Quench detection systems
DAQ caused by radiation (SEU)
DAQ other causes than radiation
DAQ all faults combined
EE600 EE13 kA DFB Instrumentation
Current Lead Heating system
Others
Faults distribution per equipment type
0
5
10
15
20
25
30
35
40
1819
17
22
39
1 12
1
6
Faults up to June 20, 2011
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
15
TE-MPE Stand-By Service 2010 – 2011 lessons learnt
Gert Jan Coelingh on behalf of TE-MPE Stand-By team,
Team members
Description of the TE-MPE Stand-By Service
Statistics 2010 + 2011
SEU
DQHDS
“IN THE SHADOW”
Personal touch
Improvements
Conclusion & Outlook
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
16
SEU – increasing number
17 treated by Stand-by team so far this year
– Symptoms: QPS_OK lost (fill can be continued until beam dump)
• continuous sending PM files or loss of communication
– Action:
• locally power cycle the crate – (and depending on time of intervention)
• Update locally the firmware or– change board for one with new firmware)
– Specialists actions: upgrade of firmware
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
17
TE-MPE Stand-By Service 2010 – 2011 lessons learnt
Gert Jan Coelingh on behalf of TE-MPE Stand-By team,
Team members
Description of the TE-MPE Stand-By Service
Statistics 2010 + 2011
SEU
DQHDS main switch
“IN THE SHADOW”
Personal touch
Improvements
Conclusion & Outlook
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
18
DQHDS main switch
18 cases treated by Stand-by team so far
– 26 treated in 2010 – we are NOT better in number but much better in time of intervention; almost always IN THE SHADOW causing no downtime to LHC operation
1650 (26%) out of 6200 have been replaced
– All DQHDS of all MQs done and regular campaigns on-going
• Software interlock adapted to MB. One out of four DQHDS of a MB can be down. No immediate intervention needed.
• Why not better in numbers? …
» plastic is getting too hard?? We should hurry up
No faults in the new switches
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
19
TE-MPE Stand-By Service 2010 – 2011 lessons learnt
Gert Jan Coelingh on behalf of TE-MPE Stand-By team,
Team members
Description of the TE-MPE Stand-By Service
Statistics 2010 + 2011
SEU
DQHDS
“IN THE SHADOW”
Personal touch
Improvements
Conclusion & Outlook
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
20
« IN THE SHADOW »
30 out of 87 interventions “IN THE SHADOW”
• less downtime for LHC operations
Mainly because
– of upgraded Interlocking software in case of a MB DQHDS failure.
– SEU can finish the fill before power cycling the crate
– Preventive maintenance
– Coordination between Equipment Responsible and Machine Coordination in case of longer downtime
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
21
TE-MPE Stand-By Service 2010 – 2011 lessons learnt
Gert Jan Coelingh on behalf of TE-MPE Stand-By team,
Team members
Description of the TE-MPE Stand-By Service
Statistics 2010 + 2011
SEU
DQHDS
“IN THE SHADOW”
Personal touch
Improvements
Conclusion & Outlook
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
22
Personal Touch (updated until June 20)
3 x
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
23
TE-MPE Stand-By Service 2010 – 2011 lessons learnt
Gert Jan Coelingh on behalf of TE-MPE Stand-By team,
Team members
Description of the TE-MPE Stand-By Service
Statistics 2010 + 2011
SEU
DQHDS
“IN THE SHADOW”
Personal touch
Extrapolate 2011 – Improvements?
Conclusion & Outlook
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
24
Extrapolate 2011 – improvement?
+170%
+178%
0%
+17%
- 21%
- 41%
189 interventions = +35% 189 hours = -14%
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
25
TE-MPE Stand-By Service 2010 – 2011 lessons learnt
Gert Jan Coelingh on behalf of TE-MPE Stand-By team,
Team members
Description of the TE-MPE Stand-By Service
Statistics 2010 + 2011
SEU
DQHDS main switch
“IN THE SHADOW”
Personal touch
Extrapolate 2011 – Improvements?
Outlook
Conclusion
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
26
Outlook:
Improvement ongoing:
– TUNNEL INTERVENTIONS
• Upgrades on iQPS firmware on SEU
• Switches DQHDS replacement – 1600 done + 500 4-9 July
– Software tools on QPS
• Remote restart – done
• Upgrade Firmware of new Busbar detector 4-9 July
• Interlocks regarding DQHDS – interventions “in-the-shadow”
Potential bottlenecks:
– More radiation to equipment SUE
– Resources are limited, both experts and technicians
– MP3 Stand-By canceled in 201x? Who takes over?
– Back-up for Stand-By team based on best effort
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
27
Conclusion :
Conclusion so far as in 2010
– Very efficient: < 2 hours/intervention on site, including access time !!
– High reliability of the systems
– No accident !
Stand-By service interventions seems to be decrease in numbers of hours
BUT..
A lot of work behind the scenes to get there
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
28
MPE STAND-BY on DFS
All information about the Stand-By service can be found on the DFS folder:
Dfs:\\cern.ch\Departments\TE\Services\MPE_on-call-service
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
29
QUESTIONS ?
THANK YOU !!!
LPCT
Local Power Cycle Tool
The absence of safety shoes and helmet is not representative for the MPE Stand-By team
TE
-MP
E,
Gro
up M
eetin
g -
G.J
. C
oelin
gh -
23
Jun
e 20
11
30
Buffer zone for pieces from LHC machine
Les pièces sont contrôlées par un
technicien DGS/RP
Sécurité Radiologique Opérationnelle: Lignes directrices générales• Tout matériel quittant les zones de rayonnement CERN doit être considéré comme potentiellement radioactif,• Tout matériel doit être contrôlé par DGS/RP,• Seul DGS/RP, est autorisé à déclarer du matériel comme non radioactif,• En cas de doute, contactez un technicien DGS/RP
Repair work shop
TE/MPEBat. 30
Bat. 281Bat. 287External
Zone de Transit
TE/MPE for non radio
active pieces
Bat. xx
Labo de Réparation dédié aux
équipements faiblement radioactifs
TE-EPC
Who?: Equipment responsible
1) Routing of faulty pieces from a TE/MPE installation towards a repair workshop
LHC Machine (Zone Opérationnelle
contrôlée)
LHC Machine (Zone Opérationnelle
contrôlée)
UA(Zone Machine
Supervisée)
UA(Zone Machine
Supervisée)
When: Weekly, when lower limit is reached
-
RR/UJ(Zone Opérationnelle
contrôlée)
RR/UJ(Zone Opérationnelle
contrôlée)When?: Only in cases of an intervention
-
Quand?: Uniquement lors d’une intervention
Quand?: Uniquement lors d’une interventionWhen?: Only in cases of an intervention
Radioactif ?
Yes
Broken Modules
Rack
Who?: MPE on call Technician
Who?: MPE on call Technician‘Picomur’
‘Picomur’
‘Picomur’
When?: Only in cases of an intervention
Who?: MPE on call Technician