numi minos far detector installation, commissioning, & operation jeff nelson south carolina...
TRANSCRIPT
NUMI
MINOS
Far Detector Installation, Commissioning, & Operation
Jeff Nelson
South Carolina Meeting March, 2003
NUMI
MINOS
USC3/03Page 2
Outline
•FD Assembly>Planes, staffing, & status>Shield update
•FD Commissioning>Front ends, LI, DAQ, DCS, Magnet>Documentation update>Data quality & operations update>Electronic echoes (Roy Lee’s talk)
•Planning for shifts & operations•CDMS 2 - progress worth noting
NUMI
MINOS
USC3/03Page 3
J. Meier
1/30/03
NUMI
MINOS
USC3/03Page 4
Plane Assembly
• 426 planes installed
> 86% completed, 4.75 kton
> Averaging 5.5 PPW for SM2(was 6.4 in SM1) due to
> Directly related to FTEs worked
• Vacation, departures, just-in-time delivery mode for much of SM2
• Some numbers
> Steel at Soudan: 90%
> Modules at Soudan: 94%
> Over 1100 loads down shaft
> 60 planes to go
> 85th week of installation
FD Plane Installation
0
1
2
3
4
5
6
7
8
9
Jul-
01
Aug
-01
Sep
-01
Oct
-01
Nov
-01
Dec
-01
Jan-
02
Feb
-02
Mar
-02
Apr
-02
May
-02
Jun-
02
Jul-
02
Aug
-02
Sep
-02
Oct
-02
Nov
-02
Dec
-02
Jan-
03
Feb
-03
Mar
-03
Apr
-03
May
-03
Pla
ne
s p
er
we
ek
Working Schedule Planes/wk
Planes/Week (Monthly Averages)
NUMI
MINOS
USC3/03Page 5
Schedule Projection
• We are planning our schedule based on the plane construction completing June 2nd
>Current projection is May 28th
>Future based on some additional departures…
NUMI
MINOS
USC3/03Page 6
Staffing & Future Projections
• Crew in flux> 7 departures since last summer (of the 22 original
builders/movers)> We have hired 3 replacements > 3 more temps will start in March
• Expect to lose more to the mines> 50 more hires at just one mine this month> Many of the new workers are from construction trades
• May leave when the road restrictions lift this spring• Some may leave sooner as construction starts
> Could be as many as 5 to 7 more departures• There will also be a substantial demobilization by
materials handling crew after the last steel/scintillator is underground
• This leaves a +2/-1 week uncertainty in projections
NUMI
MINOS
USC3/03Page 7
More Schedule
• Roughly 2 months to complete the coil, shield, & final detector shake down
• After the last large objects are removed in very early June we will go to a days-only schedule> Maybe 9-hour days summer and maybe 8 hours days
fall/winter> Significantly reduced crew staffing next summer
NUMI
MINOS
USC3/03Page 8
Scinticicles
• Scintillator shipped in early February had some problems> Some of the skins delaminated
> Some strips show difference in readout
> These were the first modules shipped in very cold weather
• 19 modules from that shipment mapped again> 4 modules with problems (15 strips)
• 2 connector, 4 manifold, 7 fibers unglued (2 more unclassified)
• Ones with strange initial maps appear more vulnerable
> Some may be installed before we noticed
> Have to compare LI to maps to know
• These are the first major differences in scintillator response observed at Soudan> Now sample mapping and transmission
testing CIT modules from Dec and late Feb shipments
NUMI
MINOS
USC3/03Page 9
Some strips in the worst module
NUMI
MINOS
USC3/03Page 10
Commissioning
• 417 planes reading out>Last commissioning on Friday>Mostly routine at this point
NUMI
MINOS
USC3/03Page 11
Shield Update
• Since December we commissioned the revised Section 1 shield> Shorter cables and
repositioned modules• Added lower walls to
sections 1 & 2 (next page)• Future work on hold until
funding situation is clarified• When/if OK’d
> Can install section 3 now> Can reconfigure section 2
as double layer• See Caius’s talk for the
justification
NUMI
MINOS
USC3/03Page 12
Shield Lower Wall Installation
NUMI
MINOS
USC3/03Page 13
FD DAQ/Rack Hardware
• Crate commissioning> Crate 15 (the last crate)
• All the marginal bits ended up here• Tass finally debugged
> Some damage in things commissioned but not used for a few months
• Swapped with spares (aka ND DAQ)• Not sure the effect• Significant modifications to the crate procedures (pull cards for
extensive work; sign off before power)
• Branches (PVICs)> Vendor has a hardware fix to the node/length problem> Swap through spares for upgrades after an initial test of a
four node branch at Soudan> Months and a number of expert visits
before we convert back to 4 branches
NUMI
MINOS
USC3/03Page 14
DAQ Software at SoudanAreas of Emphasis
• ROP configuration overrides (Tim&Tass)> Override of dynode threshold settings for a scan > Override of sparsification mode to allow unsparsified
running > Override of VARC trigger to allow untriggered running > Masks so that high rate special runs can be limited to a
sub unit of the detector readout
• LI configuration improvements (Lisa) > LI runs spanning many runs> Tuning the light for individual LEDs
• RunControl enhancements (Caius, et al) > Web status page improvements> Fix memory leak (the Sunday problem)
NUMI
MINOS
USC3/03Page 15
On the Horizon
•ROP reset and abort implemented through RC
•The ROP will also calculate sparsification thresholds on pedestal runs (3 widths above pedestal)
•RC control using new ROP exposed bits >Making a diagnosis run sequence can then follow
•New RC GUI with added feature>Upgraded web status page
NUMI
MINOS
USC3/03Page 16
FD HV
• Hardware>Replaced the faulty ferrups
• Only three drop outs since January
>Replaced a HV card
• Software>Ramp changed to hardware ramp
• No plot yet
>Changes to warning/reset rules>Testing MySQL logging database
• See DCS section
NUMI
MINOS
USC3/03Page 17
FD DCS
• RPS>Replaced the last “old” units>Upgrades so fan pack monitoring works (7 of 16 done)>Adding smoke detectors for under the shield (per agreement
with code officials)>Change in logic for loss of power at VFBs
(error->warning)• Lost a fuse the next week… is it a good idea?
>Now have a command-line daemon to query the RPS units
• Enviro>Need to get north cluster running & prep for SM2 coil>Radon measurements differ wrt CDMS 2 at factor of two>Need to replace the north radon counter
• Coil>First field drop out happened last week
NUMI
MINOS
USC3/03Page 18
State of DCS Data
• Erik Beall and rest of DCS working to get the DCS data into it’s own data stream> Starting with Nat and the HV > Getting data into a local DCS DB> Exported into rotorooter / data stream> Working on the next step with Robert> Then expand across other subsystems
• Follow on projects> Web interface for shifters
• Current have to do a lot of work to check the DCS in a remote shift
– Kerberos/ssh for HV, RPS– VNC for coil & environmental– DAQ tells the grossest problems
> Making this data available in the offline (core group)• Some issues; see computing talk
NUMI
MINOS
USC3/03Page 19
FD LI – system tune up
• Test of a UV pulser box> Looks better
• See calibration summary
> Light level/focus issues
• LI light level tuning (Hartnell/Falk)> LIMaster modified to allow
different settings for each LED> Jeff has tuned the levels> Need to have a better algorithm
for spacing the steps
NUMI
MINOS
USC3/03Page 20
FEE Echoes & LI
• Can’t run like planned now• Need ~2 hours for one stability
point without data collection• See Roy’s talk on the progress• Would need to rethink calibration
mode if not fixable
NUMI
MINOS
USC3/03Page 21
LI as a Diagnostic Tool
• Starting to use LI as a diagnosis tool is was always meant to be
< BOX/LED REPORT >Missing Pulser Box 14 Completely.Missing Pulser Box 15 Completely.Very Quiet Box 12 Only 14464361 Associated Hits (59%)Very Quiet Box 13 Only 14434576 Associated Hits (59%)
< FEE REPORT >No Hits Beyond Plane 417Missing Crate --- Crate 14Missing Crate --- Crate 15Very Low Channel/Pixel ---- 0 2 2 0 1 16 -- Plane 55 Only 155 Hits ( 0%)Very Low Channel/Pixel ---- 2 2 3 1 0 12 -- Plane 122 Only 3117 Hits ( 19%)Very Low Channel/Pixel ---- 11 1 3 1 2 5 -- Plane 347 Only 0 Hits ( 0%)Low Chip ---- 12 1 1 1 2 -- Planes 403-405 Only 234789 Hits ( 91%)Very Low Channel/Pixel ---- 13 2 1 1 0 2 -- Plane 383 Only 207 Hits ( 1%)Very Low Channel/Pixel ---- 13 2 3 1 1 13 -- Plane 393 Only 40 Hits ( 0%)
NUMI
MINOS
USC3/03Page 22
Some Status Plots (After Recent Detector Sweep)
NUMI
MINOS
USC3/03Page 23
A More “Interesting” Plot(the week before)
NUMI
MINOS
USC3/03Page 24
Still some more remedial work
• A number of PIN problems• 5 single pixels• Few dead LI fibers
Dead Fibre East: 85 box 3 led 11 Num Wrong: Near 14 Far: 8Something wrong with East: 226-228 box 6 led 4 Num Wrong: Near 10 Far: 12Something wrong with East: 266-268 box 8 led 18 Num Wrong: Near 12 Far: 16Something wrong with West: 314-316 box 11 led 9 Num Wrong: Near 12 Far: 14Something wrong with East: 339-341 box 10 led 16 Num Wrong: Near 4 Far: 8Something wrong with East: 339-341 box 10 led 20 Num Wrong: Near 10 Far: 6Something wrong with East: 342-344 box 10 led 10 Num Wrong: Near 4 Far: 12Miswired Fibre East: 351 box 11 led 19 Num Wrong: Near 20 Far: 20Miswired Fibre East: 351 box 11 led 20 Num Wrong: Near 16 Far: 18Dead Fibre East: 363 box 10 led 7 Num Wrong: Near 14 Far: 8Dead Fibre East: 375 box 11 led 20 Num Wrong: Near 16 Far: 16Miswired Fibre East: 382 box 12 led 2 Num Wrong: Near 10 Far: 12Miswired Fibre East: 382 box 12 led 3 Num Wrong: Near 10 Far: 10Miswired Fibre East: 382 box 12 led 5 Num Wrong: Near 10 Far: 10Miswired Fibre East: 382 box 12 led 20 Num Wrong: Near 12 Far: 10Something wrong with East: 402-404 box 12 led 10 Num Wrong: Near 4 Far: 14Miswired Fibre East: 407 box 13 led 7 Num Wrong: Near 20 Far: 20Miswired Fibre East: 407 box 13 led 8 Num Wrong: Near 20 Far: 20
NUMI
MINOS
USC3/03Page 25
Noise Problems
• We are seeing a recurring but spreading noise problem
• The affected tubes are essentially dead
• Problem is the HV connector on the base needs to be reseated
• Recurring but new ones coming up too> Have made 3 passes looking for
problems this year> ~ half dozen every two weeks> 3 have been back each time
• Needs a fix to make sure these are not continuing to make trouble> Doug assigned to the IU MUX
guys
NUMI
MINOS
USC3/03Page 26
Update on Electronics Mortality(Dec-Feb)
Dec Jan Feb Tot VFB swaps 3++ 8+ 0 11 Base swaps 0 9** 5* 14 Cable swaps 1 1 0 2 (1 power, 1 control) + 2 cases swapping didn't solve the problem++ 1 case swapping didn't solve the problem * All old bases have low R21 (~1kohm)
** 2 cases the swapping didn't fix the problem &2 cases with errors while changing the base
NUMI
MINOS
USC3/03Page 27
Failure Rates
• At December Karol told us about the base problem being only being partially to blame for holes (~33% of bases pulled at FarDet and CalDet really bad in UT test stand)> Since then Mike led an effort to come up with a revised set of
diagnostic steps> Results are lower rate of false swaps> Base test stand for Soudan later this month
• Recently Alfons told us similar rates were seen for the VFBs (8 of 23)> A test stand for VFBs will be revived at Soudan to reduce the
number of false shipments
• Plans for future> FNAL PREP services all front end electronics> Test off detector at Soudan before shipping
NUMI
MINOS
USC3/03Page 28
Numbers & Spares
• We evaluated the spares with Roy & Alfons to ensure we have enough to run for many years of operations
• Total spares seem reasonable … to me• Table slightly out of date (Ca. early Feb)
I tem VARC VMM VFB VAs Ctrl Cable Pow Cable Sig Cable
MI NOS TOTAL 46 259 517 1551 532 519 519
Currently in detector (SM1+SM2) 46 255 409 1227 440 427 427
spares est. f or operat. 9 20 50 150 30 20 26
on hand; not installed 3 12 69 207 34 63 118
coming f rom CERN af ter const. 4 24 30 90 0 0 0
coming f rom CERN bef ore end of const. 3 19 30 90 0 0 0
pending / on order 0 0 30 90 100 50 0
needed 55 279 567 1701 562 539 545
total 56 310 568 1704 574 540 545
excess 1 31 1 3 12 1 0
NUMI
MINOS
USC3/03Page 29
FD SinglesRates per plane
NUMI
MINOS
USC3/03Page 30
Timing System
• There are two features that where originally advertised but are currently not implemented: >A precision oscillator in the TCU that would maintain
accurate absolute timing if the GPS clock fails >The option to generate execute signals with an
arbitrary phase with respect to the 40 MHz clock distributed
• Alfons would like to know if anyone finds the useful enough to pursue
• Had 1 bad TRC & 1 bad fiber in last 2 months
NUMI
MINOS
USC3/03Page 31
Magnetics
• Coil operates routinely>Had a supply trip last week
for the first time since August
• Alex Sousa making good progress on calibration control program>Still some details to work
on
• Cabling nearly up to date • Starting to plan for SM2
coil
NUMI
MINOS
USC3/03Page 32
Safety at Soudan
• We have had 1 safety report at Soudan since (Dec-Feb)>Recordable injury related to a previous injury
>Need to be extra careful with the new crew
>Reviewing & updating procedures in advance of a refresher course later this month
>Starting to cross train crew as the number of old timers ramps down
>Have distributed the new crew among veterans in the crews
NUMI
MINOS
USC3/03Page 33
0
2
4
6
8
10
12
14
16
18
20
A-0
1M
-01
J-0
1J-
01
A-0
1S
-01
O-0
1N
-01
D-0
1J-
02
F-0
2M
-02
A-0
2M
-02
J-0
2J-
02
A-0
2S
-02
O-0
2N
-02
D-0
2J-
03
F-0
3M
-03
A-0
3M
-03
J-0
3J-
03
A-0
3
Needed
Shifters + expert
PSL + L2
Physicists & Installation
• A running total of 999 physicist weeks or 20 physicist years at Soudan on installation tasks
• Getting there but not done
> We expect to be commissioning well into the summer
> There is also a period of training the operations crew later this summer
Monthly averages
Today
NUMI
MINOS
USC3/03Page 34
Needs at Soudan
•We need people (students, postdocs) at either full or 50% time to come up to speed as experts >Needed starting in mid to
late May>Very good summer
experience for students who are taking classes or just starting
>Needs for shift leaders or postdocs to help lead
>More senior visitors are welcome but already a few so we need to coordinate the dates
0
2
4
6
8
10
12
6-J
an
20
-Ja
n3
-Fe
b1
7-F
eb
3-M
ar
17
-Ma
r3
1-M
ar
14
-Ap
r2
8-A
pr
12
-Ma
y2
6-M
ay
9-J
un
23
-Ju
n7
-Ju
l2
1-J
ul
4-A
ug
18
-Au
g
NeededShifters + expertPSL + L2
NUMI
MINOS
USC3/03Page 35
Open House
• Soudan Open House scheduled for Saturday May, 2nd (the week before fishing opening) >Last year we had 650 visitors>Need about 10 tour guides
• Sign up with Bill this weekend
>More on outreach and summer opportunities/needs from Earl
NUMI
MINOS
USC3/03Page 36
Operations PlanStarting this Summer
• We are going to test operating for max live time (e.g. like we plan when in beam operations)
Goal is >>95%
• Our plan is to staff during 5 days shifts underground
• We will have two crew (+ hoistman) on call for underground access 24/7 in case of situations that are not remotely rectifiable
> Will be used by CDMS and MINOS
• We will operate with local shifts when underground and remote shifts (FNAL + UK?) for other hours
NUMI
MINOS
USC3/03Page 37
FD Documentation
http://www-numi.fnal.gov/cgi-bin/noteFDDocsPrint
• There are currently 26 documents submitted and tagged for the FD manual
Good guys LaggardsFront ends Environmental Magnetic calibrationOnline monitoring Call lists Safety Plane assembly Survey DAQPMT handling Magnet RPSLight injection HV
• Need these for training the operations crew this spring/summer
• Please increase the priority on this activity to ensure efficient operations this summer/fall> Please add in more detailed user’s (idiot’s) guides
NUMI
MINOS
USC3/03Page 38
Detector operations
• Since the summer we have been collecting data for calibration and cosmic rays
• We have also been working to increase the live time
• Addressing potential hardware impediments> HV monitoring & control
improved; hardware swaps> Optimizing automatic recovery
hardware & software
• Will continue to have 3-4 days per month for DAQ/RPS
• Hope to get back to some days without taking the detector offline
0%
10%
20%
30%
40%
50%
60%
70%
Feb
-02
Mar
-02
Apr
-02
May
-02
Jun-
02Ju
l-02
Aug
-02
Sep
-02
Oct
-02
Nov
-02
Dec
-02
Jan-
03Feb
-03
Live
time
-0.20.40.60.81.01.21.41.61.82.0
Kto
n-m
onth
s (fi
d)
Livetime Exposure
NUMI
MINOS
USC3/03Page 39
Exposure Summary
• Total exposure (since 1/31/02) 1.52 kty• Total fiducial exposure (“ “) 1.04 kty• Total fiducial ‘good’ exposure (“ “) 0.94 kty• SM1 with field (8/8/02) 0.76 kty• Old sec 1 (8/16-11/12/02) 0.18 kty • Sec 2 (11/27/02) 0.17 kty • Reconfig’d sec 1 (12/?/02) 0.13 kty• Lower shield walls (1/27/03) 0.09 kty
NUMI
MINOS
USC3/03Page 40
CDMS 2 Update (Part I) (from Dan Bauer)
First tower of detectors installed in “icebox” at SoudanSecond tower installed this week
Radonpurge
Tower 1
NUMI
MINOS
USC3/03Page 41
CDMS 2 Update (Part II) (from Dan Bauer)
• Dilution refrigerator and icebox were cooled to 20 mK in Dec, 2002
• Shielding, DAQ, and Electronics nearly finished
NUMI
MINOS
USC3/03Page 42
CDMS 2 Update (Part III) (from Dan Bauer)
• Deep site>Much lower neutron background >Rapid improvement in sensitivity
• Expect factor of 10 improvement over CDMS I results by end of 2003
• Deploy remainder of detectors in 2004 and run until end of 2005 (or longer)
• Detector performance improvements mean we should still reach original CDMS II goals
NUMI
MINOS
USC3/03Page 43
Zippy Big Picture Summary
• Now 7/8ths complete• Will finish this summer
>Need roughly 8 additional physicist-months late spring and summer
• Will shift into operations this summer>Advanced model for operations>Physicists at Soudan beyond summer
is an issue