IceCube Project Monthly Report – January 2009

Accomplishments • January was an extremely productive month for drilling and string installation. Nine holes

were safely drilled in January resulting in a total of 19 strings by January 21st.

• A newly implemented software model provides real time forecasts of the expected lifetime of holes. This tool greatly improves the planning for deployment of strings and special devices.

• All EHWD systems were flushed and dried, disconnected and relocated to the prepared site for next season. The Main Supply Hose Reel was wrapped with heating blankets.

• Five firn holes were completed for the 2009 - 2010 season, including the RodWell.

• High level commissioning of the newly deployed strings started in late January. As of the end of January, 13 of the 19 new strings were checked for timing and geometry calibration.

• Data collection during January achieved an overall 94.1% uptime and 84.5% in-ice clean run time, lower than normal but high considering the large amount of activity during the season.

• The PY8 plan (April 1, 2009 – Mar 31, 2010) was completed and approved.

A train of 5 MDS containers is pulled by a D6 to the site for next season.

(Photo: Tom Hutchings) Cost and Schedule Performance – The project is 88.9% complete versus the plan of 88.9% complete, as measured using earned value techniques. Remaining contingency is $7.8M and the contingency as a percentage of remaining work is 26.3%. The contingency continues to be carefully managed to ensure the successful completion of the MREFC project.

Page 2

The earned value schedule variance at the end of December 2008 is a favorable $173K, slightly reduced from the variance last month. The variance is mostly related to RPSC’s FY2008 schedule performance variance.

Page 3

The earned value cost variance at the end of December 2008 is a favorable $1,577K, a decrease from the $1,662K reported at the end of last month. RPSC has a positive cost variance of $728K due to lower than planned labor and On-Ice support charges in FY2008. The $622K favorable variance in Implementation was the result of lower training costs than planned, lagging invoices and a labor ramp down ahead of schedule.

Contingency Status and Plans ─ The budget planning for PY8 and beyond (April 1, 2009 to project completion) was completed and implemented in December, resulting in an $876k draw on contingency; this is less than the $1M previously accounted for in the risk assessment, and less than in any previous year. This risk assessment now includes the PY7 closeout and the potential cost of retaining key drilling and installation personnel for the last drill season in FY2011.

12/31/08 Change Log - IceCube Total Project Budget Baseline ($K)

No. Description Date Approved

Total Baseline

Allocated Budget

Allocated Budget Change

Contingency Budget

Estimate To

Complete (ETC)

Budgeted Cost of Work

Remaining

Contingency % of

Remaining Work

NA Status as of September 2008

275,293 266,638 0 8,656 33,315 33,315 26.0%

NA Status as of October 2008

275,293 266,638 0 8,656 31,461 31,461 27.5%

CR136 CR 0136 CAU Additional Verification effort

12/17/08 275,293 266,645 7 8,649

NA Status as of November 2008

275,293 266,645 0 8,649 29,947 29,947 28.9%

CR137 CR0137 PY08 Baseline Review

01/21/09 275,293 267,520 876 7,773 29,589 29,589 26.3%

NA Status as of December 2008

275,293 267,520 0 7,773 29,589 29,589 26.3%

Page 4

Risk Assessment & Potential Contingency Adjustments

Item Estimate ($K)

1. Contingency likely to be required to mitigate technical, cost and schedule risks associated with the approved scope of work. Risk assessments are made at WBS-Level 4 to determine the value of the risk exposure as a percent of the cost of work remaining.

$2,371K

2. Pre-Operations for additional capacity enhancements to the data storage and network systems both at the South Pole and in the Northern Hemisphere, and for extending software development efforts.

$500K

3. RPSC FY08 Closeout. The closeout is expected to return of RPSC authorized budget to IceCube contingency resulting from lower labor use and reduced support charges.

-$400K

4. Cost of the limited amount of remaining instrumentation for an 86-string detector, 80 strings from the original proposal plus the six deep core strings financed by European partners. The additional instrumentation is 5 surface-to-DOM cables and ~100 DOMs.

$1,350K

5. RPSC estimate of base cost to support the installation of six strings during the 2010/11 drilling season. The current RPSC baseline already includes support for 80 strings.

$1,340K

6. Project Year 7 Closeout for all WBS items excluding RPSC and the ANG. The closeout will return authorized budget to IceCube contingency due to lower drilling training and travel costs and staffing ramping down ahead of plan.

-$300K

7. The cost to retro IceCube equipment/materials from the South Pole at the end of the project in FY2012. This is a conservative estimate.

$825K

8. Potential cost to retain experienced key personnel in PY9 to ensure the success of the last drilling and installation season and the timely completion of the construction project.

$700K

9. Potential cost exposure should a full FY2011 drilling season be required resulting in the need to send UW personnel to the South Pole in FY2012 to dismantle the drill equipment/materials and prepare the equipment for retro out of South Pole.

$450K

Total $6,836K

Available Contingency as of Dec 31, 2008 $7,773K

Drill Operation and Installation – January was a very productive month for drilling and installation operations, nine holes were safely drilled with installation completed by January 21st. In addition, two dust logging runs and a deep NARC installation were completed.

A new software prediction tool was designed and completed. The new software provides immediate expected hole lifetime based on the drilling and reaming parameters. This will increase efficiency and greatly improve the planning and deployment of strings and special devices.

All EHWD systems were flushed and dried for winter storage. All equipment, hoses, cables, and fuel lines were disconnected and relocated to the prepared site for next season’s start-up by January 29th. The Main Supply Hose Reel was wrapped with heating blankets and insulation to maintain temperatures above -55C. A complete inventory has been taken. Spare parts lists were generated and preliminary plans for off-ice repairs and improvements completed.

Five firn holes were completed for the 2009 - 2010 season, including the RodWell. The Independent Firn Drill was flushed and winterized for storage.

Page 5

The annual lessons-learned and planning meeting with RPSC will be held on March 25th in Madison. The baseline plan for string and IceTop station installation is shown in the following table.

String Installation Plans Strings (Actual & Plan) 04/05 05/06 06/07 07/08 08/09 09/10 10/11 Annual 1 8 13 18 19 18 9 Cumulative 1 9 22 40 59 77 86

IceTop Station Installation Plans IceTop (Actual & Plan) 04/05 05/06 06/07 07/08 08/09 09/10 10/11 Annual 4 12 10 14 19 15 6 Cumulative 4 16 26 40 59 74 80 There is high confidence that IceCube construction will conclude on schedule in two years. Detector Commissioning and Verification – Verification continues to run since the start of IC40 data collection. Verification was tested offline to insure it could perform with the addition of new strings. This focused mainly on tests with simulated “soft local coincidence” data and the pedestrian issue of string numbering. Basically, a “Y2K” test was done to be sure there was nothing hardwired that would prevent the Verification code from functioning properly. One minor problem was found that affected the final plotting stage related to a naming convention. This problem was easily fixed. In response to a request from the DAQ group, Verification ran various high-level tests on data produced by two different versions of the pDAQ code to look for differences that might be indicative of a problem in the new version. Dawn Williams (U. Alabama) performed qualitative tests and Jason Koskinen (Penn State) followed up on quantitative tests. These tests will prove useful in the long run. Tuning of the parameters plotted on the Verification-Monitoring page continues. We have included the plots developed by Stephanie Hickford (Canterbury/Christchurch) to monitor the local coincidence performance. We currently are working on a way to detect problems related to variations of the position of the peak of the single photoelectron (SPE) distribution. In virtually all cases where a variation was seen, it was too small to affect the quality of the data, but large enough to represent a many-sigma deviation from expectation. Anna Franckowiak (Humboldt/Berlin) and Stephanie Hickford arrived at the South Pole in late January and began high level commissioning of the newly deployed strings. As of the end of January 13, of the 19 new strings were checked for timing and geometry calibration data was taken and analyzed for eight new strings. Six DOMs have slightly wider than normal timing distributions, and two DOMs exhibit a mean time that is not consistent with the 79 ns light travel time from the adjoining flasher. These relatively minor issues are being investigated. The geometry calibration has shown for the eight strings thus far calibrated offsets of no more than 5 m from the rough Pole measurement. Data Acquisition Hardware and Software – During the month of January 2009 the IceCube DAQ collected 3.72 billion events at a live fraction of 94.1% for all runs. Consult the day-by-day graph of the live fraction - it is evident that pole hardware commissioning operations impacted the detector availability during the latter weeks of January.

Page 6

DAQ personnel at Pole were Dave Glowacki, John Jacobsen, and Joanna Kiryluk whose task it was to integrate the new strings into DAQ and test the operation of the DAQ system with the new channels connected. Tasks completed were: 1) IceCube Live was made fully operational at South Pole, both command line and Web interfaces; 2) Betelnut release of pDAQ deployed and tested with full integration with IceCube Live; 3) IceTop minbias calibration mode demonstrated w/ no problems; 4) IC53 run with SLC with no problems found so far; IC59 imminent: 5) I3Live+DAQ used to collect flasher data for verification, much smoother than last year: 6) Winterovers trained in I3Live operations and special contingencies with pDAQ (e.g. how to adapt to critical machines failing); and, 7) IC59 runs performed. Snapshot of IceCube Live:

Page 7

IceCube Maintenance and Operations

Detector Up Time 94.1% IceCube Clean Run Up-Time not including AMANDA Array 84.5% Unscheduled Downtime 1.1% Events from DAQ 3.72 billion

Data collection during January 2009 achieved an overall 94.1% uptime. The in-ice clean run uptime was 84.5%, which is lower than normal, but high considering the large amount of activity and special runs/configurations that occurs during the season. Also affecting uptime is the commissioning of new strings.

• On January 13th AMANDA trigger rates started to show large fluctuations. This was due to radio antenna tests/calibrations. From then on radio tests are done excluding Amanda from the DAQ configuration.

• On January 14th, failed DOM power supply in hub72 and hub75 were fixed. • On January 16th a DOM (59-45) started to produce negative time stamps that affected the

JEB/PnF and C&V processing. The last subruns of the runs were affected, until the bad DOM was taken out of the configuration (on January 24th), were not processed successfully for these (more than 20) runs.

• On January 21st a new JEB release was deployed to handle new payload format from DAQ.

Various attempts to upgrade the DAQ software release were made. The new release Betelnut was finally successful with the only exception of an apparent increase in the number of DOMs dropped from the DAQ-runs (overall still a small number of about one DOM for every three runs). More information will be given in the February report as information is still being gathered.

• On January 23rd and days that followed the DAQ team gradually tested the entire data acquisition (including JEB) including all the available and commissioned new strings.

• During the last week on January multiple tests of the Live experiment control system were successfully performed.

Page 8

• On January 27th the DOM power supply on hub74 had to be replaced. • On January 28th a DOR card on hub66 had to be replaced. In addition a power outage

brought down 3 of the hubs (no hardware failure). • Also on January 28th a test of the IceTop Min Bias trigger was performed (this will replace

the special IceTop calibration runs starting with the IC59 run). • Also during the last weeks of January multiple session of C&V runs were executed with

flashers to obtain time-critical information of array geometry in preparation for the IC59 run to begin around April 1st.

GPS timing report summary: In the midnight between December 30 and 31, 2008 the IceCube GPS produced a wrong leap second. It was caused by the combination of the leap year (366 days in 2008) with the leap second that was intended to be accrued in the transition to the New Year, which was indeed accrued in addition to the wrong additional one. As a result, on December 31st the AMANDA and IceCube clocks went out of phase and the overall detector rate increased (since events that are supposed to be merged into the same event by the DAQ were not due to the time offset). On January 7th (with a small additional software change on the 8th) the AMANDA time was shifted backwards by one-second to match IceCube’s time. On January 22nd the GPS unit for IceCube was replaced with one that had been reset fully and had the correct time. At the same time the AMANDA fix to match the IceCube times was undone and from then on the overall times are correct. Data from January 7th through January 22nd has thus the correct relative timing between AMANDA and IceCube but has a shift of one-second with respect to UTC. That offset will be corrected in the offline processing. Data Handling – The figure below shows the daily satellite data transfer rates. The increase in “DebugData” from testing the newly installed strings this season is evident for the end of January.

The final IT infrastructure work in the ICL at South Pole was completed in January and personnel are reading the system for winter-over technical support. The data warehouse

Page 9

upgrades continued in January. The migration to the new data/exp with improved file system performance should be completed in mid-February. The HSM tape system library is still undergoing debugging to work out some technical problems. The vendor is working closely with IceCube IT staff at the University of Wisconsin. Filtering, Software & Database – The Joint-Event Builder and Processing and Filter server (JEB/PnF) merging of IceCube-40 and Amanda TWR data streams continues at South Pole sending filtered data sets of about 35 GB/day or more over the satellite. The major work in online filtering and software systems was:

• Development of IC59 online filter and triggers

• Continued work for release of new PnF filter system at start of IC59 run. This will be the final major release of the PnF filter system and will include improved performance capable of running with the full IceCube array, improved monitoring and logging with integration into the IceCube Live system, and more robust process distribution framework for the reconstruction workers.

• Icetray V3 development work moving forward nicely. Anticipated release of this final version for end of project year is on schedule.

Offline Level 2 filtering and high level reconstruction of the IC40 data continued to make progress last month. Accomplishments include the continued mass production of high-level reconstructions at the IceCube datacenter at the University of Wisconsin, and processing of satellite-filtered data for IC40. Both of which are going very well. There is still a wait for resolving the HSM tape library issues at the data center before we can go back and reprocess the first couple months of tape, which is required because the TWR calibrations were incorrect. Simulation – IceSim monte carlo simulation program as such is fairly mature now, and the development has been somewhat limited to improved performance and fine tuning. This task is moving more to the physics groups with program maintenance and production running the focus of the project. January saw good progress in monte carlo production across several of the distributed sites in the collaboration. There is still some delay in processing the simulation data through the L1 and L2 high-level reconstructions. The L1 is ready to go and is used to run the “online filter” on the simulations data. This reduces the size of the data set by an order of magnitude. The hold up is the L2 processing, which has to be slightly adapted for running on the simulation versus data. Education and Outreach – Mark Krasberg, one of our scientists, is posting his South Pole experiences on the Exploratorium's "Dispatches from Polar Scientists." His January posting has a picture of a 15,000 lb. IceCube surface-to-DOM cable in the cargo bay of a LC-140. Mark also has some wonderful pictures of altocumulus standing lenticular clouds. In addition, another IceCube scientist, John Jacobsen, publishes an almost daily blog with photos of his travel to the South Pole Station and IceCube deployment as well as a video of the last string deployment of the season. The video is posted on YouTube and can be accessed by using the following link: http://www.youtube.com/watch?v=U4qkxM4YPow

Page 10

Quality Assurance and Safety –The drilling season was completed without a single reportable injury. This is a direct reflection of the continuing emphasis on safety by all IceCube personnel. The monthly reports are posted at IceCube Monthly Reports. The On-Ice Weekly Construction reports are posted at On-Ice Weekly Construction Report.

A fuels technician signals an arriving LC-130 carrying Jan Luneman and Timo Griesel, the last IceCubers

to arrive this season. The ICL and old SPASE-2 building are in the background. (Photo: J. Jacobsen) Meetings and Events IceCube Visit to NSF February 20, 2009 IceCube Software & Computing Review at UW-Madison March 24 – 25, 2009 Season Review Meeting at UW-Madison March 25, 2009 IceCube Maintenance & Operations and Deep Core Proposals Due April 1, 2009 Spring Collaboration Meeting at UW-Madison April 28 – May 2, 2009 NSF M&O Review at UW-Madison TBD Science Advisory Committee Meeting at UW-Madison May 20 – May 21, 2009 NSF Annual Review – Proposed Date June 3 – 4, 2009