1  

IceCube Project Monthly Report - November 2009 Accomplishments

• Thanks to the hard work and efficiency of our team at the South Pole, drilling began three days ahead of schedule for the 2009-2010 season. The first string was deployed on December 6th, at 12:29pm.

• Verification testing of online data was upgraded to provide greater sensitivity to detecting irregularities with the data. This will improve response time to problems with the detector data stream.

• IceCube was part in the first ever University of Wisconsin outreach event aimed at the

Latino community, and was represented by four Spanish-speaking researchers.

• IceCube participated with Raytheon Polar Services Company in an extremely successful emergency response training exercise. The IceCube participants responding to the simulated emergency were able to secure the situation before rescue team arrived on the scene. The quality of the communication between the participants improves each year.

 

 

279.5279.2273.6260.8

235.3

199.4

140.1

85.7

39.5

15.0

272.6

231.6

0

50

100

150

200

250

300

FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11

FISCAL YEAR

DO

LLA

RS

IN M

ILLI

ON

S

DEPLOY INITIAL STRINGS & TANKS

INITIAL OPERATIONS & DATA ANALYSIS CAPABILITY

FULL SCALE DOM PRODUCTION

FULL OPERATIONS & DATA ANALYSIS CAPABILITY

DETERMINE FULL DEPLOYMENT RATE

COMPLETE EHWD INTEGRATION & TEST

Initial In-Ice Strings & IceTop Tanks Installed Jan-2005Initial Operational Capability Mar-2007Project Completion & Closeout Sep-2011Total Project Cost $279.5Value of Foreign Contributions $37.4NSF Funding $242.1Remaining Contingency $6.8Contingency as % of Remaining Work 41.3%Strings installed 59

31-Oct-09Performance and Cost through:

IceCube Project Baseline (M$)

___Total Funds (US & NonUS)___Original Budget (BCWS)___Current Budget (BCWS) 93.9 %___Earned Value (BCWP) 93.9 %___Actual Cost (ACWP) 93.2 %

 

  2  

Project Year 8 Major Milestones – Baseline Current

Ship all Instrumentation for 09/10 Season to the Pole. 30 Sep 2009 30 Sep 2009 South Pole FY09/10 Season Readiness Review Sep 2009 16 Sep 2009 Start FY09/10 season Drilling Operations 7 Dec 2009 4 Dec 2009 End Drilling & Installation Operations Baseline = 18+ Installed Strings (77+ Cum.)

31 Jan 2010 as planned

End IceTop Installation Operations Baseline = 14 Stations (73 Cum.)

10 Feb 2010 as planned

Commissioning of Strings / IceTop Stations complete. 12 Feb 2010 as planned Strings / IceTop Stations successfully integrated into DAQ 31 Mar 2010 Complete Geometry Calibration for strings deployed in 09/10 31 Mar 2010 Deliver IC77+ Computing Infrastructure & Filtering Software 31 Mar 2010 Commence IC77+ Science Runs 30 Apr 2010

Cost and Schedule Performance – The project is 93.9% complete. Remaining contingency is $6.8 million. There has been no change to the NSF MREFC funding requirements of $242.1 million since the project performance baseline was established in February 2004.

  The Schedule Variance at the end of October 2009 was $13k. The cost variance at the end of October 2009 was $1,889k (compared to $2,131k last month). This favorable variance is

  3  

principally due to lower labor and on-ice support costs for Raytheon Polar Services Corporation and the Air National Guard in FY2009. In addition, training costs for drilling were less than budgeted (Implementation) and there are still several lagging invoices for cables (Instrumentation). The baseline schedule for string installation is 18 strings in December 2009 – January 2010 and 3 strings in the final season, December 2010 – January 2011. A total of 19 strings were installed last season. UW and the IceCube collaboration requested NSF approval to complete the 80-string array originally proposed. The 80-string array can be completed within the planned contingency and no additional funding was requested. The six DeepCore strings are already approved and included in the installation plans. The resources are in place for the successful completion of the MREFC project in FY2011. UW-Madison and the IceCube collaboration submitted revised budgets for M&O for FY2011-2015. The revisions were made in response to guidance from the NSF and the IceCube International Oversight & Finance Group (IOFG). Included in the submittal were responses to the recommendations from the NSF panel that reviewed the original proposal.    Contingency Status and Plans - One change request was implemented in the past month: IC0134 Non-US funding of six Deep Core strings, based on approved amendment #17 to the Startup & Construction cooperative agreement. Total Project Cost has been changed from $275,503k to $279,472 with no change to Contingency.  

Change Log - IceCube Total Project Budget Baseline ($K)

No. Description Approval Date

Total Baseline

Allocated Budget

Allocated Budget Change

Contingency Budget

Budget for Work

Remaining

NA Status as of July 2009 275,503 267,908 0 7,596 19,036

CR154 ICL Rooftop access 09/29/09 275,503 267,908 0 7,596

CR155 Data storage upgrade 09/30/09 275,503 268,371 464 7,132

CR156 SUBR distributed computing 09/28/09 275,503 268,409 37 7,095

NA Status as of August 2009 275,503 268,409 0 7,095 18,541

CR157 Add Computer Science effort to NPX Designs

10/29/09 275,503 268,516 107 6,988

CR159 Remaining Surface-to-DOM cables assembly at Sea Con

11/03/09 275,503 268,657 141 6,847

NA Status as of September 2009 275,503 268,657 0 6,847 17,837

CR134 Non-US Institutions contribution - Deep Core

11/25/09 279,472 272,626 3,969 6,847

NA Status as of October 2009 279,472 272,626 0 6,847 16,577

 

  4  

 Risk Assessment & Potential Contingency Adjustments ($K)

Item Estimate

Contingency potentially required for technical, cost and schedule risks associated with the approved scope of work. Risk assessments are made at WBS-Level 4 to determine the value of the risk exposure as a percent of the cost of work remaining.

$1,458

Pre-Operations costs for additional capacity to the data storage and network systems both at the South Pole and in the North, and for extending software development efforts.

$2,100

RPSC Estimated FY09 Closeout (-$700K) and moving the support for one string from FY09 to FY11 (+$250K)

-$450

RPSC estimate of base cost to support the installation of six strings during the 2010/11 drilling season. The current RPSC baseline already includes support for 80 strings.

$1,340

The cost to retro IceCube equipment/materials from the South Pole at the end of the project in FY2012. This is a conservative estimate based on Rev. 8 of the RPSC budget.

$1,050

Potential cost to retain experienced key personnel in PY9 to ensure the success of the last drilling and installation season.

$730

Potential additional Fuel Cost exposure during the last season of drilling (FY2011) $450

Total $6,678

Available Contingency as of Oct 31, 2009 $6,847

 Drill Operation and Installation – All drill and deployment staff arrived at the South Pole Station per plan. The staff did an exceptional job completing inventories, testing equipment, and de-winterizing the site. Thanks to their hard work and efficiency, we were able to being drilling three days ahead of schedule, completing deployment of the first string on December 6th at 12:29pm.

Team  members  use  snow  mobiles  to  extend  the  drill  hoses  and  prepare  for  installation  of  insulation.  

  5  

South Pole Drill Activities: • Heavy equipment was de-winterized and provided from Raytheon Polar Services

Corporation. • Drill Camp excavation was completed as scheduled. • Main fuel tanks were delivered and generators and modules were connected. • Two of the three main generators were fired up. The third generator is currently down with

equipment problems. • All power feeder cables were laid out and modules powered up. • Main heater systems were fired, tested, and repaired. • High Pressure Pumps were de-winterized and tested. • Main water tanks and transfer pumps were prepared and seed water delivered. • The RodWell was set up and developed. • Drill towers and modules were located at holes one and two. • Power, communication, and network cables throughout and between the Seasonal

Equipment Site and Tower Operation Sites were laid out and connected. • Safety systems were tested and readied for drilling, including the Emergency Stop system,

Reel Stops, Fire/CO monitors and suppression systems, and alarms. • All meters and sensors were re-installed and tested for pressure, temperature, and flow. • Hoses were laid out and connected. The entire system went “wet” on Nov 24th. • All main winches and reels were powered, inspected, repaired, and tested. • The replacement prototype heater was installed and testing has begun. • Seasonal Equipment Site fuel tank upgrades and reconfigurations were completed. • The Independent Firn Drill was heated and powered. Four firn holes were drilled by end of

the month. • Documentation filming has begun following pre-developed scripting for the Enhanced Hot

Water Drill user manual. South Pole Installation Activities:

• Equipment inventory was completed. • Pre-staging of equipment, cables, and Digital Opitcal Modules (DOMs) was completed. • Special device equipment began arriving on schedule. • The first two cables are loaded onto reels and ready for deployment. • DOMs are tested and staged as well.

Off-Ice Drilling Activities:

• Cargo: Some minor weather/flight delays to arriving cargo at Auckland International Airport, no considerable impact.

• The Northern Hemisphere Support Group received and completed 11 requests for urgent procurements/shipments to support South Pole.

Detector Commissioning and Verification - Verification testing of online data was upgraded to give greater sensitivity to deviations that occur on a single-run timescale. Previously, the sensitivity was to deviations that occurred in at least three of the previous five runs. Now that we have extensive experience and confidence in the stability of the parameters being studied (specifically: “track detection probability” and DOM occupancy), we are able to dial down the threshold. This will improve the response time to problems.

  6  

Verification continues to run smoothly. Modified versions of the verification scripts for the new operating mode for PnF were prepared and were tested on the South Pole Test System, giving identical results for key verification parameters. The scripts will be tested at the Pole in December. No additional progress to report on tests of waveforms into the standard processing. Calibration and Monitoring - There has been, and continues to be, effort toward matching real data and Monte Carlo data by exploring changes in simulated ice properties and studying parameters such as DOM occupancy. This is a very complex task and we plan to be working on it into 2010. The spatial shape of a DOM-enclosed flasher source is being measured in a UW Madison lab. These measurements will be used in a refined source description in a Monte Carlo that is used to fit the data. Additionally, researchers are studying a systematic difference in the geometry calibration between results obtained with shallow versus deep flashers. Data Acquisition Hardware & Software – Data Acquisition (DAQ) ran during November with a 97.9% uptime. Notable outages included several instances of DOMhub system software crashes, ultimately resulting in the loss of slightly more than 6 hours of availability. We are investigating the nature of the kernel crashes and also developing ideas for making the DAQ more robust against the loss of an entire string of the array. El Farolito was deployed permanently at South Pole on Nov 30th. The major features included in this new release are (1) a complete re-coding of the data transport framework (the Payload System) to increase efficiency, both saving processing power and space; and (2) enhancements to the flasher control state machine permitting the DAQ to run flasher sequences more rapidly and robustly. DAQ hardware (DOMHubs and an upgraded server node to replace the existing event builder) arrived at Pole at the end of November and is being installed in the IceCube Lab.

 Detector Up-Time: 97.8%

  7  

IceCube (in-ice) clean runs Up-Time: 90.2% Unscheduled Downtime: 2.0%  Definition of the terms: “Detector Up-Time” is the percentage of the time period for which the pDAQ data acquisition was acquiring data and delivering at least 500Hz of event rate. This uptime measure therefore includes periods in which the detector was taking data with a partial detector enabled or with light contamination from calibration sources. “Clean run Up-Time(s)” is the percentage of the time period considered to have pristine data (standard hardware and software configurations) with the full nominal detector enabled, not contaminated with light from calibration sources and for which no serious alerts were generated by the monitoring, verification or other. The criteria applied are not algorithmic but rather represent the Run Coordinator’s overall impression of the quality (including uniformity) of the runs/data. In November we saw the startup of activities at Pole, such as work on system computers, training of new winterovers, etc. Due to these activities, random failure occurrences on surface systems, and unknown cause-and-effects from increased activity, both the overall detector uptime of 97.8% and the clean in-ice uptime of 90.2% were lower than usual. A clock distribution card on hub59 failed in early November. As per normal procedure, a run without that hub was started. However, after a failed attempt to run the standard candle light source, the operator unintentionally restarted run with the configuration excluding string 59. These events caused the large unintended partial-IceCube data period of November third and fourth. The user interface of Live is being upgraded to avoid this in the future. On Nov 8th, while maintenance work was being performed on a disk array, another disk array was unintentionally affected. This ultimately caused the loss of about 6 hours of data that could not be written out because the disk was in read-only mode. Changes have been put in place in the disk checking monitoring system to send alerts if a similar event occurs. Data Handling - South Pole systems in the IceCube Lab began the 59 string physics operation on May 20th and it has continued to run smoothly through November. The figure below shows the daily satellite data transfer rates in GigaBytes/day for this month. The green is for filtered physics data, which dominates the total bandwidth  

  8  

   

 Information technology work was mostly in two areas 1) South Pole systems upgrades for increased data in 2009-2010 installation season, and 2) upgrades to data warehouse and offline processing infrastructure. South Pole systems upgrades began on schedule in November and are ongoing into the pole season. The upgrades in the north focus on continued commissioning of the new computing cluster NPX3 and upgrades to the data warehouse. The upgrades should be final in early December. Online & Offline Filtering, Software & Database - The online filtering system for the 59 string run, which began in May, continues smoothly with daily satellite transmission of filtered data to the northern data warehouse at UW as logged in the figure above. Offline:

• IceTray "v3" in full release, with continuing minor changes in new releases. • Work started on new packaging system for release and distribution of IceTray software to

users. PnF/Online filtering:

• Preparing for start of the 77 string configuration filtering, with filter proposals due in December.

Production processing of data:

• Final 40 string L1/L2 offline processing completed (major milestone). • Processing of 59 string run that ended in May 2009 is being prepared. • Processing code finalized, in final verification. • Production processing delayed by a couple weeks and will start in early December.

  9  

Simulation - Steady progress is being made on improved detector simulation. Studies continue on the systematic error for physics analysis due to modeling ice properties. New simulation data sets for the 40 string configuration were produced using various ice properties. The graph below shows the total worldwide simulation production jobs for the month. Although there was a lull in production during some information technology work on the central data center, the total average number of jobs was nearly double that of the previous month.

Education and Outreach – With the South Pole construction season underway, IceCube bloggers have been providing the public with a glimpse of what life is like at the Pole. Additionally, IceCube was a proud organizer of and participant in the UW-Madison campus event “Explorando las Ciencias,” an outreach event aimed at the Spanish speaking community. One of the bloggers for the new IceCube blog is graduate student Laura Gladstone, who is traveling to the Pole for the first time as a DOM tester. Read her account of her travels at http://blog.icecube.wisc.edu/. San Francisco high school teacher Casey O'Hara has been posting to the PolarTREC site at http://www.polartrec.com/icecube-in-ice-antarctic-telescope. As part of his project, he is a member of the IceTop team and has been communicating with his students in California. For another viewpoint, Logistics Manager Terry Hannaford sends his experiences via the University of Delaware website. Read his account at http://www.expeditions.udel.edu/antarctica/blog-dec-3-2009.html. On November 15, Spanish speaking IceCube scientists and graduate students engaged the public at the first Explorando las Ciencias at the University of Wisconsin Space Place. Advertising for the event was targeted to the Hispanic community in Madison. In addition to the IceCube display with a DOM, our scientists encouraged families to try on Antarctic gear and to explore animations of the drilling and IceCube array.

  10  

IceCube PhD candidate Marcus Santander, of Argentina (center), and Education and Outreach Specialist Evelyn Malkus (far right) answer questions about IceCube at Explorando las Ciencias.

Quality Assurance and Safety – With the beginning of the drilling season and the required preparation, the safety team has been busy checking systems, responding to issues, and ensuring safety standards are being met. Routine Drill Camp preparations have been completed:

• Testing of Smoke/Fire Alarms in Mobile Drill Structures in conjunction with the DCC screen alarms.

• The entire Emergency Stop system has been tested. • The seven fire suppression system tanks have been weighed to gage remaining charge. For

the time being, they all pass. • All thirty-two hand-held fire extinguishers have been weighed. Nine required replacement

due to low charge. Raytheon will have these extinguishers re-charged and will give IceCube other fully charged extinguishers in replacement.

• Smoke/CO detectors, Emergency Stops, and Heat Rise Sensors have been arranged to trigger the Drill Control Center operator. As soon as one of these systems engages, the operator will know immediately.

In response of a minor incident last week, we are exploring changes to a safety protocol. While following proper safety procedures, a driller who was working a grinding operation felt as though he got something in his eye (the driller was wearing safety glasses at the time). Subsequent doctor’s examinations revealed no evidence of pathology, but it has prompted us to review and make changes to safety protocol. We are investigating the addition of a safety glass attachment for the welding helmets.

  11  

This month, the combined Raytheon Polar Services Corporation/IceCube emergency drill was a huge success. For the test, we staged a fire in MHP3 at the Drill Camp. The drill went smoothly, and there was excellent communication between the Raytheon first responders and the IceCube Drill Control Center.