university of new mexico “the agony & the ecstasy… the trials, tribulations & thrills of...
Post on 28-Dec-2015
222 Views
Preview:
TRANSCRIPT
University of New Mexico
“The Agony & the Ecstasy…the Trials, Tribulations & Thrills
of an Enormous Project”
University of New Mexico
“The Agony & the Ecstasy…the Trials, Tribulations & Thrills
of an Enormous Project”
Mark ReynoldsMark Reynolds Associate DirectorAssociate Director
ITS Communications Network ITS Communications Network ServicesServices
The University of New Mexico: A City Within a CityThe University of New Mexico: A City Within a City
Founded in 1889 with 600 acres along old Route 66
27,000 students on primary campus; 7,000 at branch campuses
20,000 faculty and staff statewide including UNMH
Branch campuses in:– Gallup– Los Alamos– Taos– Valencia County
Health Science Center (HSC) University of New Mexico Hospital (UNMH)Health Science Center (HSC)
University of New Mexico Hospital (UNMH)
HSC, established in 1994, is largest academic health complex in the state
UNM Hospital (HSC's primary clinical component):– Consistently one of the 100 top-performing hospitals in the US– Primary teaching hospital for the
UNM School of Medicine– Home of the highly regarded UNM
Children's Hospital– New Mexico's only Level 1 Trauma
Center that treats more than 1,300 critically injured patients a year
UNM Children’s HospitalCommunications Challenges ($260 Million Project)
UNM Children’s HospitalCommunications Challenges ($260 Million Project)
480,000 square feet - six stories high 2000+ voice & data drops using Category Augmented 6 cable plant Telecommunications building in planned footprint
24,000 ports
19 PBX’s, multiple Key systems, DRUs
9,000 voicemail subscribers, 90 auto attendants
64 Automatic Call Distribution groups, 300 agents
2,200 cell phones, 4,000 pagers
What UNM Has…What UNM Has…
PBX SystemsSimplified Version
PBX SystemsSimplified Version
ESC
ON BATTLOAD ONBYPASSFAULT
Remote Systems - DRUs, NewBridges, Key Systems7-9-05
Pair Gain
Pair Gain
1 2 3
4 5 6
7 8 9
* 8 #
Load BatteryLineOn On
BatterySmartBoost
ReplaceBattery
Test
CSUs
CSU
Pair Gain
CSU
CSU
UNM PressIVS 2000
UNM Press (3721 Spirit Dr)
Norstar 8 x 24Key System
8 Phones, Ext. Only
Isleta Mental Health ClinicNorstar 8 x 24Key System
20 Phones, Ext. Only
Pediatrics (317 Commercial NE)
Norstar 8 x 24Key System
15 Phones, Ext. OnlyRed Ph. 837-6840
M&I(6900 Gonzales SW)
Electra Pro 1Key System
9 Phones Ext. OnlyRed Ph. 344-1080
M&I(1231 Candelaria NW)
Norstar 8 x 24Key System
11 Phones Ext. Only
Marron Hall (9)
Electra EliteKey System
43 Phones, Ext. Only
Marron Hall (19)
Norstar 8 x 24Key System
16 Phones, Ext. Only
Bandelier West (16)
Norstar 8 x 24Key System
34 Phones, Ext. Only
ISR(2808 Central SE)
Norstar 8 x 24Key System
68 Phones, Ext. OnlyNEWBRIDGE (into Key System)
T1 Ckt: 59DHDA103155
UNM Ticket Ofc (1111 Univ. SE)
DRU1 FB: 925-563915 Ports Avail.
ACD in Ticket Office
Rio Rancho(2600 American NW)
DRU (2)T1 Ckts: 59DHDA002383/84
OR 54HCXX002283/841FB: 897-3562
1, 14 Ports Avail.
Mitchell Hall (23)DRU
15 Ports Avail4 Pairs
Mesa Vista (56)DRU
15 Ports Avail16 Pairs
Poison Control (234)DRU (2)
1FB: 272-10454 Ports Avail
7 PairsContinuing Ed.
(1634 University NE)
DRU (5)1FB: 277-6851
2,4,23,23,23 Ports Avail.26 Pairs Each
Hope House(1816 Sigma Chi NE)
DRUT1 Ckt: 59DHDA 106124
8 Ports Avail
Humanities (81)DRU
11 Ports Avail3 Pairs
Scholes Hall (10)DRU (3)
9, 7, 15 Ports Avail8 Pairs Ea.
Farris Eng. (119)DRU (2)
9 Ports Avail8 Pairs
Student Services (85)DRU (2)
10, 17 Ports Avail.42 Pairs Ea.
256
Load BatteryLineOn On
BatterySmartBoost
ReplaceBattery
Test
Academy Clinic(7801 Academy)
DRU T1: 54HCXX001923
1FB: 821-399816 Ports Avail
Casa EsperanzaKey SystemNorstar 8x24
10 Phones, Ext. Only
Eye Clinic(1600 Univ)
DRU (2)T1s: 59DHXX000371/72
1FB: 242-2947
ASIC/OSIS(1213 Univ)
DRU1FB: 247-025113 Ports Avail
Nursing & PharmacyKey SystemElectra Pro I
8 Phones, Ext. Only
SEH Clinic(302 A San Pablo)
DRUT1: 59DHDA105853
1FB: 254-7791
Eye Clinic (269)DRU
1FB: 272-095916 Ports Avail.
EMS Academy(2700 Yale SE)
DRU T1: 59DHDA967175
1FB: 242-69433 Ports Avail
DMAT/CREST(2704 Yale)
DRUT1: 59DHXX000340
1FB: 242-694323 Ports Avail
Orthopedics(1101-5 Medical Arts NE)
DRU (2) T1 Ckts: 59DHDA104300/301
1FB: 242-26613, 15 Ports Avail.
Office of Evaluation(2730 San Pedro NE)
DRU T1: 54HCXX001986
1FB: 884-3986
MHC(2600 Menaul NE)
DRU (2 - Pair Gain & Direct)
1FB: 272-280010, 18 Ports Avail.
33 Pairs Ea.
CRTC
CTH(1127 Univ. NE)
DRU 1FB: 272-476811 Ports Avail
8 Pair
Truman Clinic(625 Truman NE)
DRU (2)T1 Ckts: 54HCGS967518/558 GST Ids: NM.HCE.60032/33
0, 13 Ports Avail.
Corrales/Westside(4808 McMahon)
DRU T1: 54HCXX001922
1FB: 897-754913 Ports Avail.
Dermatology ClinicDRU
T1: 59DHXX0003121FB: 247-144018 Ports Avail.
Walter Clinic(1500 Walter)
DRUT1: 54HCGS001849
1FB: 242-081512 Ports Avail
Airport/Life Guard(2505 Clark Carr Loop SE)
DRUT1 Ckt: 59DHXX000320
1FB: 244-335320 Ports Avail.
1720 Louisiana NEDRU(2)
HSC Nat.Amer T1Ckt: 59DHDA556067
1FB: 232-636012 Ports Avail.HSC Geriatrics
T1 Ckt: 59DHXX0003351FB: 232-638023 Ports Avail
UPA (PALS)(1650 Univ.)
DRUT1: 54HCGS967629
1FB: 243-991017 Ports Avail
Pair Gain
Pair Gain
Speech & Hearing (2001 Lomas NE)
DRUT1 Ckt: 59DHXX000375
1FB: 243-0692
Pair Gain
CSU
Latin Amer. Inst.Key System
1A2KSU6 Phones, Ext. Only
Latin Amer. DatabaseKey System
1A2KSU3 Phones, Ext. Only
Casaa Runaway(325 Washington SE)
NEWBRIDGET1: HCGS001852
1FB: 247-7451
Architecture(720 Vassar)
NEWBRIDGET1: 59DHDC100973
1FB: 4-119410 Avail. Ports
Mitchell HallPair Gain
NEWBRIDGE1FB: 1347/134819 Ports Avail
Elks Lodge(1642 Univ.)
NEWBRIDGET1: 54HCGS971716 &
50HMUC08606412 Ports Avail.
Dental Residency(4216 Louisiana)
DRUT1: 59DHXX00366
1FB: 883-435810 Ports Avail
NB
ACTDRU
404 San Mateo1FB: 265-5798
256 DRU (22 cards and DRU’s) CRTC DRU (25 cards and DRU’s)
1101-4 Medical Arts
5901 Zuni
First StepWhat to do with the University’s Main Central Office PBX?
First StepWhat to do with the University’s Main Central Office PBX?
Option 1: Move 11,000 port PBX to another location (estimate $10 million)
This would require:– New building and power plant
– New PBX
– Move all copper, fiber and carrier connections
Decision: Expense and time unacceptable to Hospital
Second option: Leave the PBX where it is & protect the PBX building– Hospital estimated $400,000 to fortify the PBX building
– Actual final cost was $2 million plus new office building
Decision: Work with design team to protect the University’s communication systems
Next Steps - Survival PlansNext Steps - Survival Plans
Identify risks and probability of system failures
Develop a mitigation & contingency plan
Develop a disaster recovery / incident action plan
Make improvements to construction plans
Develop detailed work plan
Perform audits of switch-room, electrical and cable plant
Rehabilitate cable vault
Replace voice processing system
Next Steps – Survival Plans Next Steps – Survival Plans
Reroute carrier facilities, negotiate with LEC for two additional POPs & make Hospital PBX independent
Reroute cell phone services
Reroute alarm / emergency services
Develop emergency services agreement with outside firm
Create position to oversee protection of switch room and manage cable plant
Next Steps - Telephone Planning StudyNext Steps - Telephone Planning Study
Five year plan with input from stakeholders
Ensure high quality customer service and “state of industry” systems
Enable cost controls
Ensure self-financial support
Plan for technical and operational convergence
Move thoughtfully toward convergence, evolve with current manufacturer (NEC)
Use multi-vendor solutions where practical
Plan / carry out VOIP trials
Perform cost allocation study to determine true costs
Develop rates and revenue strategies
Existing PBX roomExisting PBX room
Construction BeginsConstruction Begins
Construct new office building for telecom staff
Install and cutover new voicemail system
Demolish building around the PBX room without structural damage
Keep PBX room sealed, provide positive pressure to keep dust out and build vestibules at both entrances
Carrier changes (two additional POP’s)
Power design and changes
HVAC system - “clean” the duct work (original plan)
Remove copper cable feeding existing telecom building
Move the carrier fiber paths that feed the PBX room
Move two cell sites
Overcome public utility power pole complications
Remove ceiling tiles to avoid equipment damage when removing copper plant
Design new lighting fixture layout
Decommission Halon system and install FM200 system
– Halon dump accident– New piping system
Construction ContinuesConstruction Continues
Construction ContinuesConstruction Continues
Remove existing HVAC system which consisted of:– Indoor Liebert Unit– Ceiling Unit for backup
Install two 20 ton units using “chilled water” and DX backup – Hole in roof for exhaust of portable systems– Portable units in room to keep temperature
at 68 degrees with 25% humidity– Install new duct system
Install new generator to replace existing one – New grounding system – Power within the PBX room rerouted and
re-cabled
Construction Time Lapse Video Construction Time Lapse Video
Biggest Construction ConcernsBiggest Construction Concerns
Vibration and Core DrillingVibration and Core Drilling
Beam InstallationBeam Installation
The Best Laid Plans….The Best Laid Plans….
Dust– No vestibules built, see daylight through
walls and roof– PBX equipment and room fill with dust– Heat and aging PBX concerns—could fail– Engaged external and internal
professional services to document Contractor
– Installed devices to assist in damage control, but the damage was already done
SolutionSolution
Bonding authority received for switch replacement and upgrades--$5 million
RFP developed
– Replace the 20+ year old, 11,000 port main PBX
– Upgrade the Hospital 10,000 port PBX
– Various ancillary system and PBX upgrades
– Outsource the maintenance and installation functions
Telephone Planning Study recommendations followed
Next StepsNext Steps
RFP completed– Negotiation & award of contract– Upgrade all systems, replace
sets not ADA compliant– Transition maintenance group from
existing contractor to new contractor
Installation of fiber paths to the University, Hospital, and data center with full fail over
Next StepsNext Steps
Establish PBX upgrade/replacement project and communications plan
Maintain support for work orders and trouble tickets during PBX implementations
Build new Hybrid PBX and distribution frame within same room as existing PBX (space limitations)
Tie hybrid PBX’s to existing PBX for migration strategy
Design and install temporary emergency PBX for the hospital PBX upgrade
Next StepsNext Steps
Customer service responsible for “getting the word out”
Communication methods included:
– Flyers posted on entry doors and throughout affected buildings
– Telephone calls to key department representatives
– Email to building coordinators and mass emails
– UNM News Minute update
– Schedules, FAQ’s and other documentation posted on the web
– Posting on the outage notification whiteboard
Communication process changed as the criticality of the situation increased.
The Trouble Begins!The Trouble Begins!
Roof– Roof leaked during snow and rain fall– Installed sump pumps with alarm
system – New roof and drains
HVAC system– 20 Ton units come on line, backup
system does not work
The Trouble Intensifies!The Trouble Intensifies!
Our WORST FEAR!– PBX starts to fail
• No dial tone, call forwarding, auth code failures
– Cannot shut down the PBX– Weekend – PBX fails but comes back– New PBX is working but not all customers have
been moved over– Fast track the project
• Cut over 4000 pair a day
• 2-3,000 end users affected daily
• Work day and night to cut-over
• Move E911, voicemail & special circuits
• Affected Hospital auto attendant & clinics
New University Switch RoomNew University Switch Room
UNMHeart – Courage - Wisdom
UNMHeart – Courage - Wisdom
Thank You! Thank You!
top related