know it’s broke before your customers do. david thomas [email protected]
TRANSCRIPT
2011 2
Acknowledgements
Nagios World Conference
Ethan Galstad
And of Course all of you!
A little about me, and what I do.
David ThomasDirector of Information Technology and Emergency CommunicationsSandoval County, State of New Mexico
Over 20 years in the IT field, certified ITIL, Six Sigma, HDI and a bunch of other stuff.
2011 4
Sandoval County
Sandoval County a pretty unique place.
The county has a total area of 3,714 square miles.5 square miles of water.Population of 131, 56112 Indian reservations (pueblos), The second most of any county in the United States.
Bandelier National MonumentCibola National ForestTent Rocks National MonumentSanta Fe National ForestValles Caldera National Preserve
5 National Protected Areas
The county is also flanked by two national laboratories
Sandia National laboratory to the southLos Alamos National laboratory to the north
Numerous nuclear waste and weapons transportation routes run through the Sandoval County as well.
Sandoval County
A Unique place with Unique Issues:Day-to-day operations
Over 70 Buildings throughout the 3,714 square miles of Sandoval County
County Sherriff facilities including 60 sheriff vehicles (PC and radio functions)
County Fire including 24 Fire Stations 80 vehicles (PC and radio functions)
County Detention Center 500+ inmate facility
County dispatch center
And all the other functions of county government; Treasure, Assessor, public works, ext.
County Heath Clinic including dental, dietary, indigent populations operation
Tribal governments of the 12 pueblos
2011
Sandoval County Customers
So this presentation is called
So Who we talking about?
EVERYONE!
Population of 131, 561 our constituents
All County Government
All Municipality Governments
All 12 Indian Pueblos and their governments
2011
Know it’s Broke Before Your Customers Do.
Sandoval County Network
2011
Pretty typical midsize network
Here is where it get’s interesting!
2011
Talk about terrain we start on a hill shoot across the river valley back over the river valley up to a mountain peak to get to Cuba, NM 75 miles from Bernalillo, NM where we started.
Not much of a chose if we are to get high speed networks to the far reaches of the county.
So Much for the background.
Now for something completely different.
It’s still about customers!
And in this case when it broke it broke bad!
2011
Real emergencies in need of real solutions
In 2011 Sandoval County had to activate it’s Emergency Operations Center Twice.
February 2011 – Regional wide natural gas outage during a record breaking -40 degree temperature outbreak
June & July 2011 – Los Conches Fire - Largest wildfire fire in New Mexico History
2011
Nagios played a pivotal role in the successful emergency operations of
Sandoval County during these events!
Regional wide natural gas outage
What happened:
2011
During a record breaking cold spell demand was such that gas companies made the decision to cut gas to less populated areas so urban areas would not be affected. The result was a sudden lost of gas for many areas across the desert southwest.
What did Sandoval County Information Technology and Emergency Communications do?
Activation of County Emergency Operations Center and information centerCode Red emergency notification to citizens activated24 hour information line set up manned by ITHam radio coordination with county shelters and red crossUpdated information on County web sites 24/7FEMA and NIMS forms compliance Operations lasted 120 hours
Los Conches Fire
2011
What happened:Down power line turns into largest wildfire in New Mexico history.
What did Sandoval County Information Technology and Emergency Communications do?
Activation of County Emergency Operations Center and information centerCode Red emergency notification to citizens activatedOperations lasted 336 hoursMobile emergency radio communications and command unit activated IT mannedCOM/L activation on site communication coordinationUpdated information on County web sites 24/7FEMA and NIMS forms compliance Massive evacuations including the city of Los Alamos and LANLHam radio coordination with county shelters and red crossMonsoon rains now major flooding.
Smoke plumes from my backyard.
2011
This image showing the Las Conchas wildfire raging in the Jemez Mountains of the Santa Fe National Forest in north-central New Mexico the was taken by a crew member aboard the ISS on June 27
Residences: 63 destroyed Outbuildings: 49 destroyed; 2 damaged Injuries to Date: 15Size: 156,593 acres
The Role Nagios Played and it’s a big role
In both situations when the County EOC had to be activated Nagios turned into a very useful tool.
2011
Nagios was constantly being monitored by EOC staff
We have remote devices in remote locations that where in the fire
All fire vehicles and sheriffs Vehicles could be monitored by VPN’ed laptops
Thanks to IP based radios and repeaters those were monitored too
Generator power was monitored on remote microwave shelters
Mobile command truck could monitor and we could monitor it from the EOC
Don’t forget standard uses for Nagios even more important in an emergency
All our front line people have access to view including dispatch center
All of this has to be documented for FEMA, and best of all it can be used as your change management database for ITIL functions. Nagois has all that built in.
We use Cacti too!
Nagios played a vital roll:
2011
Know it’s Broke Before Your Customers Do.Or
If it’s frozen or burning downUse advanced monitoring tools to help
Save lives and property.
David Thomas
Nagios played a vital roll and will be a tool we use time and again in emergency situations and the day-to-day
operations of Sandoval County, NM