21 st century network management dale blakemore and bruce williams information services

21st Century Network Management

Dale Blakemore and Bruce Williams

Information Services

2Network Communications Services

Agenda• Overview of the Griffith University NOC operation• Griffith University Port Database (GUPD)• Plone and other tools (mrtg with rrdtool,

smokeping, layer 2 trace, attack search)• Future vision and roadmap• Questions


We had a problem…Like most people we had a number of disparate network

management systems that together gave us a view of services and systems. These systems all worked differently and only provided limited use in troubleshooting real network problems.

In addition to this other systems such as building management and control systems (eg swipe card, aircon, lighting) and video systems were moving onto the IP network and we were then told that these are critical systems and if it they have problems bad things will happen .


Administrator

The old way... previous monitoring procedures

Login

Login Login

Login


So we created the NOC

It was decided to create our own custom interface to these disparate systems that would provide us with near real time information for the whole network that we could see, understand and most importantly ACT ON!

The Griffith NOC incorporates a collection of commercial network applications and custom in house written tools that combine to provide a snapshot of high level information about the Universities network and services.


The new way…

Login

Central location for network monitoringNOC Website


Network and Service MonitoringThe Network Operations Center facilitates the Griffith

University Network Services team in providing a highly reliable and robust network infrastructure.

Some of the services monitored by the NOC Include:• Router/switch interface bandwidth and CPU utilization• Radius authentication services• Light weight access point client associations• DNS and DHCP services• Building management services (Air Conditioning etc.)


NOC Advantages

The Griffith NOC provides the following advantages

• Threshold breach alerting• Real-time fault notification• Reduced time in identifying faults and causes• The ability to become proactive• Remote network monitoring


SNMP and Syslog

NOC Server

SNMP Polling

Switch Firewall Router

SNMP Traps and Syslog


Outage and threshold alerting

NOC Server

IM Notification E-Mail NotificationSMS Notifications


Front end user interface

The front end has been written in a combination of PHP/Perl/AJAX and Javascript and is designed to be used from any standard compliant browser such as Firefox and projected onto a large display screen.

The following slides provide a description of each applet and the final product interface.


Last Alert

This applet will display the last event received form Intermapper or from a broadcast Jabber message sent to the NCS team.

This example shows the Griffith web server has returned from an outage with status OK.


NCS Staff Status

This applet displays the Instant messaging status of NCS staff, the status message below the name will be updated as the client status changes


Outages

This applet will list all network devices that are currently unable to be contacted by the network management systems.

In this example we have an access point in the n16 building down.


Campus Status

This applet displays the top level status of all Griffith campuses. NCS staff are able to drill down into this view for more detailed information.


Top Talkers

This applet gathers the top 10 interfaces on the network by frames per second.

The output is logarithmically scaled to prevent the bar graph exceeding the table boundaries


Graphs

The main display will rotate several graphs with information on bandwidth, possible network attacks and TCP session counters from key points on the network.


Griffith University Port Database (GUPD)

• Griffith University has developed a web based database to document switch port to wall outlet mapping.

• Currently the database holds over 18000 data port records for Griffith campuses with voice records to be added in the coming months.

• Documenting to this level has drastically reduced requests for port activations with support staff able to determine the VLAN and connection state of any outlet within the university.


Plone Content Management System• Griffith network team uses the Plone CMS to

augment the NOC by providing access to contracts, procedures, policies, templates, and documentation.


Handy tools• The NOC provides a launch point to some open

source and in house tools written to improve productivity and automate some of the more tedious tasks.


Handy tools continued…• Attack search queries a database of radius start records for

username to IP address binding• Subnet audit uses flood ping and SNMP requests to gather

information on port usage, DNS registrations and switch software versions

• Smoke ping written by Tobias Oetiker provided ping graphs of critical or troubled links

• Layer two trace performs an automated CDP trace from the NOC box to any IP address on the Griffith network

• Jabbercast sends a XMPP protocol message from a web page to all NCS team members

• AP Graphs grabs information on wireless radio associations and graphs the usage (even LWAPP’s!)


Future vision and roadmap

• Incorporate the cabling routes and pit information into a searchable map view perhaps using google earth maps which are GPS accurate (already have the above info in cad format which took two years to complete)

• Include a services view of information. Eg peoplesoft, blackboard and other key applications can see their infrastructure, performance details, SLA statistics, etc)

• Update GUPD to provide more realtime port information and the ability to change vlans, shut ports, etc.


Questions?

Our guiding principles for improving network management are to automate where ever possible, document thoroughly, to gain improved efficiency and to build repeatable processes.

21 st century network management dale blakemore and bruce williams information services

Documents