achieving operational excellence - cisco · presentation_id © 2006 cisco systems, inc. all rights...

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 1

Achieving Operational Excellence

Asia Pacific Technical Services


What is Operational Excellence?

PeoplePeople

TechnologyTechnology

ProcessProcess


Is this Rocket Science?


What are the risks?

• Purchase price is ONE element of the total cost

of a system

• Hidden or less obvious cost/value drivers often

outweigh the purchase price

• A network foundation with integrated advanced services

leads to lower TCO

Training CostsTraining Costs

Opportunity CostsOpportunity Costs

Cos

t Driv

ers

Cos

t Driv

ers

Network Purchase PriceNetwork Purchase Price

Network Costs = 20% Purchase, 80% OperationsSource: The Meta Group, 2005

Cost of ComplexIntegration

Cost of ComplexIntegration


Architectures

� ITIL or ETOM

� Information Technology Infrastructure Libraryhttp://www.itil-officialsite.com/home/home.asp

� Enhanced Telecom Operations Maphttp://www.tmforum.org/


A4


Act


Lets manage the network


First Problem


What information do we have?

1. Voice quality on a phone has problems

2. Quality make voice stream unrecognisable?

1. Voice quality on a phone has problems

2. Quality make voice stream unrecognisable?

Assess



Acquire- Environmental



Acquire - Lifecycle

??



1. Check the devices between phone and CCM for drops and resource issues

2. Connect a network analyser at the Phone

3. Observe phone during poor voice quality

1. Check the devices between phone and CCM for drops and resource issues

2. Connect a network analyser at the Phone

3. Observe phone during poor voice quality

Act


IOS command line

AirServicesNat# show interfaces | inc drop

Input queue: 0/75/0/0 (size/max/drops/flushes); Tot al output drops: 0




� Redirect IOS command line output to a URL example

AirServicesNat#show interfaces | tee tftp://10.5.1.2/command-out

AirServicesNat#show interfaces | append ftp://10.5.1.2/command-out


IOS command line

AirServicesNat#show buffers input-interface fast 2/0

Header DataArea Pool Rcnt Size Link Enc Flags Input Output

64030310 E0010C4 Small 1 60 7 1 2 80 Fa2/0 None

64031BAC E001984 Small 1 60 7 1 2 80 Fa2/0 None

646A4BD8 E221FE4 Small 1 60 7 1 2 80 Fa2/0 None

646A5D6C E222624 Small 1 60 7 1 2 80 Fa2/0 None

646A7608 E2234E4 Middl 1 292 7 1 280 Fa2/0 None

646A7D10 E223B64 Middl 1 292 7 1 280 Fa2/0 None

Header DataArea Pool Rcnt Size Original Flags caller_pc


XML PI and NETCONF

� Programatic interface that uses CLI or NETCONF RFC4741, RFC4742


XML PI and NETCONF

� Supported in 12.4Thttp://www.cisco.com/en/US/docs/ios/12_2sr/12_2sra/ feature/guide/srnetcon.html


Wireshark – Initial Voice Capture


Wireshark – Analyse Voice Streams


Audacity– Playback Audio


SPAN and VLAN Capture

� Supported on most switches

� Used for examining traffic on a local switch


IP Traffic Export / IP Traffic Capture

� Exports matched traffic out a specific interface or VLAN on the router

� Ideal for Network analyser or probe

� Capture to memory device (flash, tftp, usbflash)

� Only supported on software switching ISR’s 12.4(11)T

http://www.cisco.com/en/US/docs/ios/12_4t/12_4t11/h t_rawip.html


Phone during problem

Press ‘i’ twice to obtain call stastistics



1. The packets are originating from the Media Server (IVR) traversing the local switch/router infrastructure, terminating at the phone.

2. Problem is seen in packets originating from server.

1. The packets are originating from the Media Server (IVR) traversing the local switch/router infrastructure, terminating at the phone.

2. Problem is seen in packets originating from server.

Acquire - Lifecycle


More Video



Possible Causes:

1. Server is “busy” causing a corruption of the audio file as it is played on the LAN

2. The audio file itself is corrupt, incorrectly recorded or using the wrong codec

Possible Causes:

1. Server is “busy” causing a corruption of the audio file as it is played on the LAN

2. The audio file itself is corrupt, incorrectly recorded or using the wrong codec

Analyse



1. Monitor server performance

2. Examine audio file that is on the server for quality problems.

1. Monitor server performance

2. Examine audio file that is on the server for quality problems.

Act


Monitor Server Performance


Examine audio file


What’s next?

Next Problem!

What else can we do that will proactivelyexamine counters and alert us to potential problems?

Next Problem!

What else can we do that will proactivelyexamine counters and alert us to potential problems?

Act


Synchronised Clocks

� Network Time Protocol (NTP)

ntp server (host) [version n]

ntp peer (host) [version n]

http://www.cisco.com/en/US/tech/tk648/tk362/tk461/t sd_technology_support_sub-protocol_home.html


Embedded Event Manager (EEM)

� Applet-based policies� Defined via CLI� Simpler

� Tcl-based policies� Programmed in Tcl� As complex as you want

Think of a policy as an action

registered to an event

ED notifies EEM Server; which

triggers interested policies


Periodic MIB Data Collection

� Device polls specific MIB counters

� Stores this locally (memory)

� Periodically transfers data to server (tftp, rcp, and ftp)

� Introduced 12.3(2)T

http://www.cisco.com/en/US/docs/ios/netmgmt/configu ration/guide/nm_mib_collect_trans.html


Second Problem



1. Internet access is down

2. Users complaining

3. Large Business Impact

1. Internet access is down

2. Users complaining

3. Large Business Impact

Assess



??

Acquire - Lifecycle


R2

R3

R1 R4

R5

R6

R7

R9

R8

PC BPC A

Network Topology Diagram

Internet



1. Gather Source and Destination addresses

2. Verify problem existence in the network

3. Identify which device(s) is causing the issue

1. Gather Source and Destination addresses

2. Verify problem existence in the network

3. Identify which device(s) is causing the issue

Act


Gather Source and Destination addresses

� Translate the customer complaint to useful information

� Helps identify what are possible causes

� Narrow problem down to a specific lifecycle through the network


Verify problem existence in the network

� Gain access to the devices experiencing the issue

� Confirm the customer’s symptoms

� Find alternate device exhibiting same issue

� Recreate in a lab environment


Traceroute and Debug ip icmp

� Traceroute from source to destination

� Traceroute in the opposite direction

� Enable “debug ip icmp” on either side of suspected device

� Turn off console and terminal logging

no logging console

no logging monitor

logging buffered

� Traceroute again to observe debugs

http://www.cisco.com/warp/public/63/ping_traceroute .html#traceroute


R2

R3

R1 R4

R5

R6

R7

R9

R8

PC BPC A


Internet


Using access-lists to find packet flows

� Configure access lists ingress and egress

access-list 100 deny ip host a.b.c.d host w.x.y.z

access-list 100 permit ip any any

� Verify access-list matches

show access-list 100


Netflow and IP accounting

� Configure netflow ingress on the interface

ip route-cache flow ORip flow ingress

Verify access-list matches

show ip cache flow

� IP Accounting on the egress

ip accounting [access-violations] [output-packets]show ip accounting

http://www.cisco.com/en/US/docs/ios/12_4/netflow/configuration/guide/onf_bcf.html#wp1047360http://www.cisco.com/en/US/docs/ios/12_3/ipaddr/command/reference/ip1_i1g.html#wp1091971


What now?



1. Problem narrowed down to a single device R3

2. Traffic direction is broken toward the internet

1. Problem narrowed down to a single device R3

2. Traffic direction is broken toward the internet

Acquire - Lifecycle



Possible Causes:

1. Access-list blocking traffic

2. Physical interface problems

3. Forwarding problem

Possible Causes:

1. Access-list blocking traffic

2. Physical interface problems

3. Forwarding problem

Analyse



1. Implement a workaround to minimise the impact

2. Gain console access to the device in question

3. Step through possible causes to identify the culprit

1. Implement a workaround to minimise the impact

2. Gain console access to the device in question

3. Step through possible causes to identify the culprit

Act


Access-list

� Check ingress and egress interface configuration

show run interface [Interface name]

� Verify access-list configuration and deny matches

show access-list [number}


Check Ingress and Egress Interfaces

� Display interface counters

show interfaces [Interface name]

� Check the command output for the following

Duplex SettingInput and output dropsCRC’sOverrunsunderrunsIgnoresThrottles

http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml


Forwarding verification steps

� Start by checking the routing table entry

� Check arp entry for next hop

� Verify switching methodshow interfaces [Interface name] stats

show cef interface [Interface name]

show cef not-cef-switched

� Check cef entriesshow ip cef w.x.y.z detail (w.x.y.z destination ip)

show adjacency a.b.c.d (a.b.c.d next hop ip)

http://www.cisco.com/en/US/tech/tk827/tk831/tk102/tsd_technology_support_sub-protocol_home.html


Forwarding architectures

� Centralised software

� Centralised hardware

� Distributed software

� Distributed hardware


Root CauseDrop adjacency in hardware cef on GSR linecard

show ip hardware-cef exact-route a.b.c.d w.x.y.z

FIB: 0x45AE0980, FIB->hwleaf: 0x7004FD60 PSA node: 0x86000FE0PSA leaf from PSA node: 0x78003F80

Leaf FCR 1, psa_node 0x78003F80 found 1 deep

Prefix w.x.y.zLeaf FCR 1, psa_node 0x78003F80 found 1 deepdefault psa ip loadbalance

(hw rpfmask 0x80000000)16 paths (hw maxpath 0)

Hash 1,3,5,7,9,11,13,15: psa adjacency: 0x700C8D40 ( hw_adj0x700C7D80)

[0-7] loq ABAB mtu 4 (Drop) oq BABA ai 0 oi 00000000 oac lFFFF (encaps size 0)

punt gather 210 (bufhdr size 32 Punt profile 16)counters 15458387815 bytes, 134893531 pkts; reporte d 0 bytes,

0 pkts.Drop Adjacencya.b.c.d -> w.x.y.z hash: 5


What Else?

1. Scripts and Syslog

2. Test fix in lab

3. Interface usage graphs

4. Apply change management

1. Scripts and Syslog

2. Test fix in lab

3. Interface usage graphs

4. Apply change management

Act


Scripts and Syslog information

� Very useful when time is limited

� Large amount of Data

� Across many devices

� Check for any configuration changes made

� Track any physical events in the network

� Helps in isolating trigger conditions


Lab Setup and Usage Graphs

� Root cause not found in production

� Problem cleared itself

� Large business impact

� Monitored for network status

� Produce possible causes

� Pin point fault location in lifecycle


Enhanced Object Tracking and IPSLA to avoid “black holes”

ip sla 99icmp-echo source-interface Fa0/0timeout 1000threshold 1000frequency 2

ip sla schedule 1 life forever start-time now!access-list 101 permit icmp any host !route-map track-primary-if permit 10

match ip address 101set interface Fa0/0set default interface Fa0/0

ip local policy route-map track-primary-if!track 1 rtr 99 reachability

delay down 10 up 10!ip route 0.0.0.0 0.0.0.0 Fa0/0 track 1ip route 0.0.0.0 0.0.0.0 Fa0/1 200

http://www.cisco.com/en/US/docs/ios/12_4t/ip_appl/configuration/guide/taipbtrk.html#wp1054537

http://www.cisco.com/web/about/ac123/ac114/downloads/packet/packet/apr04/pdfs/apr04.pdf

(pages 9-12,88)

Internet

Cable

DSL

LAN

Fa0/0

Fa0/1


Change Management

� Develop change plans

� Prepare your team

� Notify Stakeholders

� Implement change plans and document

� Assess gaps

� Implement corrective actions


Third Problem



Concerns:

1. Router’s User Interface cannot be accessed.

2. At risk of missing the provisioning deadline as a result

Concerns:

1. Router’s User Interface cannot be accessed.


Assess



1. Need to “see” the symptom that the user experiences.

2. Need to understand the network topology

1. Need to “see” the symptom that the user experiences.

2. Need to understand the network topology

Act


WebEx – Information sharing

http://www.cisco.com/web/products/webex/index.htmlhttp://www.webex.com/index.html


Screen Shots – Information sharing


Document the network – logical/physical

Internet

Home VPN User

PC 1

Lab

Corporate

R1

2851

ASA

5520

R2

3845

SW1

4507R

SW2

3750G

AD ACS R1

R3

871



Concerns:

1. Router “SDM” interface cannot be accessed.“The page cannot be displayed”


Concerns:

1. Router “SDM” interface cannot be accessed.“The page cannot be displayed”


Assess



Relevant Lifecycles:1. Packet flow along the network lifecycle2. TCP setup3. HTTP Get/Reply lifecycle4. Network device packet processing

Facts:

?

Relevant Lifecycles:1. Packet flow along the network lifecycle2. TCP setup3. HTTP Get/Reply lifecycle4. Network device packet processing

Facts:

?

Acquire - Lifecycle



1. Get access to the network devices to run show commands, debugs etc.

2. Need to look at the traffic at other points in the network.

1. Get access to the network devices to run show commands, debugs etc.

2. Need to look at the traffic at other points in the network.

Act



Internet

Home VPN User

PC 1

Lab

Corporate

R1

2851

ASA

5520

R2

3845

SW1

4507R

SW2

3750G

AD ACS R1

R3

871

Here Here


Console access to devices

Methods:

� Access server (reverse telnet)

� Cross connecting the AUX port from one device to the neighbouring CON port for reverse telnet

� Send someone onsite to physically connect to the console

Once on the device, you might try doing a

“debug ip packet ” on an acl that matches the users traffic.


ASA CaptureSteps to capture:

1. Create an ACL to match traffic you are interested in.access-list tac_acl_in permit ip host host access-list tac_acl_in permit ip host hos t access-list tac_acl_out permit ip host host access-list tac_acl_out permit ip host ho st

2. Capture the ACL traffic.capture tac_cap_out access-list tac_acl_out packet-le ngth 1522 interface outsidecapture tac_cap_in access-list tac_acl_in packet-leng th 1522 interface inside

3. Capture traffic dropped by the ASP drop featurecapture type asp-drop all

4. Capture ARP trafficcapture ethernet-type arp interface ins idecapture ethernet-type arp interface out side

“capture” command supported in 6.2(1) onwards

http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/c1.html#wp2090739


ASA Capture cont…Viewing the capture information:

a) ASA CLI:⇒ show capture

b) Packet Decoder:

1. Download the capture file via FTP/TFTP:copy /pcap capture: tftp://10.1.1.10/< dst_capt_name>

2. Download the capture file via HTTP/HTTPS:https:///capture//pca p

Note: You may need to allow https access to download the files:http server enablehttp http 255.255.255.255 inside

3. Now just open the capture up in Ethereal/Wireshark!

“copy capture” command supported in 7.0(1) onwards

http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/c4.html#wp2104187


ASA Packet tracera) Enable Packet Tracing via the CLI:

syntax: packet-tracer input [detail] example: packet-tracer input outside tcp 10.66.64.25 4 1 10.66.76.45 http

“packet-tracer” command supported in 7.2(1) onwardshttp://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/p.html#wp1830068

Phase: 1Type: CAPTURESubtype: Result: ALLOWConfig:Additional Information:

Phase: 2Type: ACCESS-LISTSubtype: Result: DROPConfig: Implicit RuleAdditional Information:

Result:input-interface: outsideinput-status: upinput-line-status: upoutput-interface: outsideoutput-status: upoutput-line-status: upAction: dropDrop-reason: (acl-drop) Flow is denied by configured rule


ASA Packet tracer cont…b) Enable Packet Tracking via ASDM (GUI):Step 1 In the main ASDM application window, choose Tools > Packet Tracer.

Step 2 Specify the following

- source interface, protocol type, source address, s ource port, destination IP address destination port

Step 3 Click Start to trace the packet.

=> The Information Display Area shows detailed messages about the packet trace.

The packet-tracer feature was added to the ASDM in software version 5.2(1)http://www.cisco.com/en/US/docs/security/asa/asa80/asdm60/user/guide/tools.html#wp1536158



Facts:

1. Packets pass through firewall and reach router

2. No reply packets from router

3. TCP SYN sent by client, no SYN,ACK returned.

Facts:

1. Packets pass through firewall and reach router

2. No reply packets from router

3. TCP SYN sent by client, no SYN,ACK returned.

Acquire - Lifecycle

achieving operational excellence - cisco · presentation_id © 2006 cisco systems, inc. all rights...

Documents