aci troubleshooting tools and - alcatron.net live 2015 melbourne/cisco live... · #clmel aci...

#clmel

ACI Troubleshooting Tools andBest Practices

BRKACI-3001

Gerard Chami – Datacenter Solutions Engineer

Agenda

• Setting the stage for troubleshooting

• Troubleshooting Tools

• Best Practices

– Things to do

– Things to watch out for

• Conclusion

Overview of ACI Fabric Policy MechanismsWorking with ACI

Object

Browser

Python

Overview of ACI Fabric Policy MechanismsLogical Model

fvTenant

fvAEPg

fvRsBd

fvAEPg

fvRsBd

fvRsCtx

Overview of ACI Fabric Policy MechanismsResolved Model

fvTenant

fvAEPg

fvRsBd

fvAEPg

fvRsBd

fvRsCtx

Policy Element

Policy Manager

fvEpPCont

fvLocale

fvStPathAtt

fvIfConn

fvDyPathAtt

fvIfConn

fvEpPCont

fvLocale

fvStPathAtt

fvIfConn

fvDyPathAtt

fvIfConn

Overview of ACI Fabric Policy MechanismsConcrete Model

Policy Element

l2BD l2BD

vlanCktEp

l2RsPathDomAtt

vxlanCktEp

fvEpPCont

fvLocale

fvStPathAtt

fvIfConn

fvDyPathAtt

fvIfConn

Overview of ACI Fabric Policy MechanismsHardware Programing – Forwarding Plane

BGPospf

vrf overlay-1

interfaces

l2BD l2BD

vlanCktEp

l2RsPathDomAtt

vxlanCktEp

Setting The Stage For Troubleshooting

Check health scores to narrow down affected scope

Check for faults in the system. If anything fails deployment then faults are raised

Check the resolved object model is present on both APIC and relevant Leafs

Check the concrete objects are present on the relevant Leafs

Verify iNXOS using the iNXOS shell commands

Troubleshooting Checklist

Troubleshooting Tools

Graphic User Interface (GUI)GUI Tools

FaultsHealth Audits Events

Statistics Call-home Syslogs SNMP

APIC Command Line (iShell)APIC SSH Access

admin@fab1_apic1:~> lsaci debug mit

admin@fab1_apic1:~> cd aciadmin@fab1_apic1:aci> lsadmin fabric l4-l7-services system tenants vm-networking

admin@fab1_apic1:~> cd debugadmin@fab1_apic1:debug> lsapic1 apic2 apic3 leaf1 leaf2 spine1 spine2

admin@fab1_apic1:~> cd mitadmin@fab1_apic1:mit> lscomp dbgs expcont fwrepo topology uni

SSH to the APIC

CLI Available at the Switch

leaf101# lsaci bootflash data dev isan lib mit proc sys usb var bin controller debug etc lc logflash mnt sbin tmp usr volatile

leaf101# vshCisco NX-OS Software

Enter NXOS shell

leaf101# vsh_lcmodule-1#

Enter NXOS hardware internals

CLI Shell

leaf101# bcm-shell-hwbcm-shell.0>

Entering the broadcom shell

It is also possible to execute VSH/bcm-shell-hw commands directly from iShell, using the syntaxvsh -c “<command>” vsh_lc –c “<command>” bcm-shell-hw “<command>”

Visore – It’s Italian for Viewer!

Moquery - Command Line Cousin to Visore!

admin@apic1:~> moquery -d uni/tn-gchami-tn

Total Objects shown: 1

# fv.Tenant

name : gchami-tn

childAction :

descr :

dn : uni/tn-gchami-tn

lcOwn : local

modTs : 2015-02-04T23:27:05.622+00:00

monPolDn : uni/tn-common/monepg-default

ownerKey :

ownerTag :

rn : tn-gchami-tn

status :

uid : 15374

ScriptsCobra SDK ACI toolkit

API Inspector – Built into the GUI

Rest API - Postman

Tools in Action

Health, Faults, Events, Audits & Objects

Health Score Degraded - Identification

Navigating to the System Health Dashboard will identify the switch that has a

diminished health score

• Double clicking on that leaf will allow

navigation into the

faults raised on that

device. Here we click

on rtp_leaf1

Drilling Down

Double Click on Degraded Health Score or Highlight the Health Tab

Health Score

Getting to Object Fault

Interface 1/35 on this

Leaf having issues

Interface has a fault due to

being used by an EPG

however interface is

missing an SFP transceiver

Faults in ACI A fault is a Managed Object (MO)

contained in M.I.T.

It is a child of the affected MO

It has the following properties:

severity

lifecycle

description

timestamps

Faults RN is “fault-<code>”, for example, fault-F123

Can be queried by DN and class (fault:Inst)

chassis-1

card-1 card-2

port-1

fault-F123

fault -F456

fault -F789

Timer and severity values can be customized using monitoring policies

Fault Lifecycle

Look for the “faults” tab on the right

Faults in GUI

Keep an eye out for faults indications

Faults Using MoqueryGetting all faults in txt to analyze later:

leaf1# moquery -c faultInst > /tmp/fault-20141112.txtleaf1# ls -l /tmp/fault-20141112.txt-rw------- 1 admin admin 40113 Nov 13 13:37 /tmp/fault-20141112.txt

Want to get all configuration failed fault ?

leaf1# moquery -c faultInst -f 'fault.Inst.code == "F0467"' | egrep "cause|dn"cause : configuration-faileddn : uni/epp/fv-[uni/tn-testTenant2/ap-testAP/epg-testEPG]/nwissues/fault-F0467

Want that in json?leaf1# moquery -c faultInst -o json

{"imdata": [

{"faultInst": {

"attributes": {"dn": "sys/phys-[eth1/11]/fault-F1186","domain": "infra","code": "F1186","occur": "1","subject": "failure-to-deploy","severity": "warning","descr": "Port configuration failure.

Events in GUI

• Much like other navigation / HISTORY / EVENTS

Accounting - Audit Log

• A mechanism to track user-initiated configuration changes

• When a user creates/modifies/deletes an MO, we create an “audit record”containing affected MO DN, user name, timestamp and change details

• System also creates logs for log-in/log-out to controllers and nodes

• Similar to an entry in a log file: once created, they are never modified

• Configuration change logs are MOs of class aaaModLR

• Login/logout logs are MOs of class aaaSessionLR

• Accounting logs get deleted only when a maximum number specified in a retention policy is hit

Audit Log

Who created that ?

Use Moquery or Visore to Check the Model

fvTenantfvApfvAEPgfvRsBdfvBdfvRsCtxfvCtxfvSubnet…

fvCtxDeffvBDDegfvEpPfvlocalefvStPathAttfvDyPathAttfvIfconn

l3Ctxl2BDvlanCktEpvxlanCktEpl2RtDomIfConnvlanRsPathDomAttvlanRsVlanEppAttvxlanRsVxlanEppAtt

Show vlans Show system internal epm …

Vsh_lc:

Show system internal eltmc info…Show system internal epmc info

For YourReference

Datapath Troubleshooting Tools

iPing CLI

iping [options] <target ip address>

options:

-V vrf name (tenant:context)-c count-i wait-p pattern-s packet size-t timeout-S source ip address or source interface

Spine1

Leaf2Leaf1

Tenant: gchami

Context: vrf01

Subnet: 100.0.1.254/24

100.0.2.254/24

Iping –V gchami:vrf01 –S 100.0.1.254 100.0.1.1

100.0.1.1

TEP:10.0.96.95 TEP:10.0.96.92

iping from leaf1

iping from leaf2

• Recommend to set the source ipaddress to make clear which

gateway address is used• ICMP echo reply packet to the

remote leaf node is relayed by the

local leaf node

iPing Internal

iTraceroute CLI

Node traceroute:itraceroute <dst-ip> [<pld-size>]

Tenant traceroute:

For vlan encapsulated source EPitraceroute <dst-ip> vrf <vrf-name> [ encap vlan [<vlan-encap>] ] [ payload <pld-size> ]

For VxLAN encapsulated source EPitraceroute <dst-ip> vrf <vrf-name> encap vxlan [<vxlan-encap>] dst-mac <dst-mac> [ { payload <pld- size> } ]

Tenant iTraceroute - Example

pod2-leaf1# itraceroute 10.11.1.11 vrf RDTenant traceroute to 10.11.1.11, tenant VRF RD, source encap vlan-2101, from [10.0.40.66], payload 56 bytes

Path 11: TEP 10.0.64.65 intf eth1/33 0.746 ms2: TEP 10.0.40.65 intf eth1/97 0.490 ms

Path 21: TEP 10.0.64.64 intf eth1/33 0.812 ms2: TEP 10.0.40.65 intf eth1/98 0.526 ms

Spine1

Pod2-Leaf4Pod2-Leaf1

TEP: 10.0.40.66 TEP:10.0.96.92

Spine2

TEP: 10.0.64.64 TEP: 10.0.64.65

10.11.1.11

Node iTraceroute - Example

pod2-leaf1# itraceroute 10.0.40.95

Node traceroute to 10.0.40.95, infra VRF overlay-1, from [10.0.40.66], payload 56 bytes

Path 1

1: TEP 10.0.64.64 intf eth1/33 0.611 ms

2: TEP 10.0.40.95 intf eth1/98 0.608 ms

Path 2

1: TEP 10.0.64.65 intf eth1/33 0.473 ms

2: TEP 10.0.40.95 intf eth1/97 0.540 ms

Spine1

Pod2-Leaf4Pod2-Leaf1

TEP: 10.0.40.66 TEP:10.0.96.92

Spine2

TEP: 10.0.64.64 TEP: 10.0.64.65

ACI Span

Infrastructure Span:• Meant for traffic to/from access ports.

• Infra SPAN supports both local and to remote (ERSPAN) destinations.

• Infra SPAN can also be filtered by an EPG.

• Configured in Fabric access policies Troubleshoot policies Span (source and dest)

Fabric Span:• Meant for traffic to/from fabric ports (Leaves and Spines).

• Fabric SPAN supports remote destinations only, and can be filtered with a Bridge-domain or Network Context.

• Configured in Fabric Fabric policies Troubleshoot policies span

Tenant Span:• Meant for traffic to/from EPGs and supports only remote destination.

• Configured in The concerned Tenant Troubleshoot policies Span (remote always)

Atomic Counters

• Troubleshooting tool to count packets and bytes between a source and a destination

• Only packets that traverse the fabric are counted

• Locally switched packets are not counted

• Packets switched in the hypervisors are not counted

• There are two types of counters: “ongoing” and “on demand” counters

• NTP must be properly setup and operational on each nodes and APIC (check on node with show ntp peer-status)

Ongoing Atomic Counters

• Ongoing Atomic Counters are not user-configurable

• They count packets at the infrastructure level: the source and destination of the flow are Tunnel End Points (TEPs)

• Example: all packets sent from L1 to L3

• Paths are unidirectional– L1-to-L3 ≠ L3-to-L1

L1 L2 L3 L4

On Demand Atomic Counters

• On Demand counters are configured by the tenant to troubleshoot issues at the level of individual applications

• The source and destination can be EPs, EPGs, IP addresses or “Any”

• For example, packets from

EP1 to EPg2

EP1 EP2 EP3EPG2

Gathering Logs

Tech Support Features

• One interface to collect tech-support from any subset of fabric components and features

• Save to fabric, or export to remote server

• On-demand or periodic

• Configurable data collection

• Downloadable via http from the fabric

• Tech-Support are HUGE !!! (multi gig of tar data)

• They mostly contain logs useful for development. For Postmortem of an event recommended to get Tech-support of APIC’s and impacted leaves ASAP as some logs rollover quickly.

Create Tech Support Policy

In Admin Import/Export export policies Techsupport …

For YourReference

ACI Core Files

APIC Cluster Troubleshooting

admin@ifav1-ifc1:~> acidiag fnvread

ID Name Serial Number IP Address Role State LastUpdMsgId

---------------------------------------------------------------------------------------------------------------------------------------------

1017 ifav1-leaf1 SAL17267Z9S 10.0.63.127/32 leaf active 0

1018 ifav1-leaf2 SAL1739D5WU 10.0.63.125/32 leaf active 0

1200 ifav1-spine1 SAL1748H575 10.0.63.126/32 spine active 0

APIC Debug Commandsacidiag fnvread

Show Fabric node vector

APIC Debug Commands

Replica id State 6=UP

Leadership

APIC where it

is running

“acidiag rvread” shows replica which are not healthy

“acidiag rvread <svc><shard><replica>” to see the state of one replica

acidiag rvread

APIC Debug Commands

Cluster size Chassis ID ActiveSummary of

replica health

acidiag avread

Show APIC controller application vector

ACI Best Practices

Things to do

Plan Your Naming Wisely!

Every Object In ACI Must Be Named.

TenantApp

Profile

Bridge

Domain

Private

Network Contract

Filter

Subnet

EPGAttachable

Entity Profile

Filter

Interface

Profile

Switch

Profile

Interface

Selector

Domain

VLAN PoolPhysical

Domain

Outside

A Good Naming Scheme Is Essential.

VLAN Pools

VL-DVS

VL-AVS

VL-L2-Out

VL-L3-Out

Filters

flt-http

flt-https

flt-sql

Interface Profiles

iprof-ucs

iprof-fex

iprof-ext-switch

Setup NTP!

Symmetry is Beautiful!

Choose Matching Interface Numbers

1/10 1/10

Makes For Simpler Policy!

Regular Config Exports

Plan Your Maintenance!

Upgrade Maintenance Groups

Maintenance group 1 Maintenance group 2

Things To Watch Out For!

The ACI Fabric Cannot Currently Be Used as a ‘Transit’ Network.

No Transit Routing Through Fabric.

ACI Fabric

Context / VRF

L3 out

In this example, 192.168.1.0/24 will not

be advertised to Router B.

Router A Router B

192.168.1.0/24

Use a Unique ACI ‘infra’ IP Range when Provisioning the APIC.

VTEP addresses are allocated to nodes in the fabric automatically based on the pool configured at fabric initialisation.

Future services may cause overlapping address space issues.Changing the infra IP range is difficult – so choose unique range.

Leaf 110.0.104.95

Leaf 210.01.104.96

Leaf 310.0.104.97

Spine 110.0.104.92

Spine 210.0.104.93

Infra range:

10.0.0.0/16

Try to Choose a Unique ‘infra’ VLAN within the Fabric.

With AVS, the ‘infra’ VLAN gets extended out of the fabric.

Leaf 1 Leaf 2 Leaf 3

Spine 1 Spine 2Infra VLAN:

If the default infra VLAN of 4093 is used – this becomes

an issue if AVS is deployed ‘behind’ a Nexus 7K, etc due to reserved VLAN ranges on that platform.

The infra VLAN should be chosen carefully if this

scenario is required.

Nexus 7K

Remember, Adjacent OSPFDevices must use be Configured for NSSA!

External

Router

External

Router

Leaf Leaf Leaf Leaf

Spine Spine

router ospf 1

vrf Blue

area 0.0.0.1 nssa

vrf Red

area 0.0.0.2 nssa

Give us your feedback and receive a

Cisco Live 2015 T-Shirt!

Complete your Overall Event Survey and 5 Session

Evaluations.

• Directly from your mobile device on the Cisco Live

Mobile App

• By visiting the Cisco Live Mobile Site

http://showcase.genie-connect.com/clmelbourne2015

• Visit any Cisco Live Internet Station located

throughout the venue

T-Shirts can be collected in the World of Solutions

on Friday 20 March 12:00pm - 2:00pm

Complete Your Online Session Evaluation

Learn online with Cisco Live! Visit us online after the conference for full

access to session videos and

presentations. www.CiscoLiveAPAC.com

Thank you.

aci troubleshooting tools and - alcatron.net live 2015 melbourne/cisco live... · #clmel aci...

Documents

aci croatia 2014 aci certificates. governance and quality of...

ics-aci quick reference document...aci (batch system, former...

cisco aci troubleshooting kubernetes and openshift ·...

progress report update of aci - kmutt aci meeting... ·...

hoja de vida luis enrique garcÍa reyes estudios … ·...

concrete institute of...

aci component catalog - aci building systems component...

aci concrete sustainability forum€¦ · aci/pca...

cisco aci with ansible collections · 5/12/2020 ·...

handbuch der fvk-bewehrungsprodukte€¦ · usa aci • aci...

cisco aci multi-site troubleshooting guide, release...

troubleshooting asa firewalls - alcatron.net live 2015...

progress report update of aci (2018) - kmutt aci... ·...

dhcp relay in aci - cisco support community · pdf...

**********************************************************...

cisco apic troubleshooting guide...aciでのsnmp...

aci anywhere: extending the aci fabric - cisco.com · §...

cisco aci basics and updates · cisco aci 1.2 release...

aci management reporting system of aci

network insights resources for cisco aci and cisco dcnm...