cisco dmvpn troubleshooting guide

Copyright © 2015 CiscoNet Solutions All Rights Reserved

Overview

The following are standard commands for troubleshooting DMVPN connectivity. The DMVPN is comprised of IPsec/GRE tunnels that connect branch offices to the data center. DMVPN troubleshooting requires the network engineer to verify neighbor links, routing and VPN peer connectivity. The GRE protocol is required to support routing advertisements. The VPN peer connection is comprised of IKE and IPsec security association exchanges. DMVPN connectivity for branch offices and data center routers (DC-R1 and DC-R2) is shown with Figure 1. There is Tunnel 100 to DC-R1 and Tunnel 200 (failover) to DC-R2 for redundancy.

1. Verify Neighbor Connectivity

2. Ping NBMA IP Addressing

3. Verify Tunnel Interfaces Up

4. Traceroute NBMA IP Addressing

5. Verify GRE Tunnels Up

6. Verify IKE Connectivity

7. Verify IPsec Connectivity

8. Verify DMVPN Sessions

9. Verify NHRP Registration

10. Examine Router Log Files

Figure 1 Standard DMVPN Topology


1. Verify Neighbor Connectivity

The following command will verify Layer 2 connectivity between branch office routers and data center routers. Branch-R1# show cdp neighbor [detail] Device ID Local Intrfce Holdtme Capability Platform Port ID DC-R2.domain.com Tunnel200 150 R I ASR1001-X Tunnel200 DC-R1.domain.com Tunnel100 140 R I ASR1001-X Tunnel100 DC-R1# show cdp neighbor [detail] Device ID Local Intrfce Holdtme Capability Platform Port ID Branch-R1.domain.com Tunnel100 141 R S I ISR4321/K Tunnel100 Branch-R2.domain.com Tunnel100 161 R I ISR4331/K Tunnel100 Branch-R3.domain.com Tunnel100 159 R I ISR4451/K Tunnel100

2. Ping NBMA Physical IP Addresses

The following command will Ping from data center routers to Branch-R1. That verifies packets are egressing the physical interface. Note the Ping RTT latency and any packet loss occurring. DC-R1# ping 192.168.70.1

DC-R2# ping 192.168.70.1

3. Verify Tunnel 100 and Tunnel 200 Interfaces Up

The following commands verify the tunnel interfaces are up between Branch-R1 and data center routers (DC-R1 and DC-R2). In addition it will list interface settings and any errors. DC-R1# show interfaces tunnel 100

DC-R2# show interfaces tunnel 200

Branch-R1# show interfaces tunnel 100

Branch-R1# show interfaces tunnel 200


Example: Branch-R1# show interfaces tunnel 100 Tunnel100 is up, line protocol is up Hardware is Routing Tunnel Description: DMVPN connection to data center Internet address is 10.255.70.1 /24 MTU 17900 bytes, BW 100 Kbit/sec, DLY 50000 usec, reliability 255/255, txload 255/255, rxload 255/255 Encapsulation TUNNEL, loopback not set Keepalive set (10 sec), retries 3 Tunnel source 192.168.70.1, destination 192.168.25.1 Tunnel protocol/transport GRE/IP Key disabled, sequencing disabled, checksumming of packets disabled Tunnel TTL 255, Fast tunneling enabled Tunnel transport MTU 1476 bytes Last input never, output never, output hang never Last clearing of "show interface" counters never Output queue 0/0, 0 drops; input queue 0/75, 0 drops Five minute input rate 3445 bits/sec, 34 packets/sec Five minute output rate 233 bits/sec, 10 packets/sec 0 packets input, 0 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 0 packets output, 0 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets, 0 restarts

4. Traceroute Encrypted Packets

The following command will confirm routing path from branch office to data center routers. The traceroute command is based on physical NBMA IP addresses. Branch-R1# traceroute 192.168.25.1

Branch-R1# traceroute 192.168.26.1

5. Verify GRE Tunnels Up and Routing Advertisements

The following commands verify EIGRP routing at DC-R1 for branch office connectivity. The neighbor command shows established EIGRP adjacencies for any configured tunnels. The additional commands verify all EIGRP routes available to VPN peers. DC-R1# show ip eigrp neighbors EIGRP-IPv4, Address-Family Neighbors for AS (100)

H Address Interface Hold Uptime SRTT RTO Q Seq 0 10.255.70.1 Tu100 11 3w0d 4 100 0 42340 0 10.255.71.1 Tu100 12 5w0d 4 100 0 29630 0 10.255.72.1 Tu100 13 6w0d 4 100 0 34970


DC-R1# show ip route eigrp

10.0.0.0/24 is subnetted, 1 subnets C 10.0.0.0 is directly connected, Tunnel100 C 192.168.0.0/24 is directly connected, GigabitEthernet0/1 DC-R1# show ip route

D 192.168.70.1/24 [90/2944000] via 10.255.70.1, 00:1:12, Tunnel100 10.0.0.0/24 is subnetted, 1 subnets C 10.255.70.1 is directly connected, Tunnel100 C 192.168.0.0/24 is directly connected, GigabitEthernet0/1

6. Verify IKE Connectivity

The following command verifies IKE connectivity status to branch offices from DC-R1 router. The normal IKE state = QM IDLE for branch routers and data center routers. DC-R1# show crypto isakmp sa dst src state conn-id slot status

192.168.70.1 192.168.25.1 QM_IDLE 1026 0 ACTIVE 192.168.71.1 192.168.25.1 QM_IDLE 1023 0 ACTIVE 192.168.72.1 192.168.25.1 QM_IDLE 1024 0 ACTIVE Verify IKE policies at branch and data center routers with the following command. The ISAKMP policies must match with remote peer. Branch-R1# show crypto isakmp policy Default IKE policy Protection suite of priority 65507 encryption algorithm: AES - Advanced Encryption Standard (128 bit keys). hash algorithm: Secure Hash Standard authentication method: Rivest-Shamir-Adleman Signature Diffie-Hellman group: #5 (1536 bit) lifetime: 86400 seconds, no volume limit Protection suite of priority 65508 encryption algorithm: AES - Advanced Encryption Standard (128 bit keys). hash algorithm: Secure Hash Standard authentication method: Pre-Shared Key Diffie-Hellman group: #5 (1536 bit) lifetime: 86400 seconds, no volume limit Protection suite of priority 65509 encryption algorithm: AES - Advanced Encryption Standard (128 bit keys). hash algorithm: Message Digest 5 authentication method: Rivest-Shamir-Adleman Signature Diffie-Hellman group: #5 (1536 bit) lifetime: 86400 seconds, no volume limit


Protection suite of priority 65510 encryption algorithm: AES - Advanced Encryption Standard (128 bit keys). hash algorithm: Message Digest 5 authentication method: Pre-Shared Key Diffie-Hellman group: #5 (1536 bit) lifetime: 86400 seconds, no volume limit Protection suite of priority 65511 encryption algorithm: Three key triple DES hash algorithm: Secure Hash Standard authentication method: Rivest-Shamir-Adleman Signature Diffie-Hellman group: #2 (1024 bit) lifetime: 86400 seconds, no volume limit Protection suite of priority 65512 encryption algorithm: Three key triple DES hash algorithm: Secure Hash Standard authentication method: Pre-Shared Key Diffie-Hellman group: #2 (1024 bit) lifetime: 86400 seconds, no volume limit Protection suite of priority 65513 encryption algorithm: Three key triple DES hash algorithm: Message Digest 5 authentication method: Rivest-Shamir-Adleman Signature Diffie-Hellman group: #2 (1024 bit) lifetime: 86400 seconds, no volume limit Protection suite of priority 65514 encryption algorithm: Three key triple DES hash algorithm: Message Digest 5 authentication method: Pre-Shared Key Diffie-Hellman group: #2 (1024 bit) lifetime: 86400 seconds, no volume limit Note: Verify firewall rules permit required protocols through firewall. That could include UDP 500 (ISAKMP), UDP 4500 (NAT Traversal), ESP (IP 50), AH (IP 51).

7. Verify IPsec Connectivity Verify IPsec connectivity with the following command at branch office and data center routers. The peer NBMA IP address is optional and specifies a specific VPN peer instead of listing all IPsec sessions. Router# show crypto ipsec sa [peer nbma ip address]

Is the local and peer IP address correct ?

Are packets are being encrypted and decrypted ?

Does the IPsec transform set match between branch office and data center router ?


The following is for IPsec Tunnel 100 from Branch-R1 (192.168.70.1) to DC-R1 (192.168.25.1) Branch-R1# show crypto ipsec sa interface: Tunnel100 Crypto map tag: Transport-DCR1, local addr 192.168.70.1 protected vrf: (none) local ident (addr/mask/prot/port): (192.168.70.1/255.255.255.255/43/0) remote ident (addr/mask/prot/port): (192.168.25.1/255.255.255.255/43/0) current_peer 192.168.25.1 port 500 PERMIT, flags={origin_is_acl,} #pkts encaps: 1033449, #pkts encrypt: 10337449, #pkts digest: 1033449 #pkts decaps: 840001, #pkts decrypt: 8440001, #pkts verify: 840001 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 0, #pkts compr. failed: 0 #pkts not decompressed: 0, #pkts decompress failed: 0 #send errors 0, #recv errors 0 local crypto endpt.: 192.168.70.1, remote crypto endpt.: 192.168.25.1 plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb (none) current outbound spi: 0xCA72039F(3396469663) PFS (Y/N): N, DH group: none inbound esp sas: spi: 0x55AEB00D(143511693) transform: esp-256-aes esp-sha-hmac, in use settings ={Transport, } conn id: 2058, flow_id: ESG:58, sibling_flags FFFFFFFF80000003, crypto map: Transport-DCR1 sa timing: remaining key lifetime (k/sec): (4599810/1122) IV size: 16 bytes replay detection support: Y replay window size: 512 Status: ACTIVE(ACTIVE) inbound ah sas: inbound pcp sas: outbound esp sas: spi: 0xCA72039F(339669663) transform: esp-256-aes esp-sha-hmac, in use settings ={Transport, } conn id: 2057, flow_id: ESG:57, sibling_flags, FFFFFFF80000003, crypto map: Transport-DCR1 sa timing: remaining key lifetime (k/sec): (4592706/1122) IV size: 16 bytes replay detection support: Y replay window size: 512 Status: ACTIVE(ACTIVE) outbound ah sas: outbound pcp sas: interface: Tunnel200 Crypto map tag: Transport-DCR1, local addr 192.168.70.1 interface: Tunnel100 Crypto map tag: Transport-DCR1, local addr 192.168.70.1


The following is for IPsec Tunnel 100 from DC-R1 (192.168.25.1) to Branch-R1 (192.168.70.1) DC-R1# show crypto ipsec sa interface: Tunnel100 Crypto map tag: Tunnel100-DCR1, local addr 192.168.25.1 protected vrf: (none) local ident (addr/mask/prot/port): (192.168.25.1/255.255.255.255/43/0) remote ident (addr/mask/prot/port): (192.168.70.1/255.255.255.255/43/0) current_peer 192.168.70.1 port 500 PERMIT, flags={origin_is_acl,} #pkts encaps: 5031159, #pkts encrypt: 5031159, #pkts digest: 5031159 #pkts decaps: 579973, #pkts decrypt: 5799973, #pkts verify: 579973 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 0, #pkts compr. failed: 0 #pkts not decompressed: 0, #pkts decompress failed: 0 #send errors 0, #recv errors 0 local crypto endpt.: 192.168.25.1, remote crypto endpt.: 192.168.70.1 plaintext mtu 1458, path mtu 1500, ip mtu 1500, ip mtu idb (none) current outbound spi: 0x55AEB00D(1437511693) PFS (Y/N): N, DH group: none inbound esp sas: spi: 0xCA72039F(3396469663) transform: esp-256-aes esp-sha-hmac, in use settings ={Transport, } conn id: 5401, flow_id: HW:3401, sibling_flags FFFFFFFF80000003, crypto map: Tunnel100-DCR1 sa timing: remaining key lifetime (k/sec): (4598745/2456) IV size: 16 bytes replay detection support: Y replay window size: 1024 Status: ACTIVE(ACTIVE) inbound ah sas: inbound pcp sas: outbound esp sas: spi: 0x55AEB00D(1437511693) transform: esp-256-aes esp-sha-hmac, in use settings ={Transport, } conn id: 5402, flow_id: HW:3402, sibling_flags FFFFFFFF80000003, crypto map: Tunnel100-DCR1 sa timing: remaining key lifetime (k/sec): (4603398/2456) IV size: 16 bytes replay detection support: Y replay window size: 1024 Status: ACTIVE(ACTIVE)


8. Verify DMVPN Peer Sessions

The command shows status for all DMVPN peers (branch) connected to DC-R1 data center router. DC-R1# show dmvpn Legend: Attrb --> S - Static, D - Dynamic, I – Incomplete N - NATed, L - Local, X - No Socket # Ent --> Number of NHRP entries with same NBMA peer NHS Status: E --> Expecting Replies, R --> Responding UpDn Time --> Up or Down Time for a Tunnel Interface: Tunnel100, IPv4 NHRP Details Type: Hub, NHRP Peers:3 # Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb ----- ----------------- --------------- ----- -------- ----- 1 192.168.70.1 10.255.70.1 UP 00:02:58 D 1 192.168.71.1 10.255.72.1 UP 00:02:43 D 1 192.168.72.1 10.255.74.1 UP 00:02:43 D

9. Verify NHRP Registration

The following is the NHRP registration at Branch-R1. Note request failed error from line 2. Branch-R1# show ip nhrp nhs detail Legend: E=Expecting replies, R=Responding, W=Waiting Tunnel100:

10.255.70.1 RE NBMA Address: 192.168.25.1 priority = 0 cluster = 0 req-sent 4813 req-failed 2 repl-recv 4681 (00:03:04 ago) Protection Socket Requested: FALSE Current Request ID: 9396

Tunnel200: 10.255.71.1 RE NBMA Address: 192.168.26.1 priority = 0 cluster = 0 req-sent 4847 req-failed 2 repl-recv 4682 (00:00:04 ago)

Protection Socket Requested: FALSE Current Request ID: 9397

The following is a typical NHRP registration cache at DC-R1. Note the tunnel uptime and NBMA assigned address for the branch offices. The NBMA IP address and loopback management IP address for multiple branch offices are noted.

DC-R1# show ip nhrp 10.255.70.1/32 via 10.255.70.1/32 Tunnel100 created 4d10h, expire 00:09:31 Type: dynamic, Flags: registered used nhop NBMA address: 192.168.70.1


10.255.72.1/32 via 10.255.72.1/32 Tunnel100 created 4d10h, expire 00:07:13 Type: dynamic, Flags: registered used nhop NBMA address: 192.168.71.1 10.255.74.1/32 via 10.255.74.1/32 Tunnel100 created 1d11h, expire 00:08:40 Type: dynamic, Flags: registered used nhop NBMA address: 192.168.72.1

9. Verify Router Log File for Errors The following are standard troubleshooting commands for examining VPN tunnel setup and any resulting log file errors. Router# debug crypto isakmp sa

Router# debug crypto ipsec sa

Router# show log

" atts not acceptable " in log file is mismatch with local and remote peer IPsec transform set.

IKE Connectivity State Reference IKE Phase 1 Main Mode

MM_NO_STATE - The ISAKMP SA has been created.

MM_SA_SETUP - The VPN peers have agreed on ISAKMP SA parameters.

MM_KEY_EXCH - Diffie-Hellman public keys and shared secret keys exchanged between VPN peers. Note ISAKMP SA not unauthenticated yet.

MM_KEY_AUTH – The ISAKMP SA’s are authenticated between VPN peers and transition to QM_IDLE occurs. The Quick Mode exchange starts and IKE Phase 2 can begin.

IKE Phase 1 Aggressive Mode

AG_NO_STATE - The ISAKMP SA has been created.

AG_INIT_EXCH - VPN peers have done initial exchange, however SA’s not authenticated yet.

AG_AUTH - The ISAKMP SA’s authenticated in aggressive mode for VPN peers. There is transition to QM_IDLE, and IKE Phase 2 quick mode exchange begins

IKE Phase 2 Quick Mode

QM_IDLE - The ISAKMP SA is idle. The VPN peers are authenticated and available for subsequent quick mode exchanges.


Cloud Design Fundamentals

Cloud Deployment Strategies Nexus 1000V Switch, vPath, CSR 1000V Router CWS, SecureX, Virtual Security Gateway, IER FabricPath, OTV, Citrix NetScaler 1000V, PfR ASA 1000V, VXLAN, InterCloud Fabric, vNAM Hybrid, Amazon VPC, SaaS, Service Chaining Cisco Intelligent WAN, Akamai Connect, vWAAS Cloud and Enterprise Network Integration Cloud Readiness Assessment Migration Case Study Examples, Quizzes

Cloud Design Fundamentals is an essential reference for network engineers and systems

engineers. The book explains methodologies, principles and techniques used for migrating and integrating the enterprise and cloud network. There is a discussion of cloud deployment models. In addition traditional and newer cloud-based architectures are compared. There is a chapter with technical subject matter relevant to cloud migration. That includes coverage of WAN protocols, optimization features, campus design, virtualization and cloud security solutions. In addition newer features and protocols are explained including OTV, SecureX, vPath and FabricPath. The virtual appliances include CSR 1000V router, Nexus 1000V for VMware switch, vWAAS, ASA 1000V cloud firewall and NetScaler 1000v load balancer. The multilayered design approach is comprised of virtual campus design, application and data redundancy, internet connectivity application services, cloud security, configuration workflow and management The audience will apply the multilayered design strategy with case studies and quizzes for SaaS and hybrid cloud deployments. Shaun Hummel is author of Cisco Design Fundamentals and Network Performance Optimization Guide for CCNA, CCNP and CCIE Engineers. Cloud Design Fundamentals is available at Amazon.com

http://www.amazon.com/s?ie=UTF8&page=1&rh=i%3Abooks%2Cp_27%3AShaun%20Hummel

cisco dmvpn troubleshooting guide

Education