Download - Troubleshooting Forwarding Plane
Troubleshooting Forwarding Plane Telefónica España
Support Advanced Services EMEA [email protected] V1.1 20110527
2 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
INTERFACE AND BYPASS-ROUTING
3 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
PING TO REMOTE ADDRESS – DEFAULT
M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1
RE
CB
FPC #1
SIBs
PIC
so-5/1/0 .2
so-1/1/0 10.3.3/30
RE
CB
FEB #4 PIC 1/0 4/1
user@m320> ping 10.2.2.2
Assuming default configuration (without default-address-selection)
(10.2.2.1, 10.2.2.2) (SA, DA) =
(10.2.2.2, 10.2.2.1)
FPC #4
4 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
LINK KEEPALIVES & ROUTING PROTOCOL PACKETS
M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1
RE
CB
FPC #1
SIBs
PIC
so-5/1/0 .2
so-1/1/0 10.3.3/30
RE
CB
FEB #4 PIC 1/0 4/1
HDLC/OAM/LMI keepalives follow the same hardware path as ping
FPC #4
5 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
PING TO REMOTE ADDRESS – INTERFACE OPTION
M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1
RE
CB
FPC
#1
so-5/1/0 .2
so-1/1/0 10.3.3/30
RE
CB
(10.3.3.1, 10.2.2.2) (SA, DA) =
(10.2.2.2, 10.3.3.1)
user@m320> ping 10.2.2.2 interface so-1/1/0
The interface option only alters the source IP address by default
PIC
1/0
FEB
#4
FPC #4
PIC 4/1
FEB
#5
FPC #5
PIC 5/1 PIC
1/1
Similar to source option (monitor traffic interface displays packets)
6 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
PING TO REMOTE ADDRESS – BYPASS-ROUTING
M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1
RE
CB
FPC #1
SIBs
PIC
so-5/1/0 .2
so-1/1/0 10.3.3/30
RE
CB
FEB #5 PIC 1/1 5/1
user@m320> ping 10.2.2.2 interface so-1/1/0 bypass-routing
bypass-routing allows to force the packet to go out a given interface
(10.3.3.1, 10.2.2.2) (SA, DA) =
(10.2.2.2, 10.3.3.1)
FPC #5
Only works properly at SONET/SDH interfaces
7 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
PING TO LOCAL ADDRESS – DEFAULT
M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1
RE
CB
FPC #1 PIC
.2
RE
CB
FEB #4 PIC 1/0 4/1
user@m320> ping 10.2.2.1
By default, ping to local address does not leave the RE
FPC #4
Checked with show chassis ethernet-switch statistics
(10.2.2.1, 10.2.2.1)
(10.2.2.1, 10.2.2.1)
8 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
PING TO LOCAL ADDRESS – INTERFACE
M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1
RE
CB
FPC #1 PIC
.2
RE
CB
FEB #4 PIC 1/0 4/1
user@m320> ping 10.2.2.1 interface so-1/0/0
The interface option only alters the source IP address by default
FPC #4
Still packet does not leave the Routing Engine
(10.2.2.1, 10.2.2.1)
(10.2.2.1, 10.2.2.1)
9 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
PING TO LOCAL ADDRESS – BYPASS-ROUTING
M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1
RE
CB
FPC #1 PIC
.2
RE
CB
FEB #4 PIC 1/0 4/1
user@m320> ping 10.2.2.1 interface so-1/0/0 bypass-routing
FPC #4 (10.2.2.1, 10.2.2.1)
(10.2.2.1, 10.2.2.1)
bypass-routing allows to force the packet to go out a given interface Only works properly at SONET/SDH interfaces
10 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
LOCAL AND REMOTE LOOPBACK
11 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
PING TO REMOTE ADDRESS – LOOPBACK REMOTE
RE
CB
FPC #1 PIC
RE
CB
FEB #4 PIC 1/0 4/1
user@m320> ping 10.2.2.2
Packet loops until TTL expires
FPC #4
The RE originating ICMP echo packets receive ICMP time exceeded On the right: packet copies sent to the PFE hit firewall filters (counting)
(10.2.2.1, 10.2.2.2)
(10.2.2.1, 10.2.2.2)
user@m320> ping 10.2.2.2 PING 10.2.2.2 (10.2.2.2): 56 data bytes 36 bytes from 10.2.2.1: Time to live exceeded Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 8212 0 0000 01 01 1f91 10.2.2.1 10.2.2.2
M320 M120
12 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
PING TO REMOTE ADDRESS – LOOPBACK LOCAL (I)
RE
CB
FPC #1 PIC
RE
CB
FEB #4 PIC 1/0 4/1
user@m320> ping 10.2.2.2
Packet loops until TTL expires
FPC #4
The RE originating ICMP echo packets receive ICMP time exceeded
(10.2.2.2, 10.2.2.1) (10.2.2.1, 10.2.2.2)
M320 M120
user@m120> ping 10.2.2.1
user@M320# edit interfaces so-1/0/0 [ no-keepalives ; sonet-options loopback local; ]
user@M120# edit interfaces so-4/1/0 [ no-keepalives ; sonet-options loopback local; ]
(*) May be necessary to remove “family iso” and “family mpls” for the test
13 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
PING TO REMOTE ADDRESS – LOOPBACK LOCAL (II)
RE
CB
FPC #1 PIC
RE
CB
FEB #4 PIC 1/0 4/1
user@m320> ping 10.2.2.2
Output firewall filters require double lookup and fabric pass
FPC #4
The RE originating ICMP echo packets receive ICMP time exceeded
(10.2.2.2, 10.2.2.1) (10.2.2.1, 10.2.2.2)
[edit firewall family inet filter prueba-loopback] term unico then { count paquetes; accept; } [edit interfaces so-1/0/0 unit 0 family inet] filter output prueba-loopback;
user@m120> ping 10.2.2.1
SIBs
14 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
PING TO LOCAL ADDRESS – LOOPBACK REMOTE
RE
CB
FPC #1 PIC
RE
CB
FEB #4 PIC 1/0 4/1
user@m320> ping 10.2.2.1 interface so-1/0/0 bypass-routing
FPC #4 (10.2.2.1, 10.2.2.1)
(10.2.2.1, 10.2.2.1)
user@M320# set interfaces so-1/0/0 no-keepalives
user@M120# set interfaces so-4/1/0 no-keepalives
user@M120# set interfaces so-4/1/0 sonet-options loopback remote
M320 M120
Two simultaneous troubleshooting paths Original packet looped by the remote PIC and sent back to originator On the right: packet copies sent to the PFE hit firewall filters (counting)
15 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
PING TO LOCAL ADDRESS – LOOPBACK LOCAL
RE
CB
FPC #1 PIC
RE
CB
FEB #4 PIC 1/0 4/1
user@m320> ping 10.2.2.1 interface so-1/0/0 bypass-routing
FPC #4
(10.2.2.2, 10.2.2.1) (10.2.2.1, 10.2.2.2)
M320 M120
user@m120> ping 10.2.2.2 interface so-4/1/0 bypass-routing
user@M320# edit interfaces so-1/0/0 [ no-keepalives ; sonet-options loopback local; ]
user@M120# edit interfaces so-4/1/0 [ no-keepalives ; sonet-options loopback local; ]
bypass-routing allows to force the packet to go out a given interface Only works properly at SONET/SDH interfaces
16 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
IMPLEMENTATION DETAILS
17 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
IMPLEMENTATION DETAILS – BYPASS-ROUTING
RE
CB
FPC #1 PIC
RE
CB
FEB #4 PIC 1/0 4/1
With a logical loop, the packet traverse both PIC framers
FPC #4
This would spot interoperability issues between the framers Problem can be isolated to be caused by the line or by the endpoints
• Not necessarily by which of the endpoints
(10.2.2.1, 10.2.2.1)
M320 M120
user@m320> ping 10.2.2.1 interface so-1/0/0 bypass-routing
SONET FRAMERS
18 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
IMPLEMENTATION DETAILS – LOOPBACK
RE
CB
FPC #1 PIC
RE
CB
FEB #4 PIC 1/0 4/1
The PIC just loops the SONET frame
FPC #4
The PIC framers do not modify the SONET frame at all There is no way with loopbacks to traverse both PIC framers
(10.2.2.1, 10.2.2.2)
M320 M120
user@m320> show interfaces so-1/0/0 extensive | match trace
Received path trace: m320 so-1/0/0
Transmitted path trace: m320 so-1/0/0
SONET FRAMERS
loopback local at M120 so-4/1/0 loopback remote at M120 so-4/1/0
19 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
IMPLEMENTATION DETAILS – TRANSIT PING
RE
CB (control)
The record-route option is useful to spot fabric failures Different hardware path followed for each type of packet
FEB #4
PIC 4/0 FPC #4
FEB
#5 PIC 5/1 FPC #5
CB (fabric)
transit ping with record-route option transit ping with no special option
20 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
Interface Type loopback mode ping options
local remote interface interface & bypass-routing
SONET/SDH Yes Yes Yes Yes GE/100GE Yes No Yes No
ATM Yes No Yes No FR (E3 IQ) Yes No Yes No
NON-SONET INTERFACE – CAPABILITIES The bypass-routing option can be used, but it does not work
The remote loopback option is not available either
How to use loops? ping the remote link address, and count TTL expired packets
user@m320> show system statistics icmp | match exceed
time exceeded: 177
21 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
TOS OPTION
22 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
IP Precedence DSCP 3 bit 6 bit 8 bit
bin dec bin dec bin dec hex 000 0 000000 0 00000000 0 0x00 001 1 001000 8 00100000 32 0x20 010 2 010000 16 01000000 64 0x40 011 3 011000 24 01100000 96 0x60 100 4 100000 32 10000000 128 0x80 101 5 101000 40 10100000 160 0xa0 110 6 110000 48 11000000 192 0xc0 111 7 111000 56 11100000 224 0xe0
TOS VALUES – DIFFERENT ENCODINGS The table below displays the formats used for:
3-bit & 6-bit bin: inet-precedence & dscp – classifiers & rewrite-rules 3-bit & 6-bit dec: from precedence & dscp | traffic-class – firewall filters 8-bit dec: ping command tos option, both for IPv4&IPv6 8-bit hex: dscp or traffic class field displayed in tcpdump decoding
23 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
ICMP AS CONTROL TRAFFIC Control traffic COS is determined by the RE
The Routing Engine sets the DSCP/IP Precedence as well as the internal FC+PLP values of a packet before sending it to the Egress PFE
By default, locally originated ICMP goes to queue 0 Regardless of the ping “tos” value
The ping “tos” option can change the DSCP/IP Precedence but not the queue the packet goes to
The ICMP echo reply mirrors the DSCP/IP Precedence from the original ICMP echo request
In Junos OS 10.4 output lo0 firewall filters support actions to rewrite FC,PLP (queue number) and DSCP/IP Precedence independently before sending packet to PFE
Egress control packets are never processed by rewrite rules
24 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
LAB DIAGRAMS
25 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
M320 M120 so-1/0/0 so-4/1/0 10.2.2/30 .1
so-5/1/0 .2
so-1/1/0 10.3.3/30
M7i
lo0.0 10.100.3.3
NETWORK DIAGRAM
lo0.0 10.100.1.1
lo0.0 10.100.2.2
26 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
FAILURE SCENARIO – COMPLETE TRAFFIC LOSS IN A LINK
RE
CB
FPC #1 PIC
RE
CB
FEB #4 PIC 1/0 4/1 FPC #4
M320 M120
test failure test success
27 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
FAILURE SCENARIO – TRAFFIC DEGRADATION IN A SINGLE LINK
RE
CB
FPC #1 PIC
RE
CB
FEB #5
PIC 1/1 5/1 FPC #5
M320 M120
test failure test success
PIC 1/1
FEB #4
PIC 4/1
PIC 4/0
FPC #4
28 Copyright © 2009 Juniper Networks, Inc. www.juniper.net
FAILURE SCENARIO – TRAFFIC DEGRADATION IN A DOUBLE LINK
RE
CB
FPC #1 PIC
RE
CB
FEB #5
PIC 1/1 5/1
FPC
#5
M320 M120
test failure test success
PIC 1/0
FEB #4
PIC 4/1
PIC 4/0
FPC #4
M7i
CHANGE-LOG:
When Who Rev What
20110526 [email protected] v1.0 Presented to customer 20110527 [email protected] v1.1 Added lab slides, sending to customer