junipertroubleshooting-12433222279-phpapp02
TRANSCRIPT
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 1
Control and Forwarding plane
Synchronization1) 100-Mbps fxp1 Ethernet link is used between
RE and PFE2) For M320 case, 100-Mbps Ethernet switch is
being used to provide a dedicated link to each FPC. For RE, these links are presented at bcm0
3) Fxp0: management interface4) Fxp2: communication between Primary RE and
backup RE3) Forwarding table (FT) can hold over 800,000
routes.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 2
Difference between M7i and M10i
1. Redundant RE: M10i support, not M7i2. Built-in Adaptive Service: M7i. M10i needs an
external AS PIC.3. RE: the same
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 3
System storage
3 types of storages:1) Compact Flash(ad0) : built-in at the board. 2) Hard Drive(ad1)3) External storage -PCMCIA card(da0??) -USB(da1??)
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 4
JUNOS CLI basics Space bar to complete a command Command :Help topic <command> for general concepts Command: help reference <command to look> for configuration syntax Rebooting system: request system reboot Shut down system: request system halt Log and Trace files are located at /var/log Command: Show log | messages | file-name At more prompt, use forward slash(/) to search or use “h” to get a context
help screen Log commands examples: - show log messages | match so-0/3/1 | match TRAP --- AND -- - show log messages | mach “fpc | sfm | kernel” --- OR --- Monitor log/trace in real time: monitor start file-name | match fail Stop monitoring in real time: monitor stop Enable/disable real-time output to screen: Esc-Q Stop traceing operation: delete flag open Truncate(clear) log/trace files: clear log file-name Delete log/trace files: file delete file-name
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 5
JUNOS CLI basics Entering configuration: Type configure or edit Exclusive configuration (configure exclusive) and Private
configuration (configure private??) Moving within the configuration hierarchy: edit (equivalent to
cd), up, top, exit (to previous location in the hierarchy) Show command at configuration mode vs. show command at
operational-mode Relative configuration commands Starting with JUNOS5.3:
top - top show system login (show system login no matter
where you are. Examples: - top edit protocols ospf ( to enter protocols ospf no
matter where you are) Viewing configuration in operational mode: show
configuration < configuration path> View configuration with set: show xxx | display set Viewing candidate configuration: show chassis alarm, show
(at the current sub-hierarchy)
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 6
JUNOS CLI basics Change the candidate configuration. Examples: - set alarm sonet lol red - delete alarm sonet pll Display difference between the candidate and active
configurations: At the current statement-path, show | compare Viewing difference in files. Example: - file show filename1 | compare file filename2 - show configuarion | compare rollback number Removing statements: delete Delete the statements and all its subordinate statements
and identifieres. Wildcard delete. Example: wildcard delte interfaces fe-* Ignore portion of the configuration hierarchy: deactivate / activate Disable an interface: set disable interface Delete and disabled interface: delete interface <interface-name>
disable
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 7
JUNOS CLI basics Activate a configuration commit ----- candidate file is checked, actived and marked as the current operational sofware configuration file. commit check ----- only validate a candidate configuration without placing it into effect. rollback n -------- recover the previous configuration. And then commit rollback 0 is current configuration First 3 roll back (1-3) are stored in solid-state flash disk /config/juniper.conf.n (n=1-3) rest roll back (4-49) are stored in hard disk /var/db/config commit confirmed time-out ---- temporarily activate a configuration
(default is 10 minutes). If the final commit is not executed, the system will performs a “rollback 1, commit” commands.
commit synchronize ---- after committed on the master RE internally copied and committed on the backup RE automatically. commit at time ----- commit at some time clear system commit ---- cancel a pending commit
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 8
JUNOS CLI basics Save a configuration
save filename
save terminal -- for copy and paste into other others show | display set – create configuration for simplifying configuration
editing.
Loading configuration files ( load and then commit) load override filename – override the current config with the loaded one. Do it at
the root of the configuration hierarchy. load merge filename - combine the new and old load merge terminal (then copy/paste hierarchical configuration)
load replace filename – statements with replace tag will replace the statements with the same name
load relative – load at where it is current at the configuration hierarchy.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 9
Junos CLI Basics Only save the configuration under certain hierarchy. To
save the whole configure, issue this command at the top of the hierarchy.
#Save <filenam> Display the contents of the file you saved #Run file show <filename> To load a configuration after clear the current
configuration # delete
#show#load override <filename>
To recover a mistake made previously after committing.#rollback 1
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 10
Junos CLI Basics
show log messages | last Show log interactive-commands | match restart Use sysctl –a to display kernel parameters. sysctl –a | grep icmp (under shell prompt) show chassis 0 pic slot 1 information.
Show chassis pic fpc-slot 0 pic-slot 1 Master switchover
Request chassis cfeb master switchRequest chassis routing-engine master switch
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 11
Junos CLI Basics Find out who logins the system and kick out some particular
users. show system users reequest system logout help syslog <log strings>Example: lab@santro-re0> help syslog ACCT_ACCOUNTING_FERROR Name: ACCT_ACCOUNTING_FERRORMessage: Unexpected error <error-code> from file <filename>Help: Error occurred during file processingDescription: An error prevented the accounting statistics
process from processing the indicated file.Type: Error: An error occurredSeverity: warning
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 12
Junos CLI Basic
show configuration with inheritance show configuration interfaces ge-4/3/3 | display
inheritance
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 13
Syslog
set system syslog file messages any notice
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 14
Hardware troubleshooting process Show chassis alarms Show chassis craft-interface Show log messages Show log chassid Monitor start [message | chassid] Show chassis hardware Show chassis fpc Show pfe stat error Show interface terse Show interface detail Show log <log-file-name>
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 15
Display PIC status Show chassis pic fpc-slot 0 pic-slot 1Example: lab@santro-re0> show chassis pic fpc-slot 0 pic-slot 1 FPC slot 0, PIC slot 1 information: Type 10x 1GE(LAN), 1000 BASE ASIC type H chip State Online PIC version 1.13 Uptime 1 day, 22 hours, 25 minutes, 17 seconds
PIC port information: Fiber Xcvr vendor Port Cable type type Xcvr vendor part number Wavelength 0 GIGE 1000SX SM FINISAR CORP. FTRJ8519P1BNL-J2 850 nm
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 16
Boot image If you need to reboot from PCMCIA card, you
need to copy a special image called jinstall-mediaxxxx.
Interrupt normal bootHit space when the system is rebooting until it goes to either boot: or OK prompt. If you get boot: prompt, the loader is not run yet. You need to do this:
Boot: /boot/loader Change a boot device at OK promptOk nextboot compact-flashOk reboot
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 17
Interfaces
Disable(admin down) an interface Admin LinkSo-0/1/1 down upSo-0/1/1.0 up down
Deactivate an interfaceAdmin Link
So-0/1/1 up up
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 18
RE overview (Q: how to find out RE <-> Platform compatibility list?) Primary coopy of JUNOS resides on the flash memory. Use this command to create a
backup copy: request system snapshot Mgd manages CLI RE has different versions: RE-333, RE-400, RE-600, RE-1600. Each RE is supported by
certain platforms. RE uses Intel processor from P III to P IV. Use this command to find out what RE is being used: show chassis hardware. Hard disk monitoring: Self-Monitoring Analysis and Reporting Technology
System(SMART). From 5.5, SMART is enabled by default. To disable: set system processes disk-monitoring disable
Configuration file compression: default starting Release 7.0 (maybe). To enable:set system compress-configuration-file
RE versionsRE5(RE-400): only supported in M7i and M10iRE4(RE-600): All M and T series. Except M7i/M10i/M320. The only RE to have flash memory upgradeRE3 (RE-333): M5/10/20/40/40e, and M160RE-1600: M320 and T320/T640. Using Broadcom chipset for Ethernet connectivity to PFE.
While used on M320, the GE link is supported as bcm0. While on T-series, 100- Mbps is supported(???)
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 19
PFE overview on M-series Different names but referring to the route lookup module:1. M40 – System Control Board (SCB)2. M20 – System Switch Board (SSB)3. M5/10 – FPC and SCB are combined into a single
board called the Forwarding Engine Board (FEB)4. M7i/10i – Compact FEB (CFEB)5. M40e and M160 – Switching and Forwarding Module
(SFM). 4 SFM on M160, each one provides 25% of lookup capability. 2 SFM on M40e, only one can be active.
Special stuff on M40e and M160 platform: MCS card (Miscellaneous Control Subsystem): provide control and monitoring functions for the various components in the chassisPCG (PFE clock generation): 125-MHZ signal. Redundant PCGS
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 20
PFE on T-series and M320 M320 is different than T and M-series. It is a combio of two using I and J chips. T640 PFC2 has single PFE, PFC3 has two PFE
T-Series nonblocking cross-bar switch fabric – Switch Interface Boards(SIBs).
T320: 3 SIBs with 2 are active. SIB 1 and 2 are active, SIB0 is standby. SIB0 has only one high-speed line (HSL) connected to FPC. SIB1 and SIB2 has 2 HSL. So when SIB0 becomes active, system performance is degraded.
T640: 5 switch fabric cards or SIBs, 4 are active, 1 standby. Something like Cisco’s GSR.
M320: 4 SIBs. M320 FPC1: use single I chip
M320 FPC2: dual I chip, thus two PFE
M320 FPC3: dual J chip, thus two PFE
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 21
Physical Interface Cards (PIC) IP service PIC is to hardware assist complex packet processing and
has no physical ports.
IP service PIC include:1)Tunnel service PIC for IP-IP, GRE tunnel and PIM-SM tunnel. 2)Multlink PIC: Multilink Point-to-Point (MLPPP) and Multilink Frame Relay (MLFR, FRF 1.5)
Hot-Pluggable except M20 and M40 which need to remove FPC.
Take PIC offline before physically removing it. Otherwise would cause system damage or PFE reset.
Packet loss is expected on M-serials except M320 because of FPC reset.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 22
Flexible PIC Concentrator (FPC) Support 1 to 4 PICs. M160 OC-192 has an FPC support only one PIC. Each FPC on M-serial pooled to create shared memory switch fabric.
So hot-swap FPC cause system to repartition the shared memory pool; 200 ms packet loss.
FPC is hot-swappable in all platforms except M5 and M10 which is using FEB. However M7i and M10i are OK even using CFEB.
Build-in FPC at some high-speed quad-wide PICs such as OC-48c/STM-16 for M20/40. OC-192c/STM-64 SONET/SDH on M160.
New FPC to support reuse of old PICs: M160 FPC1: intend to reuse M20/40 PIC
M160 FPC2: design to support M160 only PIC, such as OC-48cFPC3: support native T-series PICs.
T640 only support FPC2 and FPC3. How to power off FPC?
set chassis fpc power off
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 23
M-series System Board General functions
Names very by platforms1. M40 – System Control Board (SCB)2. M20 – System Switch Board (SSB)3. M5/10 – FPC and SCB are combined into a single board called the Forwarding Engine Board
(FEB)4. M7i/10i – Compact FEB (CFEB)5. M40e and M160 – Switching and Forwarding Module (SFM). 4 SFM on M160, each one provides 25%
of lookup capability. 2 SFM on M40e, only one can be active.
Enhanced System Boards:- 2nd generation Internet Processor II ASIC (not on M5/10 and M7i/10i)- support 840K routing entries, double from old board 420K.- Double on-chip memory to 16MB on IP II- CPU memory 128 M for M40, 256M for M20, M40e and M160.- Increased CPU speed to 256 MHZ. - First shipped with JUNOS 5.5 Sep 2002.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 24
IP II ASIC Performance: 40 Mpps, 40 byte with 80K prefixes at routing
table.
Packet processing features:Filtering, sampling, logging, counting, load balancing
All M-series have enhanced S-board which as IP II ASIC. M5/10 doesn’t have enhanced S-board.
T-series might contain as many as 16 IP II ASIC. Each FPC has one or two PFE which contains its own IP II ASIC.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 25
Craft Interface What is it?
Collection of mechanisms on M-series and T-series View System status messagesTrouble shooting
Where is it?On the front of the chassis
What does it have?System status LEDsFPC/PIC online/offline buttons.LCD screen provide status reporting for the entire system.
What alternatives on other platforms? M7i: FIC (Fixed Interface Card)provide PIC offline/online buttonsM10i: HCM (High-Availability Chassis Manager) Card provide PIC offline/online bottons.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 26
Password recovery Connect to console Power cycle the RE and watch it booting up Enter a space character at the boot loader quick
help manue to get a command prompt (don’t enter space too quickly)
Enter “boot –s” When system boots up, answer “ recovery” to
recover password Follow the on-screen steps to change password Commit the change Reboot the system again.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 27
Coredump analysis – using syslog messageStep 1: Get the stack trace from syslog messageslab@hissy> show log messages | find "machine check"
Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 machine check caused by error on the PC
I Bus
Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error detect register 1: 0x08, 2: 0x00
Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error ack count = 0
Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error address: 0x08004014
Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 PCI bus error status register: 0x02
Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 was the PCI master
Dec 5 01:51:17 hissy tnp_sfm_3 C/BE bits: I/O read [0b0010]
Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 error detection reg1: PCI cycle
Dec 5 01:51:17 hissy tnp_sfm_3 mpc106 PCI status reg: parity error
Dec 5 01:51:17 hissy tnp_sfm_3 ^B
Dec 5 01:51:17 hissy tnp_sfm_3 last message repeated 7 times
Dec 5 01:51:17 hissy tnp_sfm_3 Registers:
Dec 5 01:51:17 hissy tnp_sfm_3 R00: 0x000e8c4c R01: 0x0775dad4 R02: 0x0000334
4 R03: 0x00000000
Dec 5 01:51:17 hissy tnp_sfm_3 R04: 0x0775dae0 R05: 0x00142e34 R06: 0x06006b3
6 R07: 0x00006b36
Dec 5 01:51:17 hissy tnp_sfm_3 R08: 0x00142e4c R09: 0x88000000 R10: 0x0000000
0 R11: 0x00000000
Dec 5 01:51:17 hissy tnp_sfm_3 R12: 0x00100004 R13: 0x000cc411 R14: 0x0000c43
0 R15: 0x00040000
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 28
Coredump analysis – using syslog messageDec 5 01:51:17 hissy tnp_sfm_3 R16: 0x00000000 R17: 0x00041410 R18: 0x0004c420 R19: 0x8004c618Dec 5 01:51:17 hissy tnp_sfm_3 R20: 0x0002c490 R21: 0x00110000 R22: 0x0000000Juniper Confidential. For Internal use only.0 R23: 0x001151ccDec 5 01:51:17 hissy tnp_sfm_3 R24: 0x00000001 R25: 0x00000000 R26: 0x0775db14 R27: 0x06006b36Dec 5 01:51:17 hissy tnp_sfm_3 Stack Traceback:Dec 5 01:51:17 hissy tnp_sfm_3 Frame 01: sp = 0x0775dad4, pc = 0x000e8c4cDec 5 01:51:17 hissy tnp_sfm_3 Frame 02: sp = 0x0775db0c, pc = 0x0005cd9cDec 5 01:51:17 hissy tnp_sfm_3 Frame 03: sp = 0x0775db34, pc = 0x00108914Dec 5 01:51:17 hissy tnp_sfm_3 Frame 04: sp = 0x0775db4c, pc = 0x00108888Dec 5 01:51:17 hissy tnp_sfm_3 Frame 05: sp = 0x0775db54, pc = 0x000eec84Dec 5 01:51:17 hissy tnp_sfm_3 Frame 06: sp = 0x0775db5c, pc = 0x00037e78Dec 5 01:51:17 hissy tnp_sfm_3 Frame 07: sp = 0x0775dc1c, pc = 0x000380f8Dec 5 01:51:17 hissy tnp_sfm_3 Frame 08: sp = 0x0775dcfc, pc = 0x000eeadcDec 5 01:51:17 hissy tnp_sfm_3 Frame 09: sp = 0x0775dd2c, pc = 0x000eefd0Dec 5 01:51:17 hissy tnp_sfm_3 Frame 10: sp = 0x0775dd3c, pc = 0x000f0184Dec 5 01:51:17 hissy tnp_sfm_3 Frame 11: sp = 0x0775dd74, pc = 0x000b28ccDec 5 01:51:17 hissy tnp_sfm_3 Frame 12: sp = 0x0775dd84, pc = 0x000b29f4Dec 5 01:51:17 hissy tnp_sfm_3 Frame 13: sp = 0x0775ddac, pc = 0x000b2a8cDec 5 01:51:17 hissy tnp_sfm_3 Frame 14: sp = 0x0775ddcc, pc = 0x000b2c80Dec 5 01:51:17 hissy tnp_sfm_3 Frame 15: sp = 0x0775ddec, pc = 0x000b2d5cDec 5 01:51:17 hissy tnp_sfm_3 Frame 16: sp = 0x0775de04, pc = 0x0002665c
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 29
Coredump analysis – using syslog messageWhat do I want? I will copy the following into a file called “stack”
single% cat stackDec 5 01:51:17 hissy tnp_sfm_3 Stack Traceback:Dec 5 01:51:17 hissy tnp_sfm_3 Frame 01: sp = 0x0775dad4, pc = 0x000e8c4cDec 5 01:51:17 hissy tnp_sfm_3 Frame 02: sp = 0x0775db0c, pc = 0x0005cd9cDec 5 01:51:17 hissy tnp_sfm_3 Frame 03: sp = 0x0775db34, pc = 0x00108914Dec 5 01:51:17 hissy tnp_sfm_3 Frame 04: sp = 0x0775db4c, pc = 0x00108888Dec 5 01:51:17 hissy tnp_sfm_3 Frame 05: sp = 0x0775db54, pc = 0x000eec84Dec 5 01:51:17 hissy tnp_sfm_3 Frame 06: sp = 0x0775db5c, pc = 0x00037e78Dec 5 01:51:17 hissy tnp_sfm_3 Frame 07: sp = 0x0775dc1c, pc = 0x000380f8Dec 5 01:51:17 hissy tnp_sfm_3 Frame 08: sp = 0x0775dcfc, pc = 0x000eeadcDec 5 01:51:17 hissy tnp_sfm_3 Frame 09: sp = 0x0775dd2c, pc = 0x000eefd0Dec 5 01:51:17 hissy tnp_sfm_3 Frame 10: sp = 0x0775dd3c, pc = 0x000f0184Dec 5 01:51:17 hissy tnp_sfm_3 Frame 11: sp = 0x0775dd74, pc = 0x000b28ccDec 5 01:51:17 hissy tnp_sfm_3 Frame 12: sp = 0x0775dd84, pc = 0x000b29f4Dec 5 01:51:17 hissy tnp_sfm_3 Frame 13: sp = 0x0775ddac, pc = 0x000b2a8cDec 5 01:51:17 hissy tnp_sfm_3 Frame 14: sp = 0x0775ddcc, pc = 0x000b2c80Dec 5 01:51:17 hissy tnp_sfm_3 Frame 15: sp = 0x0775ddec, pc = 0x000b2d5cDec 5 01:51:17 hissy tnp_sfm_3 Frame 16: sp = 0x0775de04, pc = 0x0002665c
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 30
Coredump analysis – using syslog messageStep2: Find out which version and build of the image.
So it is on M160, 4.4B3.2 and build 4.4-20010408-b20191
lab@hissy> show version briefHostname: hissyModel: m160JUNOS base [4.4B3.2] (Export restricted edition)JUNOS Kernel Software Suite [4.4-20010408-b20191]JUNOS Routing Software Suite [4.4-20010408-b20191]JUNOS Packet Forwarding Engine Support [4.4-20010408-b20191]JUNOS Online Documentation Files [4.4-20010408-b20191]
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 31
Coredump analysis – using syslog messageStep 3: Find out which symbol file to use. ‘debug’ package for the crashing code if the crash is in the kernel or routing, or the normalpackage for the PFE. The perl script ‘jemsym’ can be used to decodethe stack. Recent dailies;
single% cd /volume/buildsingle% ls20010201-0805@ 20010217-0805@ 20010305-0805@ 20010320-0910@ 20010405-0810@20010202-0805@ 20010218-0805@ 20010306-0805@ 20010321-0910@ 20010406-0810@
older dailies for released versions;single% cd /volume/ftp/private/unregressed/single% ls3.4/ 4.0/ 4.1/ 4.2/ 4.3/ 4.4/ 5.0/
released code;single% cd /volume/ftp/private/junos/single% ls4.0B1/ 4.0R5/ 4.1R4/ 4.3B1.2/ 4.4B2.1/4.0B2/ 4.1B1.1/ 4.2B1.1/ 4.3B2.1/ 4.4B3.2/
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 32
Coredump analysis – using syslog messagesingle% cp /volume/build/20010408-0810/jpfe-4.4-20010408-b20191-debug.tgz .single% tar zxfv jpfe-4.4-20010408-b20191-debug.tgz+CONTENTS+COMMENT+DESC+INSTALL+REQUIREusr/share/pfe/scb.jbfusr/share/pfe/scb.symusr/share/pfe/scb.elfusr/share/pfe/fpc.jbfusr/share/pfe/fpc.symusr/share/pfe/fpc.elfusr/share/pfe/sfm.jbfusr/share/pfe/sfm.symusr/share/pfe/sfm.elfusr/share/pfe/fpc160.jbfusr/share/pfe/fpc160.symusr/share/pfe/fpc160.elfusr/share/pfe/sbr.jbfusr/share/pfe/sbr.sym
usr/share/pfe/sbr.elf
fpc.sym - M20/M40 fpc stack traces
fpc160.sym -- M160 fpc stack traces
sbr.sym -- M5/M10 stack traces
scb.sym -- M40/M20 S-Board traces
sfm.sym --M160 SFM traces.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 33
Coredump analysis – using syslog message
What is Jemsym file?
#!/usr/local/bin/perl##$Id: jemsym,v 1.7 1998/04/21 01:15:33 jim Exp $##This file takes a Juniper panic stack trace and turns it# into a user-readable output from the symbol table file# for the running micro-kernel.Juniper Confidential. For Internal use only.##By default, gmake produces a symbol table file for each# target, and then you run the text of the panic stack trace,# perhaps saved to a temporary file, as follows:##cat temp-backtrace_file | jemsym target.sym
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 34
Coredump analysis – using syslog messageStep 4: Do the stack trace
single% cat stack | ~dbovis/bin/jemsym usr/share/pfe/sfm.sym0x000e8c4c cchip_ab_pio (0x000e8b2c) +0x1200x0005cd9c pfe_bmemchip_pio_write (0x0005cd44) +0x580x00108914 bchip_write_sram_opaque (0x00108898) +0x7c0x00108888 bchip_write_sram_hton (0x00108878) +0x100x000eec84 bchip_write_sram_mem_val (0x000eec64) +0x200x00037e78 diags_pfe_mem_address_test (0x00037dfc) +0x7c0x000380f8 diags_pfe_mem_test (0x0003802c) +0xcc0x000eeadc bchip_mem_test (0x000eea08) +0xd40x000eefd0 bchip_diags_sram_test (0x000eef30) +0xa00x000f0184 bchip_probe_diag (0x000f00fc) +0x880x000b28cc cm_probe_slot (0x000b284c) +0x800x000b29f4 cm_probe_slots (0x000b297c) +0x780x000b2a8c cm_probe_chassis (0x000b2a64) +0x280x000b2c80 cm_probe_event_loop (0x000b2b98) +0xe80x000b2d5c cm_probe_thread_init (0x000b2ca8) +0xb40x0002665c thread_suicide (0x0002665c) +0x0
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 35
Coredump analysis – using syslog messageStep 4: Do the stack trace
single% cat stack | ~dbovis/bin/jemsym usr/share/pfe/sfm.sym0x000e8c4c cchip_ab_pio (0x000e8b2c) +0x1200x0005cd9c pfe_bmemchip_pio_write (0x0005cd44) +0x580x00108914 bchip_write_sram_opaque (0x00108898) +0x7c0x00108888 bchip_write_sram_hton (0x00108878) +0x100x000eec84 bchip_write_sram_mem_val (0x000eec64) +0x200x00037e78 diags_pfe_mem_address_test (0x00037dfc) +0x7c0x000380f8 diags_pfe_mem_test (0x0003802c) +0xcc0x000eeadc bchip_mem_test (0x000eea08) +0xd40x000eefd0 bchip_diags_sram_test (0x000eef30) +0xa00x000f0184 bchip_probe_diag (0x000f00fc) +0x880x000b28cc cm_probe_slot (0x000b284c) +0x800x000b29f4 cm_probe_slots (0x000b297c) +0x780x000b2a8c cm_probe_chassis (0x000b2a64) +0x280x000b2c80 cm_probe_event_loop (0x000b2b98) +0xe80x000b2d5c cm_probe_thread_init (0x000b2ca8) +0xb40x0002665c thread_suicide (0x0002665c) +0x0
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 36
Coredump analysis – using core files Where to get coredump files? 1) Coredump files are stored at: /volume/ftp/pub/incomfing/<case_number>/<core_filenma>For Example: /volume/ftp/pub/incoming/2008-0104-0511 2) For some freaking .tgz file, you need to do this
gunzip < cosd.core-tarball.0.tgz.2 | tar -xvf - Using GUI
http://jtac-tools.juniper.net/crashdecode/coredump.html
Using Manual methods:Step 1: Using Jdebug to find out the stack traces. jdebug='/volume/buildtools/bin/jdebug‘
/volume/buildtools/bin/jdebug <core_file name>
Examples: The core file is saved at /volume/ftp/pub/incoming/2008-0104-0511/core-SSB0.core.0Step 2: Use query-pr to find out the possible PRs based on the stack trace.
query-pr -m "thread_debug" -m "sched_suspend_thread" –summary
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 37
Coredump analysis – using core (continued)-bash-2.05b$ /volume/buildtools/bin/jdebug core-SSB0.core.0
GNU gdb 6.5 juniper_2006a_411
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "--host=i386-unknown-freebsd4.11 --target=powerpc-juniper-eabi".
#0 0x000330a0 in panic (
format_string=0x25f204 "CCHIP: Too many SRAM parity errors; restart required\n")
at ../ukern/cpu-ppc/ppc603e_panic.c:63
63 asm volatile ("sc");
(gdb) bt
#0 0x000330a0 in panic (
format_string=0x25f204 "CCHIP: Too many SRAM parity errors; restart required\n")
at ../ukern/cpu-ppc/ppc603e_panic.c:63
#1 0x0018bf7c in cchip_error_hardware (C=0x35, hwerror=402653184)
at ../common/drivers/cchip/cchip_int.c:238
#2 0x0018c158 in cchip_error_scan () at ../common/drivers/cchip/cchip_int.c:352
#3 0x0006baec in pfe_error_scan (info=0x0) at ../common/toolkits/pfe/pfe_scb.c:172
#4 0x000da8c8 in cm_handle_pfe_error (rate_limit=FALSE)
at ../common/applications/cm/cm_pfe_restart.c:1463
#5 0x000dabc0 in cm_restart_handle_timer_event (timer=0x35)
at ../common/applications/cm/cm_pfe_restart.c:1652
#6 0x000daff0 in cm_restart_event_loop () at ../common/applications/cm/cm_pfe_restart.c:1898
#7 0x00026fa0 in thread_wake (thread=0x210000) at ../ukern/common/thread.c:572
(gdb)
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 38
Coredump analysis – core file from special image Step 1: to find out the image path using “what” on
image or core file.
-bash-2.05b$ what core-SSB0\[1\].core.3 core-SSB0[1].core.3:
scb release 8.2I20071212_2313_pgoyette built by pgoyette on 2007-12-12 23:14:53 UTC
jtac-bbuild01.juniper.net:/b/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb
-bash-2.05b$ cd /volume/nfsbuild40
-bash-2.05b$ ls
jcano pgoyette ramanathan sdoshi yuris
So the whole path is:
/volume/nfsbuild40/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb
Step 2: Find out the *.elf file. In the above case, it is scb.elf under the above path.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 39
Coredump analysis – core file from special imageSoemtimes it take more trouble to untar the compressed jpfe file to get the elf file.
lab@iggy> show version brief | grep packetJUNOS Packet Forwarding Engine Support [4.0-20000608-s22432]
(From above number I don’t know where to get the jpfe file)single% tar zxfv jpfe-4.0-20000608-regressed-debug.tgz+CONTENTS+COMMENT+DESC+INSTALL+REQUIREusr/share/pfe/scb.jbfusr/share/pfe/scb.symusr/share/pfe/scb.elfusr/share/pfe/fpc.jbfusr/share/pfe/fpc.symusr/share/pfe/fpc.elfusr/share/pfe/sfm.jbfusr/share/pfe/sfm.symusr/share/pfe/sfm.elfusr/share/pfe/fpc160.jbfusr/share/pfe/fpc160.symusr/share/pfe/fpc160.elf
fpc.sym M20/M40 fpc stack traces
fpc160.sym M160 fpc stack traces
sbr.sym M5/M10 stack traces
scb.sym M40/M20 S-Board traces
sfm.sym M160 SFM traces.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 40
Coredump analysis – core file from special image-bash-2.05b$ /volume/cross/cygnus-i386-ppc/bin/gdb-core.ppc -nw
/volume/nfsbuild40/pgoyette/VZ-8.2-20071012/src/juniper/pfe/obj-scb/scb.elf core-SSB0[1].core.3
GNU gdb 4.16-97r2aCopyright 1997 Free Software Foundation, Inc.GDB is free software, covered by the GNU General Public License, and you areThis GDB was configured as "--host=i386-unknown-freebsd2.2.5 --target=powerpc-eabi"...#0 topo_connect (topo=0xd5af08, next=0x28, reconnect=FALSE) at ../common/toolkits/topo/topo.c:428../common/toolkits/topo/topo.c:428: No such file or directory.(gdb) bt ----------------------------------------------------------- #0 topo_connect (topo=0xd5af08, next=0x28, reconnect=FALSE) at ../common/toolkits/topo/topo.c:428#1 0x155a84 in nh_indirect_add_sub (nh=0x2163a3c, unilist=0x0, indirect_elementpp=0x2163a98) at ../common/applications/nh/nh_indirect.c:193#2 0x155a84 in nh_indirect_add_sub (nh=0x2163a3c, unilist=0x0, indirect_elementpp=0x2163a98) at ../common/applications/nh/nh_indirect.c:193# at ../common/applications/pfeman/pfeman_rt.c:413#11 0x276cc in thread_suicide () at ../ukern/common/thread.c:951
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 41
Coredump analysis – Kernel core of special image Find out where is the symbol file by using
what. Ex: /volume/nfsbuild40/pgoyette/VZ-
8.2I20071212_2313/ship/ jkernel-8.2I20080311_1541_jtac-builder-debug.tgz
copy the jkernel file to your home directory and unzip it.
Ex: gunzip < jkernel-8.2I20080311_1541_jtac-builder-debug.tgz | tar -xvf-
Debug the vmcore.0 fileEx: gdb -k kernel.debug vmcore.0
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 42
Coredump analysis – daemon crash1) uncompress the freaking core *.tgz filegunzip < cosd.core-tarball.2.tgz | tar -xvf -
cosd.core.0juniper.confmessagescosd.info.0juniper.conf.1.gz
2) Where is the symbol file by doing “what”bash-2.05b$ what cosd.core.0cosd.core.0: COSD release 7.3R3.6 built by builder on 2006-02-01
08:03:43 UTC
xathanon.juniper.net:/build/xathanon-c/7.3R3.6/obj-i386/juniper/usr.sbin/cosd
getsubopt.c 8.1 (Berkeley) 6/4/93 Copyright (c) 1994 Powerdog Industries. All rights reserved.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 43
Coredump analysis – daemon crash3) Decode the core file-bash-2.05b$ gdb /build/xathanon-c/7.3R3.6/obj-i386/juniper/usr.sbin/cosd/cosd
cosd.core.0GNU gdb 4.18 (FreeBSD)Copyright 1998 Free Software Foundation, Inc.-bash-2.05b$ gdb /build/xathanon-c/7.3R3.6/obj-i386/juniper/usr.sbin/cosd/cosd
cosd.core.0GNU gdb 4.18 (FreeBSD)Copyright 1998 Free Software Foundation, Inc.Core was generated by `cosd'.Program terminated with signal 11, Segmentation fault./usr/lib/libisc.so.2: No such file or directory.#0 0x806d6f2 in cos_ifd_configure (dop=0x81e4300, conf=0x81ba000, name=0xbfbff850 "ge-0/3/0", match_len=10, wc_match=0 '\000', ifd_has_ieee_classifier=1 '\001', errmsg=0xbfbffc70 "", errmsglen=256) at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:27052705 cos_ifd->if_flags |= COS_IFD_CONF_F_IEEE_CLASSIFIER;(gdb) bt#0 0x806d6f2 in cos_ifd_configure (dop=0x81e4300, conf=0x81ba000, name=0xbfbff850 "ge-0/3/0", match_len=10, wc_match=0 '\000', ifd_has_ieee_classifier=1 '\001', errmsg=0xbfbffc70 "", errmsglen=256) at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:2705#1 0x806f851 in cos_config_interfaces (dop=0x81e4280, conf=0x81ba000, errmsg=0xbfbffc70 "", errmsglen=256) at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:3944
#2 0x807bb53 in cos_config (conf=0x81ba000, errmsg=0xbfbffc70 "", errmsglen=256)
at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:10816
#3 0x807be0e in cosd_parse_config (cos_conf=0x81ba000, check_only=0 '\000')
at ../../../../src/juniper/usr.sbin/cosd/cosd_parser.c:10924
#4 0x8069ac4 in main (argc=1, argv=0xbfbffe0c)
at ../../../../src/juniper/usr.sbin/cosd/cosd_main.c:330
(gdb) l2700 } else {
2701 cos_ifd = cos_pat_to_ifd(pnode);
2702 }
2703
2704 if (ifd_has_ieee_classifier) {
2705 cos_ifd->if_flags |= COS_IFD_CONF_F_IEEE_CLASSIFIER;
2706 }
2707
2708 /*
2709 * in commit check, cosd hasn't built its interface data
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 44
Coredump analysis – Software or Hardware issues?
Case #1 Panic, TLB Data miss, Data access etc type of system exceptions:most probably
software related. What you should do is to enable the coredump on the chassisd and gather all the base information mentioned above.
Case #2: pci parity error being reported on the CPU DRAM address space, this means that this isa bogus pci error. The reason is, there is no pci bus connected to the CPU DRAM.Action: In this case, we have to enable the coredump on chassisd and get the coredump
of the PFE component along with the base information. No RMA should be issued.Example:mpc106 machine check caused by error on the PCI Busmpc106 error detect register 1: 0x08, 2: 0x00mpc106 error ack count = 2mpc106 error address: 0x001d0048 < belongs to CPU DRAMmpc106 PCI bus error status register: 0x02mpc106 was the PCI master C/BE bits: I/O read [0b0010]mpc106 error detection reg1: PCI cyclempc106 PCI status reg: parity error < parity error.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 45
Coredump analysis – Software or Hardware issues?
Case #3:There is parity protection enabled (ECC is disabled) on the CPU DRAM, if a hw failure occurs here, the message that you should see is: "memory parity/ECC error".Action: Run the memory diagnostics tests and RMA.
Example:mpc106 machine check caused by error on the Processor Bus <
reported by Processor Busmpc106 error detect register 1: 0x04, 2: 0x00mpc106 error ack count = 0mpc106 error address: 0x02f39e18mpc106 Processor bus error status register: 0x72 transfer type 0b01110, transfer size 2mpc106 error detection reg1: memory parity/ECC error < parity error.mpc106 PCI status reg: parity error
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 46
Coredump analysis – Software or Hardware issues?
Case #3:There is parity protection enabled (ECC is disabled) on the CPU DRAM, if a hw failure occurs here, the message that you should see is: "memory parity/ECC error".Action: Run the memory diagnostics tests and RMA.
Example:mpc106 machine check caused by error on the Processor Bus <
reported by Processor Busmpc106 error detect register 1: 0x04, 2: 0x00mpc106 error ack count = 0mpc106 error address: 0x02f39e18mpc106 Processor bus error status register: 0x72 transfer type 0b01110, transfer size 2mpc106 error detection reg1: memory parity/ECC error < parity error.mpc106 PCI status reg: parity error
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 47
Monitoring - logs Step 1: configure logging file Example: isis { traceoptions { file mike-isis; flag state; flag error; flag spf; flag lsp receive detail; }
Step 2: monitor start <log-file-name>
Step 3: monitor start messageExample:lab@falcons> monitor start mike-isis lab@falcons> monitor start messages
lab@falcons> *** mike-isis ***Feb 5 20:05:53.517506 Updating LSP falcons.00-00 in databaseFeb 5 20:05:53.517654 Updating L2 LSP falcons.00-00 in TED
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 48
Booting up systemrequest system snapshot partition as-primaryrequest system media usb request system reboot media usb - when reboot from another media, all
the file systems will be under this media. request system snapshot part as-primary media compact-flashrequest system reboot media compactrequest system software add /var/tmp/junojseries-8.4R2.4-
domestic.tgz no-validateRequest system snapshot -- make a image at another storage(if you
are using disk, this will mirror the image to CF. If you are using CF, this will makes an image at disk.
request system software delete backup
request system storage cleanupTo remove swap space at the compact-flash:
http://www.juniper.net/techpubs/software/junos/junos85/rn-sw-85
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 49
Tools and quick reference http://clie.juniper.net /volume/build - junos releases and source code. After
8.4, go to extra hierarchy /volume/build/junos. For example: /volume/build/junos/8.4/release/8.4R2.4/ship
http://jam.jnpr.net http://www-in.juniper.net/eng/cvs_pdf/ https://deepthought.juniper.net/app/ http://cvs/cgi-bin/viewcvs.cgi/ http://confluence.jnpr.net/ /volume/current - cvs functional specs /volume/labcores http://rogers.jtac-emea.jnpr.net/wiki/index.php?title=
Enginee
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 50
How to find out what syslog means? [email protected]> help syslog SNMPD_SUBAGENT_NO_RESOURCES
Name: SNMPD_SUBAGENT_NO_RESOURCESMessage: No resources available for subagent (<subagent-name>):<error-message>Help: Subagent resources were temporarily exhaustedDescription: The SNMP agent process (snmpd) uses certain resources forcommunication with subagents. Resources were not available for communication with the indicated subagent.Type: Error: An error occurredSeverity: noticeCause: An internal software failure occurred.Action: Contact your technical support representative.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 51
How to find out the data between 2 proc sockets? 1. Find out the processes ID (use snmpd and mib2d as example)
root@Kelly_RE0% ps -aux | egrep -i "snmpd|mib2d"
root 8322 0.0 0.2 5036 3932 ?? S 4Feb08 0:12.24 /usr/sbin/snmpd -N
root 8302 0.0 0.2 4464 3892 ?? I 4Feb08 0:10.35 /usr/sbin/mib2d –N
2. Find out socket stream.
root@Kelly_RE0% fstat -p 8302
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
.....
root mib2d 8302 17* local stream faab6c80 <-> fab03e60
root@Kelly_RE0% fstat -p 8322
USER CMD PID FD MOUNT INUM MODE SZ|DV R/W
.....
root snmpd 8322 15* local stream fab03e60 <-> faab6c80
3. Then, check the socket data.
root@Kelly_RE0% netstat -Aan | egrep -i "mib2d|snmpd|Send"
PCB Proto Recv-Q Send-Q Local Address Foreign Address (state)
PCB Proto Recv-Q Send-Q Local Address Foreign Address (state)
Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr
f5f4e6c0 stream 0 0 0 faad35a0 0 0 /var/run/snmpd_stream
f5f4b300 stream 0 0 0 faa47aa0 0 0 /var/run/snmpd_stream
f5f4fc20 stream 0 0 0 fab67dc0 0 0
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 52
How to do RMA? 1. Logistics
csr-apac(emea, usa)
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 53
Trouble shoot T-series show chassis hardware show pfe statistics traffic show interface [int] extensive start shell su vty fpc[x] show sys mess show nvram show lchip ifd show ifl brief show lchip [x] error show lchip [x] lout stat show lchip [x] lout sw lsif show lchip [x] lout sw desrd show lchip [x] lout sw hdrf show lchip [x] lout sw nlif show lchip [x] lout hw lsif show lchip [x] lout hw nlif show lchip [x] lout hw hdrf
show lchip [x] lout hw nlif
show lchip [x] stream [stream_#]
show lchip [x] lout registers lsif lsif [stream_#]
( where [stream_#] is the stream you found which corresponds to the
interface that has the problem using the show lchip ifd output above )
show lchip [x] lout registers nlif nlif
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 54
Trouble shoot T-series start shell su vty fpc[x] show sys mess show nvram show lchip ifd show ifl brief show lchip [x] error show lchip [x] lout stat show lchip [x] lout sw lsif show lchip [x] lout sw desrd show lchip [x] lout sw hdrf show lchip [x] lout sw nlif show lchip [x] lout hw lsif show lchip [x] lout hw nlif show lchip [x] lout hw hdrf show lchip [x] lout hw nlif show lchip [x] stream [stream_#] show lchip [x] lout registers lsif lsif [stream_#]
•(where [stream_#] is the stream you have seen on the "show lchip ifd"
•output under the lchip [x])
•show lchip [x] lout registers nlif nlif
•show lchip [x] lout reg nlif dbufpart
•show lchip [x] lout reg nlif bdispmon
•Wait a little, hopefully after a few more errors go by.
•show nchip [x] all
•show mq [x] wan stat
•show mq [x] wan stream active stat
•Show chassis fabric topology Show chassis fabric sibs Show chassis fabric fpcs
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 55
How to trouble shoot SNMP and MIB2drtsockmon -c mib2drtsockmon -ge mib2dshow snmp statistics extensivenetstat –anshow system virtual-memory[edit snmp]lab@Johnny-re1# showcommunity public;traceoptions { file test size 10m; flag all;}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 56
How to trouble shoot routing and forwarding issues? FPC7(FED1DSRJ01-LAB-re0 vty)# show route
ip prefix 192.12.1.2 IPv4 Route Table 0, default.0, 0x0: Destination NH IP Addr Type
NH ID Interface --------------------------------- --------------- -------- -----
--------- 192.12.1.2 Hold 716
ge-7/0/4.0
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 57
How to trouble shoot routing and forwarding issues? install@FED1DSRJ01-LAB-re0> show route forwarding-table destination 192.12.1.2 Routing table: inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif 192.12.1.2/32 dest 1 192.12.1.2 hold 716 2 ge-7/0/4.0
Routing table: __juniper_private1__.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default perm 0 rjct 116 1
Routing table: __juniper_private2__.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default perm 0 rjct 196 1
Routing table: FED1J1MIS.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default perm 0 rjct 521 1
Routing table: TEST-L3VPN.inet Internet: Destination Type RtRef Next hop Type Index NhRef Netif default perm 0 rjct 530 1
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 58
How to trouble shoot routing and forwarding issues? install@FED1DSRJ01-LAB-re0> show arp MAC Address Address Name Interface
Flags 02:01:00:00:00:05 10.0.0.5 10.0.0.5 em0.0
none 00:04:80:9d:b5:00 10.1.1.1 10.1.1.1 fxp0.0
none 00:0c:29:9a:e5:38 10.1.1.115 10.1.1.115 fxp0.0
none 00:05:85:9b:5d:f5 31.1.1.2 31.1.1.2 ge-7/0/3.493
none 00:14:f6:56:b8:7e 68.1.0.204 68.1.0.204 ge-7/1/0.0
none 02:01:00:00:00:05 128.0.0.5 128.0.0.5 em0.0
none 00:00:c0:10:01:02 192.16.1.2 192.16.1.2 ge-7/0/5.0
none Total entries: 7
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 59
How to trouble shoot routing and forwarding issues? install@FED1DSRJ01-LAB-re0> show arp MAC Address Address Name Interface
Flags 02:01:00:00:00:05 10.0.0.5 10.0.0.5 em0.0
none 00:04:80:9d:b5:00 10.1.1.1 10.1.1.1 fxp0.0
none 00:0c:29:9a:e5:38 10.1.1.115 10.1.1.115 fxp0.0
none 00:05:85:9b:5d:f5 31.1.1.2 31.1.1.2 ge-7/0/3.493
none 00:14:f6:56:b8:7e 68.1.0.204 68.1.0.204 ge-7/1/0.0
none 02:01:00:00:00:05 128.0.0.5 128.0.0.5 em0.0
none 00:00:c0:10:01:02 192.16.1.2 192.16.1.2 ge-7/0/5.0
none Total entries: 7
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 60
How to trouble shoot routing and forwarding issues? install@FED1DSRJ01-LAB-re0> show route protocol ospf
inet.0: 260 destinations, 387 routes (186 active, 0 holddown, 77 hidden) @ = Routing Use Only, # = Forwarding Use Only + = Active Route, - = Last Active, * = Both
0.0.0.0/0 *[OSPF/10] 09:25:03, metric 16777215 Discard 3.1.1.0/24 *[OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 10.1.0.0/16 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 10.1.1.0/24 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 10.1.200.0/28 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 10.99.0.0/16 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 10.99.99.0/24 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108 24.234.6.0/24 *[OSPF/10] 00:54:30, metric 182 > to 68.1.0.204 via ge-7/1/0.0 24.234.6.0/27 *[OSPF/10] 00:54:30, metric 166 > to 68.1.0.204 via ge-7/1/0.0 24.248.129.0/27 [OSPF/150] 09:23:28, metric 0, tag 0 > via so-0/1/0.108
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 61
How to trouble shoot routing and forwarding issues? FFPC7(FED1DSRJ01-LAB-re0 vty)# show route ip prefix
192.12.1.2 IPv4 Route Table 0, default.0, 0x0: Destination NH IP Addr Type NH ID
Interface --------------------------------- --------------- -------- ----- --------- 192.12.1.2 192.12.1.2 Unicast 716 ge-7/0/4.0 FFPC7(FED1DSRJ01-LAB-re0 vty)# show route ip lookup
192.12.1.2 Route Information (192.12.1.2): interface : ge-7/0/4.0 (87) Nexthop prefix : 192.12.1.2 Nexthop ID : 716 MTU : 1514 Class ID : 0
FFPC7(FED1DSRJ01-LAB-re0 vty)#
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 62
How to trouble shoot routing and forwarding issues? install@FED1DSRJ01-LAB-re0> show interfaces filters ge-7/0/4 Interface Admin Link Proto Input Filter Output Filter ge-7/0/4 up up ge-7/0/4.0 up up inet multiservice FFPC7(FED1DSRJ01-LAB-re0 vty)# show nhdb interface ge-7/0/4 ID Type Interface Next Hop Addr Protocol Encap MTU ----- -------- ------------- --------------- ---------- ------------ ---- 625 Bcast ge-7/0/4.0 - IPv4 Ethernet 0 626 Receive ge-7/0/4.0 192.12.1.0 IPv4 Ethernet 0 628 Resolve ge-7/0/4.0 - IPv4 Ethernet 0 716 Unicast ge-7/0/4.0 192.12.1.2 IPv4 Ethernet 1514
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 63
Lab stuff Agilent Router Tester. Remote access:Top 3 chassis: 172.19.59.28Bottom 3 chassis: 172.19.58.12User name: AdministratorPassword: n2xLaunch padCreate new sessionFor FE, need to config SFP
IXIA: VNC 172.19.58.2 (SV) 172.25.84.219(HD) ixia-2.jtac-west IXIA application server: 172.19.58.17
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 64
How to trouble shoot EOAM? http://www.juniper.net/techpubs/software/junos
/junos82/swconfig82-network-interfaces/html/interfaces-ethernet-config50.html#1272612
http://www.juniper.net/techpubs/software/junos/junos82/swconfig82-network-interfaces/html/interfaces-summary298.html#11618684
Known PRs: -PR81057
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 65
How to trouble shoot EOAM?protocols {
oam { ethernet {
link-fault-management { interfaces {
[xge/ge/fe]-<fpc>/<pic>/<port> { pdu-interval <value>; link-discovery <active|passive>; pdu-threshold <count>; remote-loopback;
} }
} } } }
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 66
How to trouble shoot EOAM?protocols {
oam { ethernet {
link-fault-management { interfaces {
[xge/ge/fe]-<fpc>/<pic>/<port> { pdu-interval <value>; link-discovery <active|passive>; pdu-threshold <count>; remote-loopback;
} }
} } } }
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 67
How to Manually mount a USB/CF storage? http://kb.juniper.net/KB8017
First upload the desired JUNOS image to the router via ftp to /var/tmp. Connect the USB mass storage device. Format the USB device by dropping to shell (start shell) then enter "dd
if=/dev/zero of=/dev/da0 bs=128k" (root access required). Note this step can take several minutes to complete with no output to the CLI window.
Label the device by entering "disklabel -r -w da0 auto". (!! if you move the USB/CF around, you need to execut this command before mounting)
Create the file system with "newfs -U /dev/da0c". Create a dir to be used as a mount point with "mkdir /var/tmp/usb". Mount the USB device using "mount /dev/da0c /var/tmp/usb". df -h can be used to verify the mount.
Copy the JUNOS install image to the USB device. cp /var/tmp/junos-jseries-8.0R2.8-domestic.tgz /var/tmp/usb Delete the original image to free up space on the CF. rm /var/tmp/junos-jseries-8.0R2.8-domestic.tgz Use the "request system software add /var/tmp/usb/junos-jseries-
8.0R2.8-domestic.tgz" command to install the new JUNOS version.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 68
How to do tcpdump at Junos? You have to login as root You have to know which incoming interface? Command:
root@bananas-re0% tcpdump -xvf -i so-1/1/0
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 69
Ethernet OAM Ethernet OAM types
In short, there are two types of Ethernet OAM:
1. Ethernet OAM as defined by 802.3ah
This is referred as LFM (Link Fault Management) and are identified by the ether-type 0x8809 (slow protocol type packets), sub-type 3.
2. Ethernet OAM as defined by IEEE 802.1ag
This is referred as CFM (Connectivity Fault Management) and can be by the ether-type 0x8902.
Ethernet OAM implementation in JunOS
Ethernet OAM is implemented using the RE user space daemons "lfmd" and "cfmd". Also, both "lfmd" and "cfmd" use the "ppmd" daemon on the PFE for some periodic packet processing.
There is a packet processing path in the RE kernel as well in addition to the daemons mentioned above.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 70
Ethernet OAM Ethernet OAM for regular Ethernet interfaces
Both 802.3ah (LFM) and 802.1ag (CFM) type Ethernet OAMs are supported in JunOS for the regular Ethernet interfaces with the following restrictions.
802.3ah (LFM) type OAM can be configured only on the Ethernet IFDs and NOT on the Ethernet IFLs. Also, these packets are always VLAN untagged.
However, 802.1ag (CFM) type OAM can be configured either on an Ethernet IFD or IFL. If this is configured on an IFD, the packets will be always VLAN untagged. If this is configured on an IFL, it will be either VLAN tagged or untagged based on the "vlan-tagging" keyword configuration on an Ethernet IFD.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 71
Ethernet OAM Link MonitoringLink monitoring in Ethernet OAM detects and indicates link faults under
a variety of conditions. Link monitoring uses the event notification OAMPDU and sends events to the remote OAMentity when there are problems detected on the link. The error events include the following:
• Error Symbol Period (error symbols per second)—The number of symbol errors that occurred during a specified period exceeded a threshold. These errors are coding symbol errors.
• Error Frame (error frames per second)—The number of frame errors detected during a specified period exceeded a threshold.
• Error Frame Period (error frames per n frames)—The number of frame errors within the last n frames has exceeded a threshold.• Error Frame Seconds Summary (error seconds per m seconds)—The number of error seconds (1-second intervals with at least one frame error) within the last m seconds has exceeded a threshold. Since IEEE 802.3ah OAM does not provide a guaranteed delivery of any OAM PDU, the eventnotification OAM PDU may be sent multiple times to reduce the probability of a lost notification. A sequence number is used to recognize duplicate events
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 72
Ethernet OAM Ethernet OAM for regular Ethernet interfaces
Both 802.3ah (LFM) and 802.1ag (CFM) type Ethernet OAMs are supported in JunOS for the regular Ethernet interfaces with the following restrictions.
802.3ah (LFM) type OAM can be configured only on the Ethernet IFDs and NOT on the Ethernet IFLs. Also, these packets are always VLAN untagged.
However, 802.1ag (CFM) type OAM can be configured either on an Ethernet IFD or IFL. If this is configured on an IFD, the packets will be always VLAN untagged. If this is configured on an IFL, it will be either VLAN tagged or untagged based on the "vlan-tagging" keyword configuration on an Ethernet IFD.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 73
Ethernet OAM one scenario (2008-0401-0623) Scenario: Two T640s with JUNOS 8.2SR are connected together through
an optical transport network (e.g., Fujitsu 7500/7600), using LAN-PHY on 10GE IQ2 PICs.
Question: If there is a link failure in the transport network and the 10GE links between the Fujitsu switches and the T640s stay up, will the Local T640 send out Ethernet 802.3ah OAMPDUs with the Flags for Critical Link Events(1) and the Link Event TLVs(2) to the Remote T640?
Answer: No. None of that will happen. What will happen is, the OAM Discovery INFO PDUs will timeout and both sides will detect that and mark a failure on their respective links. If only one direction of the link is down, one side will be in "Active Send Local" state and the other side will be in "Send Local Remote" state. There is no reason to send Link Event TLVs in the above situation as it's a link fault, not a framing error.
The reason we do not send Link-Fault or Dying Gasp is, by the time we detect a Rx fault, the ifd is marked down and the Tx is also brought down. The Critical Event is not defined in the 802.3ah for any specific purposes,and is implementation dependant. In Juniper implementation, we use Critical event to simulate RDI functionality. We only send Critical event in case we have a CCC-DOWN on the ifls on the interface marked by RPD and an action profile to send a critical event is defined.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 74
Ethernet OAM one scenario (2008-0401-0623) syslog { archive { files number; size size; (world-readable | no-world-readable); } console
{ facility severity; } file filename { facility severity; explicit-priority; match "regular-expression"; archive { files number; size size; (world-readable | no-world-readable); } } host (hostname | other-routing-engine | scc-master) { facility severity; explicit-priority; facility-override facility; log-prefix string; match "regular-expression"; } source-address source-address;time-format (year | millisecond | year millisecond); user (username | *) { facility severity; match "regular-expression"; }}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 75
CoS configuration (2008-0523-0448)http://www.juniper.net/techpubs/software/junos/junos90/swconfig-cos/frameset.htmlIn the following classifier example, packets with EXP bits 000 are assigned to the data-queue forwarding class with a low loss
priority, and packets with EXP bits 001 are assigned to the data-queue forwarding class with a high loss priority.
[edit class-of-service]
classifiers {exp exp_classifier {
forwarding-class data-queue {loss-priority low code-points 000;loss-priority high code-points 001;}
}
}
In the following drop-profile map example, the scheduler includes two drop-profile maps, which specify that packets are evaluated by the low-drop drop profile if they have a low loss priority and are from any protocol. Packets are evaluated by the high-drop drop profile if they have a high loss priority and are from any protocol.
[edit class-of-service]
schedulers {best-effort {
drop-profile-map loss-priority low protocol any drop-profile low-drop;drop-profile-map loss-priority high protocol any drop-profile high-drop;
}
}
In the following rewrite rule example, packets in the be forwarding class with low loss priority are assigned the EXP bits 000, and packets in the be forwarding class with high loss priority are assigned the EXP bits 001.
[edit class-of-service]
rewrite-rules {exp exp-rw {
forwarding-class be {loss-priority low code-point 000;loss-priority high code-point 001;
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 76
How to verify packages are corrupted? root@% mount /altroot root@% mount /altconfig root@% cd /altroot/packages/
root@% sha1 j*8.5R3.4 SHA1 (jbase-8.5R3.4) = 51a9f2cfe95a53d1dbda2daedd6b5dd6dd66213c SHA1 (jdocs-8.5R3.4) = c56296f2016d5ddbf8b22c00cb8c06dc5c664271 SHA1 (jkernel-8.5R3.4) = fedc82d6e8edb6b5ff972ac4c0f22885841ee48e SHA1 (jpfe-T-8.5R3.4) = f8ea2b28cf27a168a1023b0e544cdfb047ac2f0e ---> corrupted SHA1 (jpfe-common-8.5R3.4) = 0034ccbd5bd1b2bbd9b9ee41d3b42c50443e5562 --->
corrupted SHA1 (jroute-8.5R3.4) = 5c22ca387a78d4a3cb47af79ef6bdcfa0e0bc26f
root@% sha1 /packages/j*8.5R3.4 SHA1 (/packages/jbase-8.5R3.4) = 51a9f2cfe95a53d1dbda2daedd6b5dd6dd66213c SHA1 (/packages/jdocs-8.5R3.4) = c56296f2016d5ddbf8b22c00cb8c06dc5c664271 SHA1 (/packages/jkernel-8.5R3.4) = fedc82d6e8edb6b5ff972ac4c0f22885841ee48e SHA1 (/packages/jpfe-T-8.5R3.4) = f14de1eb8e537a35088864192d6838bb24804492 SHA1 (/packages/jpfe-common-8.5R3.4) =
270c4f4cc9c0afb6ba52c6916c2213eeba851ddc SHA1 (/packages/jroute-8.5R3.4) = 5c22ca387a78d4a3cb47af79ef6bdcfa0e0bc26f
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 77
Class-of-Service trouble shooting There is bug in Gimlet FPC where the PLP high defined at classifier will
*NOT* be copied to notification. Thus if egress FPC might have rewrite rule messed up.
1. Gimlet FPC to Gimlet FPC has no problem. 2. Gimble FPC to Stoli FPC has problem3. Gimlet FPC to Gimlet FPC with drop-profile has problem.
To work around this problem for scenario 2 & 3:lab@slayer-re1# set class-of-service copy-plp
Default forwarding class:Queue Forwarding-class0 best-effort1 Assured-forwarding2 expedited-forwarding3 network-control
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 78
Class-of-Service trouble shooting http://www.juniper.net/techpubs/software/junos/
junos90/swconfig-cos/swconfig-cos.pdfTable 43: Default MPLS EXP Rewrite Table(P230)------------------------------------------------Forwarding Class Loss Priority CoS Valuebest-effort(0) low 000best-effort high 001expedited-forwarding(1) low 010expedited-forwarding high 011assured-forwarding(2) low 100assured-forwarding high 101network-control(3) low 110network-control high 111
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 79
Class-of-Service trouble shooting http://www.juniper.net/techpubs/software/junos/
junos90/swconfig-cos/swconfig-cos.pdfTable 42: Default Packet Header Rewrite Mappings (p225)Map from Forwarding Class PLP Value Map to DSCP/DSCP IPv6/ EXP/IEEE/IP
expedited-forwarding low ef
expedited-forwarding high ef
assured-forwarding low af11
assured-forwarding high af12 (DSCP/DSCP IPv6/EXP)
best-effort low be
best-effort high be
network-control low nc1/cs6
network-control high nc2/cs7
The mapping of alias to EXP code point is at next slide. Same thing to look up alias to DSCP code point.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 80
Class-of-Service trouble shootinglab@slayer-re1> show class-of-service code-point-aliases exp Code point type: exp Alias Bit pattern af11 100 af12 101 be 000 be1 001 cs6 110 cs7 111 ef 010 ef1 011 nc1 110
nc2 111
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 81
PLP Treatment on LMNR Platforms Overview
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 82
Problem
Customer Cox was seeing an increase of Non-Real-Time class traffic in the network when replacing IQ2 10GE PICs by 10GE XENPAK (non-IQ2) PICs.
Hard to isolate as there was a mix of traffic from different sources.
Initially though the problem was due to missclasification.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 83
Topology
IP unlabeled Traffic
IP unlabeled Traffic
LSP
xe-0/1/0
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 84
Configuration: Forwarding Classes
> ...service forwarding-classesqueue 0 BEST-EFFORT;queue 1 NON-REAL-TIME;queue 2 INTERACTIVE;queue 3 REAL-TIME;queue 4 VIDEO;queue 5 VOICE;queue 6 NETWORK-CONTROL;
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 85
Configuration: IP-Prec. Classifierforwarding-class BEST-EFFORT { loss-priority high code-points BEST-EFFORT-be;}forwarding-class NON-REAL-TIME { loss-priority high code-points NON-REAL-TIME-af11;}forwarding-class INTERACTIVE { loss-priority low code-points INTERACTIVE-af21;}forwarding-class REAL-TIME { loss-priority low code-points REAL-TIME-af31;}forwarding-class VIDEO { loss-priority low code-points VIDEO-af41;}forwarding-class VOICE { loss-priority low code-points VOICE-ef;}forwarding-class NETWORK-CONTROL { loss-priority low code-points NETWORK-CONTROL-nc1;}
inet-precedence {
BEST-EFFORT-be 000;
NON-REAL-TIME-af11 001;
INTERACTIVE-af21 010;
REAL-TIME-af31 011;
VIDEO-af41 100;
VOICE-ef 101;
NETWORK-CONTROL-nc1 110;
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 86
Configuration: EXP Classifierforwarding-class BEST-EFFORT { loss-priority high code-points BEST-EFFORT-be;}forwarding-class NON-REAL-TIME { loss-priority high code-points NON-REAL-TIME-af11;}forwarding-class INTERACTIVE { loss-priority low code-points INTERACTIVE-af21;}forwarding-class REAL-TIME { loss-priority low code-points REAL-TIME-af31;}forwarding-class VIDEO { loss-priority low code-points VIDEO-af41;}forwarding-class VOICE { loss-priority low code-points VOICE-ef;}forwarding-class NETWORK-CONTROL { loss-priority low code-points NETWORK-CONTROL-nc1;}
BEST-EFFORT-be 000;
NON-REAL-TIME-af11 001;
INTERACTIVE-af21 010;
REAL-TIME-af31 011;
VIDEO-af41 100;
VOICE-ef 101;
NETWORK-CONTROL-nc1 110;
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 87
Configuration: Rewrite Rules, EXPexp WRITE-EXP {
forwarding-class BEST-EFFORT {
loss-priority low code-point BEST-EFFORT-be;
loss-priority high code-point BEST-EFFORT-be;
}
forwarding-class NON-REAL-TIME {
loss-priority low code-point NON-REAL-TIME-af11;
loss-priority high code-point NON-REAL-TIME-af11;
}
forwarding-class INTERACTIVE {
loss-priority low code-point INTERACTIVE-af21;
loss-priority high code-point INTERACTIVE-af21;
}
forwarding-class REAL-TIME {
loss-priority low code-point REAL-TIME-af31;
loss-priority high code-point REAL-TIME-af31;
}
forwarding-class VIDEO {
loss-priority low code-point VIDEO-af41;
loss-priority high code-point VIDEO-af41;
}
forwarding-class VOICE {
loss-priority low code-point VOICE-ef;
loss-priority high code-point VOICE-ef;
}
forwarding-class NETWORK-CONTROL {
loss-priority low code-point NETWORK-CONTROL-nc1;
loss-priority high code-point NETWORK-CONTROL-nc1;
}
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 88
PLP handling
BA
Cla
s si fi
e r
Lin
MF
Cla
ssifi
e r
Jtree Lookup
Sim
ple
Fi lt
e r
Rew
r ite
Rul
e
Lout
PIC
IQ2 PICN
on-I
Q2
PI C
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 89
Which PLP ?
The L to N notification cell contains two bits (three with tri-color marking) of interest:
The pseudo-plp bit: This is bit 2 of the QoS field (6-bits), and it’s used by the Lin BA Classifier and Rewrite rules
The real plp bit: this is a separate bit, see the Lin functional description for location.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 90
PLP On LMNR
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 91
Example: IP packet, precedence 000, non-IQ2 PIC Let’s say we receive a packet with IP-Prec bits 000.
Let’s say we have a BA Classifier that classifies IP-Prec: 000 as Best-Effort (queue 0) and plp=high:
# show class-of-service code-point-aliases inet-precedenceBEST-EFFORT-be 000; NON-REAL-TIME-af11 001;INTERACTIVE-af21 010;REAL-TIME-af31 011;VIDEO-af41 100;VOICE-ef 101;NETWORK-CONTROL-nc1 110;
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 92
Contd…# show class-of-service classifiers inet-precedence
CLASSIFY-IPPforwarding-class BEST-EFFORT { loss-priority high code-points 000;}# show class-of-service forwarding-classesqueue 0 BEST-EFFORT; queue 1 NON-REAL-TIME;queue 2 INTERACTIVE;queue 3 REAL-TIME;queue 4 VIDEO;queue 5 VOICE;queue 6 NETWORK-CONTROL;
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 93
Ctd…
Because this packet’s real-plp bit will remain 0, RED will treat it as such. If we have the following rewrite rule:
apena@austinp-re0# show class-of-service rewrite-rules
exp WRITE-EXP { forwarding-class BEST-EFFORT { loss-priority low code-point 000; loss-priority high code-point 000; <<<< }
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 94
Will this work ?
The answer is:• It depends on the incoming PIC.• By default we OR the LSB of EXP and DSCP with the
real PLP (see flow chart):
• EXP 000 ORed with plp=1 gives EXP=001• This produces incorrect classification at next hop router• With IQ2 PIC, Lin can write proper real PLP thanks to cookie.• Without IQ2, Lin can’t write real plp, just pseudo plp.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 95
Workaround:
Use compatible markings Enable copy-plp hidden knob. Enable tri-color marking
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 96
Multicast trouble shootinglab@ 320_1> show pim rps extensive
Instance: PIM.master
Address family INET
RP: 198.140.33.2 Learned from 198.140.33.7 via: auto-rp
Time Active: 17w5d 05:03:53
Holdtime: 150 with 139 remaining
Device Index: 134
Subunit: 32780
Interface: pe-2/0/0.32780
Group Ranges:
224.0.2.64/32, 139s remaining
224.0.2.65/32, 139s remaining
224.0.2.66/32, 139s remaining
224.0.2.67/32, 139s remaining
Active groups using RP:
233.43.202.9
233.43.202.8
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 97
IPSec configuration and troubleshooting This is a wiki for a very bad Google
IPSeT defrag case.
http://confluence.jnpr.net/confluence/display/IPGE/Google+2009-0106-+IPSec+Fragmentation+Issue+-+PR+414885
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 98
IPSec configuration and troubleshootinglab@kings-re0# show services
service-set ny2ny02jt-payload {
max-flows 2m;
next-hop-service {
inside-service-interface sp-0/0/0.1;
outside-service-interface sp-0/0/0.2;
}
ipsec-vpn-options {
local-gateway 200.1.1.2;
}
ipsec-vpn-rules ny2ny02jt-payload;
}
ipsec-vpn {
rule ny2ny02jt-payload {
term 1 {
then {
remote-gateway 200.1.1.1;
dynamic {
ike-policy ny2ny02jt-payload;
ipsec-policy stream;
}
tunnel-mtu 9188;
anti-replay-window-size 1024;
}
}
match-direction input;
}
ipsec {
proposal brook {
protocol esp;
authentication-algorithm hmac-md5-96;
encryption-algorithm 3des-cbc;
}
policy stream {
proposals brook;
}
}
ike {
proposal rivlet {
authentication-method pre-shared-keys;
dh-group group1;
authentication-algorithm md5;
encryption-algorithm 3des-cbc;
}
policy ny2ny02jt-payload {
mode main;
proposals rivlet;
pre-shared-key ascii-text "$9$O4v9BEyleWXxd"; ## SECRET-DATA
}
}
establish-tunnels immediately;
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 99
IPSec configuration and troubleshootingOn T640 or other platforms where you
have service PIC, you need to configure
the SP interfaces.
lab@kings-re0# show interfaces sp-0/0/0
description ipsec-vpn;
mtu 9192;
unit 1 {
description ipsec-vpn-inside;
family inet;
service-domain inside;
}
unit 2 {
description ipsec-vpn-outside;
family inet;
service-domain outside;
}
Direct traffic to the IPSec tunnel.1) Static route
lab@kings-re0# show routing-options
static {
route 172.0.0.0/8 {
next-hop 172.25.44.1;
retain;
no-readvertise;
}
route 0.0.0.0/0 {
next-hop sp-0/0/0.1;
retain;
}
}
2) IGP
3) BGP
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 100
IPSec configuration and troubleshootinglab@kings-re0# run ping 111.0.0.1
PING 111.0.0.1 (111.0.0.1): 56 data bytes
64 bytes from 111.0.0.1: icmp_seq=0 ttl=64 time=1.335 ms
64 bytes from 111.0.0.1: icmp_seq=1 ttl=64 time=1.026 ms
64 bytes from 111.0.0.1: icmp_seq=2 ttl=64 time=1.050 ms
64 bytes from 111.0.0.1: icmp_seq=3 ttl=64 time=1.065 ms
64 bytes from 111.0.0.1: icmp_seq=4 ttl=64 time=1.032 ms
64 bytes from 111.0.0.1: icmp_seq=5 ttl=64 time=0.869 ms
64 bytes from 111.0.0.1: icmp_seq=6 ttl=64 time=1.078 ms
64 bytes from 111.0.0.1: icmp_seq=7 ttl=64 time=0.905 ms
64 bytes from 111.0.0.1: icmp_seq=8 ttl=64 time=1.073 ms
64 bytes from 111.0.0.1: icmp_seq=9 ttl=64 time=1.084 ms
64 bytes from 111.0.0.1: icmp_seq=10 ttl=64 time=0.885 ms
64 bytes from 111.0.0.1: icmp_seq=11 ttl=64 time=1.095 ms
64 bytes from 111.0.0.1: icmp_seq=12 ttl=64 time=0.948 ms
64 bytes from 111.0.0.1: icmp_seq=13 ttl=64 time=0.912 ms
lab@jazz-re0> monitor traffic interface sp-0/0/0.1 verbose output suppressed, use <detail> or <extensive> for full protocol decode
Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay.
Address resolution timeout is 4s.
Listening on sp-0/0/0.1, capture size 96 bytes
Reverse lookup for 111.0.0.1 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use <no-resolve> to avoid reverse lookups on IP addresses.
19:03:10.506267 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 6, length 64
19:03:10.506285 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 6, length 64
19:03:11.507050 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 7, length 64
19:03:11.507061 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 7, length 64
19:03:12.507977 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 8, length 64
19:03:12.507988 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 8, length 64
19:03:13.508794 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 9, length 64
19:03:13.508802 Out SERVICES service id 64 flags 0x82 service set id 1 iif 78 IP 111.0.0.1 > 101.1.1.1: ICMP echo reply, id 51991, seq 9, length 64
19:03:14.509561 In IP 101.1.1.1 > 111.0.0.1: ICMP echo request, id 51991, seq 10, length 64
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 101
IPSec configuration and troubleshootinglab@jazz-re0# run show log kmd
Jul 17 18:32:20 jazz-re0 clear-log[8331]: logfile cleared
Jul 17 18:33:26 Initialising the KMD ipsec-interface-id pool
Jul 17 18:33:26 Deleted SA pair with index=0 tunnel index=1 to kernel
Jul 17 18:33:26 Initializing certificate manager
Jul 17 18:33:26 Added SA pair with index=0 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel
Jul 17 18:34:06 Added SA pair with index=1 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel
Jul 17 18:34:11 Added SA pair with index=2 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel
Jul 17 18:57:25 Initialising the KMD ipsec-interface-id pool
Jul 17 18:57:38 Initialising the KMD ipsec-interface-id pool
Jul 17 18:58:53 Initialising the KMD ipsec-interface-id pool
Jul 17 19:31:56 Deleted SA pair with index=1 tunnel index=1 to kernel
Jul 17 19:31:56 Added SA pair with index=3 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel
Jul 17 19:34:11 Deleted SA pair with index=2 tunnel index=1 to kernel
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 102
IPSec configuration and troubleshootinglab@jazz-re0# run show log kmd
Jul 17 18:32:20 jazz-re0 clear-log[8331]: logfile cleared
Jul 17 18:33:26 Initialising the KMD ipsec-interface-id pool
Jul 17 18:33:26 Deleted SA pair with index=0 tunnel index=1 to kernel
Jul 17 18:33:26 Initializing certificate manager
Jul 17 18:33:26 Added SA pair with index=0 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel
Jul 17 18:34:06 Added SA pair with index=1 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel
Jul 17 18:34:11 Added SA pair with index=2 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel
Jul 17 18:57:25 Initialising the KMD ipsec-interface-id pool
Jul 17 18:57:38 Initialising the KMD ipsec-interface-id pool
Jul 17 18:58:53 Initialising the KMD ipsec-interface-id pool
Jul 17 19:31:56 Deleted SA pair with index=1 tunnel index=1 to kernel
Jul 17 19:31:56 Added SA pair with index=3 tunnel index=1 PIC index=0 Interface name: sp-0/0/0 Length:1392 to kernel
Jul 17 19:34:11 Deleted SA pair with index=2 tunnel index=1 to kernel
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 103
How to compare rollback?rprivette@CHRL-HAGG-03> show system rollback compare 0 2
[edit interfaces ge-3/3/1 unit 3478]
- description "16/VLXX/010009/TWCS - FREEMAN WHITE # 255277 [ENLAN]";
+ description "16/KDFN/010010/TWCS - Freeman White # FW115671";
- encapsulation vlan-vpls;
+ encapsulation vlan-ccc;
+ family ccc {
+ policer {
+ input LIMIT_10M;
+ output LIMIT_10M;
+ }
+ }
- family vpls {
- policer {
- input LIMIT_10M;
- output LIMIT_10M;
- }
- }
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 104
MX VLAN configuration – what are the new stuff?
STP’s: original 802.1D1) MSTP: based on 802.1s2) RSTP: based on 802.1w3) MISTP: Cisco Proprietary
Multiple Instance STP4) PVST+: Per-VLAN spanning-tree
plus5) Rapid PVST+
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 105
MX VLAN Trunking configuration – General guideline
Generally, there are four things that you must configure in an L2 environment:
Interfaces and virtual LAN (VLAN) tags—L2 interfaces are usually various type of Ethernet links with VLAN tags used to connect to customer devices or other bridges or routers.
Bridge domains and virtual switches—Bridge domains limit the scope of media access control (MAC) learning (and thereby the size of the MAC table) and also determine where the device should propagate frames sent to broadcast, unknown unicast, and multicast (BUM) MAC addresses. Virtual switches allow for the configuration of multiple, independent bridge domains.
Spanning Tree Protocols (xSTP, where the “x” represents the STP type)—Bridges function by associating a MAC address with an interface, similar to the way a router associates an IP network address with a next-hop interface. Just as routing protocols use packets to detect and prevent routing loops, bridges use xSTP frames to detect and prevent bridging loops. (L2 loops are more devastating to a network because of the broadcast nature of Ethernet LANs.)
Integrated bridging and routing (IRB)—Support for both Layer 2 bridging and Layer 3 routing on the same interface. Frames are bridged if they are not sent to the router's MAC address. Frames sent to the router's MAC address are routed to other interfaces configured for Layer 3 routing.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 106
MX VLAN Trunking configuration – vlan tagginginterfaces ge-2/2/6 {
encapsulation flexible-ethernet-services;vlan-tagging; # Customer interface uses singly-tagged
framesunit 200 {
encapsulation vlan-bridge;vlan-id 200;
}}interfaces ae1 {
encapsulation extended-vlan-bridge;vlan-tagging;unit 100 {
vlan-id 100;}unit 200 {
vlan-id 200;}
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 107
MX VLAN Trunking configuration – bridge domainConfigure the virtual switches and bridge domains on all three routers.
There is always a default virtual switch in the router for L2 functions; however, if there is only one L2 network, then the virtual switch instance type is not needed.
Configure a bridge domain on Router 1:[edit]bridge-domains {
vlan100 {domain-type bridge;vlan-id 100;interface ge-2/2/1.100;interface ae1.100;interface ae2.100;
}vlan200 {
domain-type bridge;vlan-id 200;interface ge-2/2/1.200;interface ge-2/2/6.200;interface ae1.200;interface ae2.200;
}
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 108
MX VLAN Trunking configuration – MSTP-1Key words:
MSTI: Multiple Spanning Tree Instances CIST: Common and Internal Spanning Tree MSTP: Multiple Spanning Tree Protocol
Configuration name: The names must match to be in the same region
Revision Level: must be the same across the same region.
VLAN-to-MSTI mapping: vlans mapped to this MSTP instance.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 109
MX VLAN Truncking configuration – MSTP-2 protocols {
mstp {configuration-name mstp-for-R1-2-3; # The names
must match to be in the same regionrevision-level 3; # The revision levels must matchbridge-priority 0; # This bridge acts as root bridge for
VLAN 100 and 200interface ae1;interface ae2;msti 1 {
vlan100; # This VLAN corresponds to MSTP instance 1
}msti 2 {
vlan200; # This VLAN corresponds to MSTP instance 2
}}
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 110
MX VLAN Truncking configuration – IRB-1 You configure IRB in two steps: (1) Configure the IRB interface using the irb
statement. (2) Reference the IRB interface at the bridge
domain level of the configuration. IRB supports Layer 2 bridging and
Layer 3 routing on the same interface. If the MAC address on the arriving frame is the same as that of the IRB interface, then the packet inside the frame is routed. Otherwise, the MAC address is learned or looked up in the MAC address database.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 111
MX VLAN configuration – IRB-2edit interfaces]
xe-2/1/0 {unit 0 {
family inet {
address 10.0.10.2/24; # Routing interface
}
}
}
irb {unit 0 {
family inet {
address 10.0.1.2/24 {
vrrp-group 1 {
virtual-address 10.0.1.51;
priority 254;
}
}
}
}unit 1 {
family inet {
address 10.0.2.2/24 {
vrrp-group 2 {
virtual-address 10.0.2.51;
priority 100;
}
}
}
}
}
bridge-domains {
vlan-100 {
domain-type bridge;
vlan-id 100;
interface ge-2/2/2.100;
interface ae1.100;
interface ae3.100
routing-interface irb.0;
}
vlan-200 {
domain-type bridge;
vlan-id 200;
interface ge-3/3/3.200;
interface ae1.200;
interface ae3.200
routing-interface irb.1;
}
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 112
MX VLAN configuration- host interfaceNew CLI introduced at in the fix of PR 299511
lab@Atlas_re0# show interfaces ge-5/0/4 encapsulation ethernet-bridge;unit 0 { family bridge;}[edit]lab@Atlas_re0# show interfaces ge-0/0/4 encapsulation ethernet-bridge;unit 0 { family bridge;}Bridge-domain{vlan333 { domain-type bridge; vlan-id 333; interface ge-5/0/4.0; interface ge-0/0/4.0; }}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 113
Firewall Troubleshootinglab@slayer-re1> show firewall filter log-
as0.0-i
Filter: log-as0.0-i Counters:Name Bytes
Packetsrsvp-as0.0-i 0
0ospf-as0.0-i 0
0bgp-as0.0-i 0
0all-as0.0-i 149963421000
99975614
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 114
Firewall Troubleshooting -templab@slayer-re1> show firewall filter log-
as0.0-i
Filter: log-as0.0-i Counters:Name Bytes
Packetsrsvp-as0.0-i 0
0ospf-as0.0-i 0
0bgp-as0.0-i 0
0all-as0.0-i 149963421000
99975614
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 115
Firewall Troubleshooting -templab@slayer-re1> show firewall filter log-
as0.0-i
Filter: log-as0.0-i Counters:Name Bytes
Packetsrsvp-as0.0-i 0
0ospf-as0.0-i 0
0bgp-as0.0-i 0
0all-as0.0-i 149963421000
99975614
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 116
Firewall Troubleshooting -templab@slayer-re1> show firewall filter log-
as0.0-i
Filter: log-as0.0-i Counters:Name Bytes
Packetsrsvp-as0.0-i 0
0ospf-as0.0-i 0
0bgp-as0.0-i 0
0all-as0.0-i 149963421000
99975614
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 117
Firewall Troubleshooting -templab@slayer-re1> show firewall filter log-
as0.0-i
Filter: log-as0.0-i Counters:Name Bytes
Packetsrsvp-as0.0-i 0
0ospf-as0.0-i 0
0bgp-as0.0-i 0
0all-as0.0-i 149963421000
99975614
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 118
MX-960 pegasus DPC auto-negohttps://tools.online.juniper.net/cm/case_note_detail.jsp?
cid=Up9%2FoWPEU57FR9OFIsO0vQ%3D%3D&type=WQDDoTj%2Bp28%3D&num=fF6aYIYjhYCr4QBubu3%2BXg%3D%3D&isInternal=false
http://cvs.juniper.net/cgi-bin/viewcvs.cgi/sw-projects/platform/atlas/pegasus/pegasus_unit_test_plan.txt?rev=1.3&view=markup
7. Speed/Duplex selection from RE CLI - 100m/full-duplex Goal: Test configuration of speed, link-mode from RE CLI Test Steps: 1. Issue the below command on RE CLI -> set interfaces ge-x/y/z
speed 100m link-mode full-duplex -> commit 2. Issue the below command on DPC console -> "show bcm5466 registers y
z" 3. Compare the values from "MII Control Register" with Broadcom 5466 data
sheet. 4. Issue the below command on DPC console -> "show npez y rgmii z" Success Criteria: Description in the Data sheet should match with the
values read. From output of step 4 verify rgmii rate Result: PASS Output: Step 2: MII Control Register (0x00) : 0x3100 Step 4: The rate of the RGMII port is 100Mb
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 119
How to trouble shoot RSVP/LSP issues? RSVP related operational mode
commands: - clear rsvp session - show rsvp session - clear mpls lsp - show mpls lsp - show rsvp interface - show ted database extensive -
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 120
How to trouble shoot RSVP/LSP issues?
[email protected]> show ted database 168.215.52.177 extensive
TED database: 0 ISIS nodes 671 INET nodes
NodeID: 168.215.52.177
Type: Rtr, Age: 271072 secs, LinkIn: 2, LinkOut: 2
Protocol: OSPF(0.0.0.0)
To: 66.192.245.68-1, Local: 66.192.245.78, Remote: 0.0.0.0
Local interface index: 0, Remote interface index: 0
Color: 0 <none>
Metric: 100
Static BW: 1000Mbps
Reservable BW: 700Mbps
Available BW [priority] bps:
[0] 699.21Mbps [1] 699.21Mbps [2] 699.21Mbps [3] 699.21Mbps
[4] 699.21Mbps [5] 699.21Mbps [6] 699.21Mbps [7] 699.21Mbps
Interface Switching Capability Descriptor(1):
Switching type: Packet
Encoding type: Packet
Maximum LSP BW [priority] bps:
[0] 699.21Mbps [1] 699.21Mbps [2] 699.21Mbps [3] 699.21Mbps
[4] 699.21Mbps [5] 699.21Mbps [6] 699.21Mbps [7] 699.21Mbps
To: 66.192.245.116-1, Local: 66.192.245.126, Remote: 0.0.0.0
Local interface index: 0, Remote interface index: 0
Color: 0 <none>
Metric: 100
Static BW: 1000Mbps
Reservable BW: 700Mbps
Available BW [priority] bps:
[0] 699.07Mbps [1] 699.07Mbps [2] 699.07Mbps [3] 699.07Mbps
[4] 699.07Mbps [5] 699.07Mbps [6] 699.07Mbps [7] 699.07Mbps
Interface Switching Capability Descriptor(1):
Switching type: Packet
Encoding type: Packet
Maximum LSP BW [priority] bps:
[0] 699.07Mbps [1] 699.07Mbps [2] 699.07Mbps [3] 699.07Mbps
[4] 699.07Mbps [5] 699.07Mbps [6] 699.07Mbps [7] 699.07Mbps
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 121
How to trouble shoot commit problem?
•Commit synch | display details
•Show log ksyncd, same as the /var/log/ksyncd
•Roll back configuration of backup RE and sych up from RE0
•Copy configuration from master RE to backup RE:
Configure files are saved under /config. The running config is juniper.conf.gz.(execute this command from master RE, be careful of the permission on backup RE’s directory)
rcp –T juniper.config.gz re1:/var/tmp
will copy the file to backup RE1’s /var/tmp directory
# commit check
[email protected]> show system commit
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 122
Trouble shoot PFE CPU high start shell vty fpc6 sh nvram sh syslog messages FFPC4(cer-core-01 vty)# show pfe
statistics traffic FFPC4(cer-core-01 vty)# show pfe
statistics notification FFPC4(cer-core-01 vty)# show icmp
statistics Show chassis fpc (to find out fpc cpu
utilization)
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 123
Trouble shoot PFE CPU high start shell vty fpc6 sh nvram sh syslog messages FFPC4(cer-core-01 vty)# show pfe
statistics traffic FFPC4(cer-core-01 vty)# show pfe
statistics notification FFPC4(cer-core-01 vty)# show icmp
statistics Show chassis fpc (to find out fpc cpu
utilization)
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 124
6PE trouble shootingPE configurationlab@Magenta# show protocols
rsvp {
interface as0.0;
}
mpls {
ipv6-tunneling;
label-switched-path to_PE2 {
to 4.4.4.4;
}
interface as0.0;
}
bgp {
group purple {
type internal;
local-address 2.2.2.2;
family inet6 {
labeled-unicast {
explicit-null;
}
}
peer-as 100;
neighbor 4.4.4.4;
}
group to_CE2 {
type external;
local-address 8002::1;
family inet6 {
unicast;
}
peer-as 300;
neighbor 8002::2;
}
}
isis {
interface as0.0 {
level 2 metric 10;
}
interface lo0.0;
}
fe-0/1/0 {
unit 0 {
family inet {
address 99.1.1.1/24;
}
}
}
gr-1/2/0 { // GSR tunnel
unit 100 {
tunnel {
source 99.1.1.1;
destination 99.1.1.2;
}
family inet6 {
address 8002::1/126;
}
}
}
lo0 {
unit 0 {
family inet {
address 2.2.2.2/32;
}
family iso {
address 49.0001.0005.0005.0005.00;
}
}
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 125
6PE trouble shootingCE configurationinterfaces {
fe-0/1/0 {
unit 0 {
family inet {
address 99.1.1.2/24;
}
}
}
gr-1/2/0 {
unit 100 {
tunnel {
source 99.1.1.2;
destination 99.1.1.1;
}
family inet6 {
address 8002::2/126;
}
}
}
lo0 {
unit 0 {
family inet {
address 127.0.0.1/32;
}
family inet6 {
address 9001::5/128;
}
}
}
}
routing-options {
static {
route 172.0.0.0/8 {
next-hop 172.19.58.1;
no-readvertise;
}
}
autonomous-system 300;
}
protocols {
bgp {
group to_PE2 {
type external;
local-address 8002::2;
family inet6 {
unicast;
}
export policy1;
peer-as 100;
neighbor 8002::1;
}
}
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 126
MPLS Auto-bandwidth Auto-bandwidth configuration
mpls {
apply-groups [ lspHigh-common lspStnd-common lsp-optimize-timer ];
path-mtu {
rsvp mtu-signaling;
}
statistics {
file mpls.stat size 300k files 20 world-readable;
interval 300;
auto-bandwidth;
display-id;
}
traceoptions {
file mpls.log size 10m files 21 world-readable;
flag error;
flag state;
flag cspf;
flag connection;
flag graceful-restart;
}
}
label-switched-path lspStndT6toT1 {
to 166.34.95.71;
optimize-timer 60;
node-link-protection;
adaptive;
auto-bandwidth {
adjust-interval 300;
adjust-threshold 10;
minimum-bandwidth 100k;
maximum-bandwidth 10g;
adjust-threshold-overflow-limit 5;
}
primary use-ge-620;
}
path use-ge-620 {
192.100.36.37;
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 127
MPLS Auto-bandwith trouble shooting
lab@Magenta> file show /var/log/mpls.stat
Oct 30 15:41:21 trace_on: Tracing to "/var/log/mpls.stat" started
to_PE2 132491 pkt 139233752 Byte
Oct 30 15:41:21 2008 UTC Total 2 sessions: 1 success, 0 fail, 1 ignored
Oct 30 15:43:09 trace_on: Tracing to "/var/log/mpls.stat" started
to_PE2 132491 pkt 139233752 Byte 0 pps 0 Bps
auto-bw 0 pkt 0 Byte
Oct 30 15:43:09 2008 UTC Total 3 sessions: 2 success, 0 fail, 1 ignored
Oct 30 15:44:19 trace_on: Tracing to "/var/log/mpls.stat" started
auto-bw 0 pkt 0 Byte 0 pps 0 Bps Util 0.00%
lab@Magenta> file show /var/log/mpls.log
Oct 30 15:48:20 trace_on: Tracing to "/var/log/mpls.log" started
Oct 30 16:03:09.172425 RPD_MPLS_PATH_BANDWIDTH_CHANGE: MPLS path (lsp auto-bw) bandwidth changed, path bandwidth 4140760 bps
Oct 30 16:03:10.173337 RPD_MPLS_LSP_BANDWIDTH_CHANGE: MPLS LSP auto-bw bandwidth changed, lsp bandwidth 4140760 bps
Oct 30 16:08:09.173234 RPD_MPLS_PATH_BANDWIDTH_CHANGE: MPLS path (lsp auto-bw) bandwidth changed, path bandwidth 1000 bps
Oct 30 16:08:10.174771 RPD_MPLS_LSP_BANDWIDTH_CHANGE: MPLS LSP auto-bw bandwidth changed, lsp bandw
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 128
MPLS Auto-bandwith trouble shooting
edit protocols mpls statistics]
lab@Magenta# run show mpls lsp extensive
Ingress LSP: 1 sessions
4.4.4.4
From: 2.2.2.2, State: Up, ActiveRoute: 0, LSPname: auto-bw
Description: test2
ActivePath: (primary)
Node/Link protection desired
LoadBalance: Random
Autobandwidth
MinBW: 1000bps MaxBW: 10Gbps
AdjustTimer: 300 secs AdjustThreshold: 10%
Max AvgBW util: 0bps, Bandwidth Adjustment in 5 second(s).
Overflow limit: 5, Overflow sample count: 0
Encoding type: Packet, Switching type: Packet, GPID: IPv4
*Primary State: Up
Priorities: 7 0
Bandwidth: 1.824kbps
OptimizeTimer: 60
SmartOptimizeTimer: 180
Reoptimization in 18 second(s).
Computed ERO (S [L] denotes strict [loose] hops): (CSPF metric: 10)
5.5.5.1 S
Received RRO (ProtectionFlag 1=Available 2=InUse 4=B/W 8=Node 10=SoftPreempt):
5.5.5.1(Label=3)
90 Oct 30 17:27:24.553 CSPF: computation result ignored[5 times]
89 Oct 30 17:23:09.175 Record Route: 5.5.5.1(Label=3)
88 Oct 30 17:23:09.175 Up
87 Oct 30 17:23:09.175 Automatic Autobw adjustment succeeded
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 129
NAT stuff
To enable random port allocation, user has to configure
"set services nat pool <pool-name> port automatic random-allocation" or
"set services nat pool <pool-name> port range low <low-port-num> high <high-port-num> random-allocation".
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 130
How to look up RE CPU and Memory?
lab@jazz-re0> show chassis routing-engine
Routing Engine status:
Slot 0:
Current state Master
Election priority Master (default)
Temperature 41 degrees C / 105 degrees F
CPU temperature 43 degrees C / 109 degrees F
DRAM 3584 MB
Memory utilization 13 percent
CPU utilization:
User 0 percent
Background 0 percent
Kernel 2 percent
Interrupt 0 percent
Idle 97 percent
Model RE-A-2000
Serial ID 9009002764
Start time 2008-11-18 08:15:10 PST
Uptime 8 hours, 54 minutes, 29 seconds
Load averages: 1 minute 5 minute 15 minute
0.06 0.10 0.05
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 131
Translate Cisco ATM to Juniper ATM
interface ATM1/0/0
description ### Google DEDICADA###
bandwidth 155000
no ip address
no ip directed-broadcast
no ip proxy-arp
no ip mroute-cache
load-interval 30
atm sonet stm-1
atm uni-version 3.1
no atm ilmi-keepalive
no atm enable-ilmi-trap
no snmp trap link-status
!
interface ATM1/0/0.1 point-to-point
description Link Google_Akwan (50Mbps)*5531004003 bandwidth 50000 ip address 200.162.89.161 255.255.255.252 no ip redirects no ip unreachables no ip directed-broadcast no ip proxy-arp no atm enable-ilmi-trap snmp trap link-status pvc 5531004003 2/901
vbr-nrt 55209 55209 1
no ilmi manage
oam-pvc manage
oam retry 10 5 1
encapsulation aal5snap
!
!----------------------------
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 132
Translate Cisco ATM to Juniper ATM
chassis {
fpc 0 {
pic 3 {
framing sdh;
}
}
}
interfaces {
at-0/3/0 {
atm-options {
pic-type atm2;
vpi 2;
}
unit 1 {
encapsulation atm-snap;
point-to-point;
no-traps;
vci 2.901;
shaping {
vbr peak 55209000 sustained 55209000 burst 1;
}
oam-period 10;
oam-liveness {
up-count 10;
down-count 5;
}
family inet {
address 200.162.89.162/30;
}
}
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 133
Translate Cisco ATM to Juniper ATM
http://www.juniper.net/techpubs/software/junos/junos90/swconfig-network-interfaces/frameset.html
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 134
T1 / T3 trouble shooting
1. Loopback testing
http://www.juniper.net/techpubs/software/erx/erx41x/swconfig-physical-link/html/t1-e1-ji-config8.html
• Either Local loopback or remote loopback can be configured at any given time.
• For local loopback, best use an external loopback plug because it can also tests the PICs transmit and receive circuitry.
• SONET, T1/DS1 type P-T-P interfaces support remote loopback
• Configuring remote loopback only results in a line loop on local router.
• Configuration:
sonet-options {
loopback local/remote;
}
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 135
A good status write up
[Action] Spoke with Bob Walsh and Mark Rippe. [Issue summarized] The issue was they were seeing physical layer T1 issues as well as intermittent ping loss. [Issue details] For T1 errors they were seeing BEE and LOF errors. When looking at the ping loss issue, [Start of cause analysis – top layer of root cause] I determined that the reason for network outage was due to PPP going down and renegotiating over and over again. [ real root cause] This was due to the T1 error condition. [ here is why the real root cause is] Setting t1-0/0/3 hold-time up 0 down 100 stabilized the PPP connection. But that does not resolve the underlying issue with the T1 errors. BEE and LOF indicates a problem with upstream provider equipment. BEE is typically triggered when upstream switch has a problem in TX side and then notifies the upstream equipment of the problem. LOF implies that we are not seeing frames on the link for a period of time. Bob had also tested same J2300 router and cable on Verizon T1 circuit and observed no errors. So not likely a J2300 hardware issue.
[address possible doubt to prove the root cause] Cox testing with end-to-end loopback and all zeroes testing indicated no errors. However, it is possible that the testing equipment sensitivity may not be great enough detect the failure compared to Juniper router T1 interfaces which tend to be very sensitive to any errors on the line. Going forward we [workaround recommendation] recommend keeping hold-time configured on the T1 interface for this very reason. But ultimately it would be up to provider to correct any line defects.
[game plan] Current action plan is to wait for new ATM circuit to be installed to bypass the Amica equipment that this J2300 connects to. That will likely occur within the next several days. Will keep case open in the interim.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 136
Juniper Smartd Issues
PSN-2008-10-046 apparently covers multiple hdd related PRs. I looked at these PRs. If smartd is off, it may help PR/288011. However, I don't see how it would help PR/278580, PR/389540 and PR/390306.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 137
VPLS tagging configuration
****Old Way to config****
unit 25 {
description "DSH - ubr02 : 28/GCXG/061828//COXC";
encapsulation vlan-ccc;
vlan-id 25;
input-vlan-map {
swap;
vlan-id 1212;
}
output-vlan-map swap;
}
*****New Way to config*****
unit 4000 {
description "Lab - Todd SPN Test 1";
encapsulation vlan-ccc;
vlan-tags outer 4000 inner-range 1-4094;
input-vlan-map {
swap;
vlan-id 1101;
}
output-vlan-map swap;
}
Got a case with vpls tagging. Customer closed this case immediately for the reason of mis-configuration. Might worth for reference in the future
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 138
Juniper interface trouble shooting
To disable keepalive on a point-to-point interface. This is a tricky one as I have kept forgot it.
set no-keepalive
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 139
Platform code name
Atlas - The MX960, 14-slot carrier-class Ethernet platform, part of Harry. ATLAS
Alexander - M40e ALEXANDER
Autobahn - JUNOS upgrade to FreeBSD 6.1
Bellini - Bellini - Fine-grained (per VLAN) queuing for DPC (Dense Port Cards) on ATLAS
Bombay - T320 BOMBAY
Callypso - 7-slot chassis Ethernet switch MX480 Matrix takes Atlas cards, part of Harry.(IPG)
Calvin - M7i CALVIN
Chaser - M5 / M10 CHASER
Cosmo - M 20 COSMO
Dr Pepper - JUNOS on Saipan
Flamingo - M320 FPCs
Gibson - T640 GIBSON-LLC GIBSON-SHMC
Gimlet - LMNR chipset GIMLET
Greyhound - SONET OC768 PIC
Haddock - HGE-PIC qpp HADDOCK
Harry - Ethernet switch/router platforms HARRY
Havana HAVANA
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 140
Platform code name
Heavy Metal - T640 based platform (IPG)
Hobbes - M10i
Hobson - TX platform HOBSON
Hurricane - Hardware Stackable switch
- Java Fixed configuration switches:
- Espresso (Fixed configuration switch)
- Latte (Virtual chassis Switch)
- Caffeine :
- Biscotti (Software)
- Grande (8 slot 1.6Tbps chassis Switch)
- Venti (16 slot 3.2Tbps chassis switch)
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 141
Jsim Procedure (M120)
lab@blackjack-re0> show chassis fpc-feb-connectivity
lab@blackjack-re0> start shell pfe network feb0
RFEB0(blackjack-re0 vty)# show ichip ifd
RFEB0(blackjack-re0 vty)# show ichip 0 r counters
RFEB0(blackjack-re0 vty)# show ichip 0 iif statistics
RFEB0(blackjack-re0 vty)# jsim reset full 0 (must reset)
RFEB0(blackjack-re0 vty)# show ifl brief
RFEB0(blackjack-re0 vty)# set jsim iif 73 (must bind intf)
RFEB0(blackjack-re0 vty)# set jsim ipsrc 201.1.1.2
RFEB0(blackjack-re0 vty)# set jsim ipdst 200.1.1.2
RFEB0(blackjack-re0 vty)# jsim lookup verbose
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 142
Jsim Procedure (M120)
1) Find out which FPC (cFPC) is connected to which FEB
lab@blackjack-re0> show chassis fpc-feb-connectivity
FPC FPC type FPC state Connected FEB FEB state Link status
0 cFPC Online None
1 cFPC Online 1 Online OK
2 Type 3 Online 0 Online OK
3 Type 2 Online 3 Online OK
4 Type 2 Online 4 Online OK
5 Empty 5 Online
2) Console to the corresponding FEB (FEB 0 is connected to FPC3 @ slot 2)
lab@blackjack-re0> start shell pfe network feb0
RFEB platform (666Mhz MPC 8541 processor, 512MB memory, 512KB flash)
RFEB0(blackjack-re0 vty)# exit
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 143
Jsim Procedure (M120)
3) Find out which iCHIP is being used (from here, we know ICHIP 0 is being used)
RFEB0(blackjack-re0 vty)# show ichip ifd
I-chip global information:
ICHIP 0: Initialized, Version 2,
STREAM 32 (wan stream 0) has 1 IFDs.
IFD 191: so-2/0/0
ICHIP 1: Not Initialized,
ICHIP 2: Not Initialized,
ICHIP 3: Not Initialized,
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 144
Jsim Procedure (M120)
4) Collect some statistics of iCHIP 0
RFEB0(blackjack-re0 vty)# show ichip 0 r counters
Traffic stats:
Counter Name Total Rate Peak Rate
---------------------- ---------------- -------------- --------------
rcp_input_ucast 167035601285 31638906 39270060
(BYTE) 6868449722474 1265556255 1832927823
rcp_output_ucast 164600940855 31638902 39270077
(BYTE) 6771063304262 1265556088 1832926045
RFEB0(blackjack-re0 vty)# show ichip 0 iif statistics
Traffic stats:
Counter Name Total Rate Peak Rate
---------------------- ---------------- -------------- --------------
GFAB_BCNTR 91405146968728 592351311 784316693
KA_PCNTR 0 0 0
KA_BCNTR 0 0 0
Discard counters:
Counter Name Total Rate Peak Rate
---------------------- ---------------- -------------- --------------
WAN_DROP_CNTR 2194246089959 7582075 11888478
FAB_DROP_CNTR 15144376205 0 2380431
KA_DROP_CNTR 0 0 0
HOST_DROP_CNTR 194 0 0
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 145
Jsim Procedure (M120)
5) Reset JSIM ( everytime you change something, you need to reset JSIM)
RFEB0(blackjack-re0 vty)# jsim reset full 0
6) Find out the interface ifl ( here it is 73) we will bind to JSIM lookup
RFEB0(blackjack-re0 vty)# show ifl brief
Index Name Type Encapsulation Flags
----- -------------------- ------------- -------------- ------
71 ge-4/2/0.0 VLAN Tagged Ethernet 0x000000000000c000
73 so-2/0/0.0 Cisco HDLC Cisco HDLC 0x0000000000008010
72 ge-4/2/0.32767 VLAN Tagged Ethernet 0x000000000000c000
64 lo0.0 Unspecified Unspecified 0x0000000000000052
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 146
Jsim Procedure (M120)
7) Bind iif to jsim and setup stream lookup key
RFEB0(blackjack-re0 vty)# set jsim iif 73
RFEB0(blackjack-re0 vty)# set jsim ipsrc 201.1.1.2
RFEB0(blackjack-re0 vty)# set jsim ipdst 200.1.1.2
8) Finally, do the lookup (this is the data we are looking for)
RFEB0(blackjack-re0 vty)# jsim lookup verbose
Step Kp Address Data Description
---- -- ----------- -------- -----------
[ 1] 16 reg 000000 0000a679 nh: TID itable tid=10 offset=-7
itid 00000a 00040000 itable address (seg 0)
04000010 itable descriptor addr=0x000100 size=65536 idx_bits=16 bit_offset=0
lookup index=73
[ 2] 9 sram 00014b 10292f28 nh: extended buff-modify intermediate-nh addr=0x040a4a
sram 040a4a 7840b2ab Buffer Translate: write kb(8), off 42, bits 12, data 0xffffc40
[ 3] 9 sram 040a4b 44060b61 nh: multiple SER(no SE) hops=1 addr=0x110182
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 147
Tethereal to decode ixia packets.
-bash-2.05b$ tethereal -r cap.enc -V
Frame 1 (70 bytes on wire, 70 bytes captured)
Arrival Time: Feb 4, 2017 16:03:16.453824000
Time delta from previous packet: 0.000000000 seconds
Time relative to first packet: 0.000000000 seconds
Frame Number: 1
Packet Length: 70 bytes
Capture Length: 70 bytes
Ethernet II, Src: 00:1f:12:23:e6:02, Dst: 00:00:c8:01:01:64
Destination: 00:00:c8:01:01:64 (AltosCom_01:01:64)
Source: 00:1f:12:23:e6:02 (00:1f:12:23:e6:02)
Type: IP (0x0800)
Internet Protocol, Src Addr: 100.4.4.3 (100.4.4.3), Dst Addr: 200.1.1.100 (200.1.1.100)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 148
IPSec SP-MTU and Tunnel-MTU(M/J series)
On m-series: with sp-mtu of 1440, the max IP payload size that is 8 byte aligned is 1416, adding 20 bytes of IP header len results in 1436.
On j-series: with mtu of 1446 (tunnel-mtu-ipsec overheads), the max IP payload size that is 8 byte aligned is 1424, adding 20 bytes of IP header len becomes 1444.
Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 149Copyright © 2007 Juniper Networks, Inc. Proprietary and Confidential www.juniper.net 149