learner guide troubleshooting hp networks 1041 no watermark
TRANSCRIPT
Troubleshooting HP Networks
Learner Guide
Version 10.41
Copyright 2010 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice. The only warranties for HP products and
services are set forth in the express warranty statements accompanying such products and services. Nothing
herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial
errors or omissions contained herein.
This is an HP copyrighted work that may not be reproduced without the written permission of HP. You may not
use these materials to deliver training to any person outside of your organization without the written permission
of HP.
Troubleshooting HP Networks
Learner Guide
Rev 10.41
Rev 10.41 i
Contents Module 1: Troubleshooting Methodologies and Practices ................................. 1 - 1
Troubleshooting Methodology ................................................................ 1 - 2
Problem Solving Methodology ............................................................... 1 - 4
Identification and Analysis .................................................................... 1 - 6
Hypothesis and Validation ..................................................................... 1 - 8
Implementation and Verification ............................................................ 1 - 10
Summary ........................................................................................... 1 - 11
Module 2: Layer 1 (Physical Layer) Troubleshooting and Problem Resolution ....... 2 - 1
“It’s the cable” ..................................................................................... 2 - 2
Physical Layer Symptoms ....................................................................... 2 - 3
Module 3: Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution ..... 3 - 1
Switching ............................................................................................ 3 - 2
VLANs ................................................................................................ 3 - 3
Switch VLAN port types ........................................................................ 3 - 4
Link Aggregation .................................................................................. 3 - 9
LACP – Link Aggregation Control Protocol .............................................. 3 - 14
Configurable LACP States ..................................................................... 3 - 14
Static vs. Dynamic Link Aggregation ...................................................... 3 - 15
Spanning Tree .................................................................................... 3 - 16
Basic IRF Concepts .............................................................................. 3 - 21
How IRF simplifies networks ..................................................................3 - 23
Lab 4: VLAN Switching ....................................................................... 3 - 29
Module 4: Layer 3 (Network Layer) Troubleshooting and Problem Resolution ...... 4 - 1
Forwarding between VLANs .................................................................. 4 - 2
VRRP Basics ......................................................................................... 4 - 5
OSPF Basics ........................................................................................ 4 - 7
External and internal Border Gateway Protocol (BGP) .............................. 4 - 12
Network Address Translation (NAT) ....................................................... 4 - 14
Static and Dynamic NAT ...................................................................... 4 - 16
Lab 5: Layer 3 Practice and Tools .......................................................... 4 - 17
Lab 6: OSPF Routing Issues .................................................................. 4 - 18
Lab 7: Addressing Issues ...................................................................... 4 - 19
Lab 8: Inter-VLAN and Routing ............................................................. 4 - 20
Module 5: Layer 4 (Transport Layer) Troubleshooting and Problem Resolution ..... 5 - 1
Troubleshooting TCP/UDP ..................................................................... 5 - 2
Firewalls.............................................................................................. 5 - 7
Firewall types ....................................................................................... 5 - 9
Network address translator (NAT) ......................................................... 5 - 11
Module 6: Layer 5 (Application Layer) Troubleshooting and Problem Resolution .. 6 - 1
QoS process flow ................................................................................. 6 - 2
802.1p traffic prioritization .................................................................... 6 - 8
Traffic marking by an end station .......................................................... 6 - 11
Retaining priority between VLANs ......................................................... 6 - 12
Normal priority data traffic ................................................................... 6 - 14
Lab 10: Quality of Service .................................................................... 6 - 15
Troubleshooting HP Networks
ii Rev 10.41
Module 7: Troubleshooting an End-to-End Complex, Integrated Multi-Protocol
Network .................................................................................................... 7 - 1
Lab 11: Final lab ................................................................................... 7 - 2
Rev 10.41 1 –1
Troubleshooting Methodologies and Practices Module 1
No network or networking technology operates smoothly all of the time. Every network technician will be required at some time to troubleshoot issues in network configuration and performance. This module introduces basic techniques for network troubleshooting.
After completing this module, you will be able to:
Describe a framework for basic network troubleshooting
Troubleshooting HP Networks
1 –2 Rev 10.41
Troubleshooting Methodology Network troubleshooting benefits from having:
Methodology
A discipline for evaluating, analyzing and investigating problem conditions
Includes determining the scope of the problem, developing a hypothesis, testing it out, and if successful, implementing a resolution
Skill sets
Familiarity with network devices, how they operate and how they are managed
Technical tools that may be useful for investigating and verifying problems; from CLI commands and protocol analyzers
Good Q&A skills
Experience
Over time, applying a methodology and the technical tools helps develop your own “library” of problem recognition capabilities and yields a more efficient problem resolution process
The basics of troubleshooting any kind of networking trouble might be succinctly stated as “keep eliminating obvious causes until the real cause presents itself.” But understanding what this means requires a systematic approach and real discipline when attempting to identify causes from symptoms and apply the right fixes or workarounds.
Troubleshooting is a skill that all networking professionals learn by trial and error. But skipping some of the more painful or obvious errors can make your learning somewhat less trying than it might be otherwise. The most important characteristic to cultivate when solving problems is calmness. If you can keep a clear head when things fail or start degrading seriously, you’ll be better able to assess your situation and better equipped to solve whatever problems you discover.
Methodology Development of problem solving techniques is often an on-the-job acquisition process. Few of us can expect much along the lines of formal network troubleshooting training in our job positions for a number of reasons. These reasons may include:
The relatively fast pace of the day-to-day job tasks and challenges yields little time to pursue formal training on troubleshooting aspects such as technical tools like a protocol analyzer.
Troubleshooting Methodologies and Practices
Rev 10.41 1 –3
Few business environments provide the luxury of a “test lab” and the time to hone your skills where a progression of test problems can be examined, worked through, and resolutions tried out.
In the absence of a more ideal situation, a problem solving methodology can increase the effectiveness of support staff by standardizing the approach used to some extent. With a fairly modest amount of discipline, network technicians can improve their problem resolution efficiency in terms of the effort needed and the number of other people that must be directly involved.
Skill Sets There are a variety of skill sets that can enhance a network technician’s success in problem solving. Some of these skills are purely technical in nature. For instance, it is important to understand the fundamentals of how network devices operate and how they are managed. Having proficiency in reading logs or interpreting a protocol analyzer display are examples of having familiarity with the potential tools you may need to call upon from your “toolbox”.
Other skills are much less technical, but still very important. As part of the problem investigation process, a network technician may need to talk with various levels of staff. The staff may include non-technical end-users and business unit managers to software and hardware vendor support people. Having sufficient interpersonal skills coupled with good investigative reporter-like skills can expedite the isolation of a problem and eliminate the “noise” that often conceals the real problem.
Proactive IT support groups tend to spend time on developing procedures and tools to facilitate problem resolutions. Some examples of technical tools used by the network technicians are:
Device logs—Archived instances of the logs as well as the current one may provide hints of where the problem may be. At the very least, familiarity with a log file’s typical contents helps you differentiate normal from abnormal situations.
Device statistics and status information—Being able to determine the health of a system or the network is important for gathering the “vital” signs. This type of information can include anything from port statistics and CPU utilization to network reachability results.
Protocol analyzer—Although this may not be a frequently used tool, it can be invaluable for examining what conversations are or are not occurring between communicating devices.
A problem solving methodology that is refined over time can be very beneficial to network technicians. Being methodical and learning from the macro and micro levels of mistakes can help network technicians improve problem recognition capabilities and yield a more efficient application of a problem resolution process.
Troublesh
1 –4
Prob
hooting HP Ne
blem So
A probleAlthoughsituationin the effprobabiligeneral ptoday’s c
There aremust be methodoor possib
The six st
Identechpotemult
Analtools
tworks
olving
m solving mh there is no s, a general forts made tity of a succeproblem solvontemporar
e six steps toexecuted in
ology state tbly return to
teps are:
ntification—Unical perspe
ential probletiple perspec
lysis—Evaluas, product do
Metho
methodologyone specificframework co solve a proessful resoluving methodry network e
o the problemorder startinhat if a step the top leve
Understand aective. Somem is before ctives.
ate the situatocumentatio
dology
Figure 1
y is a processc model thatcan provide oblem. Apption. This g
dology that hnvironments
m solving mng with idenfails, you m
el step.
and documeetimes it is psearching fo
tion by inveson and user
y
1
s for managit may be useguidelines a
plying a methraphic illustrhas many aps.
methodology ntification. Tust return to
ent the probpossible to loor a cause wh
stigating usiinput.
ing problemeful for all prand help enshodology carates the frapplications, i
y outlined heThe rules of to the preced
blem from boose sight of when we don’
ing problem
Rev 1
resolution. roblem sure efficiencan improve tmework for ncluding in
ere. The stepthe
ding step abo
oth a user anwhat the ’t consider
m resolution
10.41
cy he a
ps
ove
nd
Troubleshooting Methodologies and Practices
Rev 10.41 1 –5
Hypothesis—Develop possible resolutions based on the analysis and document a possible resolution. This documentation may be fairly informal, but it is important to be able to explain it in writing. Doing so can reveal a hypothesis that is unclear and for which a possible resolution may not be plausible.
Validation—Run a validation process to prove or disprove the hypothesis. This may not be particularly feasible, for example, if you have no test lab equipment to try out your hypothesis. At the very least, performing a walk-through of the hypothesis in an articulate manner with other team members may help.
Implementation—Develop an implementation plan along with a back-out plan, just in case, and then implement the resolution. For example, have a backup configuration and software image readily available.
Verification—Verify the success or failure of the implementation. If it fails, implement the back-out plan.
Troublesh
1 –6
Iden
hooting HP Ne
ntificati
The first process. assumingevaluatinanalysis o
For examhost, a gwhat thehosts, buin the samissue maconcerniperformsdefault g
The iden
Docudepe
tworks
on and
step of the s Try to obseg somethingng and resolvof symptoms
mple, it is improup of hosty have in co
ut not remotme VLAN cay be a logicang connectivs Layer 3 forwgateway is do
tification pro
umenting thending on th
d Analy
six-step metrve everythi
g. Because nving connects to determi
portant to dets, or the entmmon. For e hosts, veri
an communical problem wvity with thewarding on one using La
ocess consis
he physical sehe problem s
ysis
Figure 2
hodology is ng, not just
network troutivity issues, ne the scop
etermine whtire networkinstance, if afy connectivcate with locwith the defae default gatebehalf of loc
ayer 2 addre
ts of doing t
ettings. Thescenario, bu
2
identificatiothe apparenbleshooting the generale of the issu
hether the pr. If many hoa host can c
vity with its dcal hosts, buault gatewayeway. Althocal hosts, thessing.
tasks that ca
e specifics wit some exam
on, which is ant problem, ag primarily inl procedure
ue.
roblem is affosts are affecommunicate
default gatewut not remotey or a physicough the defeir commun
an include:
ill of course mples are the
Rev 1
an observatiand avoid nvolves begins with
fecting a sincted, determe with local way. If all hoe hosts, the al problem
fault gatewayication with
vary e following:
10.41
ion
an
gle mine
osts
y the
Troubleshooting Methodologies and Practices
Rev 10.41 1 –7
What client, server and network device hardware and software are in use?
What is the network topology between the client and server?
Where are the applications and services located? Determine the effects the problem has on the user/customer and the business.
Developing a problem definition—Document probable failures.
Prioritizing the problem—Prioritize based on defined user/customer policies. Is this a problem that must be investigated immediately or can it wait until you can assemble a strategy using the problem solving methodology?
Step 2 is analysis. Analysis is the process of isolating the problem with the objective to narrow down the different possibilities.
The analysis process considers such factors as the following:
Does the system work without the problem
Previous changes to the system
Something new, such as networking equipment, that may have been introduced
Any changes to peripheral equipment that may have been made
Whether the hardware or software is being used correctly
With the scope of the problem having been narrowed down, that can help suggest the type of network troubleshooting tools you may want to use to test probable causes. For example, the problem investigation may involve using simple network reachability tools, such as traceroute or ping, or examination of the logs of multiple switches, or even use of a protocol analyzer.
Troublesh
1 –8
Hyp
hooting HP Ne
othesis
Step 3 is informatcauses.
Some thi
Wha
You youron thproc
Evenimm
Validatiohypothesfact be re
tworks
s and V
hypothesis. ion acquired
ings to keep
at is the tech
need a validr intuition mhat primarilycess.
ntual resolutmediately obv
n, step 4, typsis is reasonaesolved after
Validati
The hypothd from the an
in mind are
hnical reason
dation proceday prove to y makes it di
ion of the prvious.
pically involvable. It increr implement
on
Figure 3
hesis step invnalysis step t
e:
n for the bus
dure for the be correct aifficult for m
roblem could
ves experimeeases the cotation of a po
3
volves the evto determin
iness proble
hypothesis at times, in thanagement
d create side
entally deternfidence levotential solu
valuation of e a number
em?
to be usablehe business to feel confi
e effects, som
rmining whevel that the pution.
Rev 1
the of probable
e. Althoughworld, relyinident about
me that are
ether the problem will
10.41
ng the
not
in
Troubleshooting Methodologies and Practices
Rev 10.41 1 –9
The validation step involves:
Testing each hypothesis until you validate a probable cause with a high degree of certainty. The objective is not necessarily to be 100% sure, but to balance the time criticality of resolving a problem with the information you have available.
If validation fails for all probable causes you developed, then you may need to return to the problem definition phase and start over. Despite what may appear to be time wasted, you will likely have improved your awareness of the problem situation and will have some additional facts to use when you attempt to redefine the problem.
Troublesh
1 –10
Imp
hooting HP Ne
lement
Implemeor netwoimplemestate.
The plan
Deve
Devesucc
Deveremo
Verificatiand deteeffects arphase is
Upon sucproblem documen
tworks
tation a
entation, stepork fix or montation fails,
ning involve
elopment of
elopment ofcessful.
elopment ofoved, if it fai
on, step 6, isermining thatre unacceptaexecuted.
ccessful comresolution s
ntation can l
and Ve
p 5, requires dification alo, you must b
es:
f a specific im
f a verificatio
f a back-out ls. It should
s the processt any side efable, the bac
mpletion, theshould be doead to lengt
rificatio
Figure 4
planning foong with prebe able to re
mplementati
on process to
plan to ensud also addres
s of proving ffects are accck-out plan d
e user or custocumented ithy resolutio
on
4
or installationeparation fostore the sys
on plan.
o prove the i
ure the impless how to ha
the implemceptable. If developed in
tomer must n a trouble l
on for recurri
n of some for failure. If astem to a pr
implementa
ementation andle side ef
mentation waverification n the implem
be informedlog. Lack of ing problem
Rev 1
orm of systeman revious stabl
tion was
can be ffects.
as successfulfails or side
mentation
d and the f
ms.
10.41
m
e
Troubleshooting Methodologies and Practices
Rev 10.41 1 –11
Summary Network troubleshooting benefits from having a methodology, skill sets and
experience
General problem solving methodology consists of six steps:
Identification: Develop a problem statement
Analysis: Narrow the scope
Hypothesis: Define procedures to validate
Validation: Test probable causes
Implementation: Make changes with back-out plan ready
Verification: Ensure that changes resolve problem without side effects
Troubleshooting HP Networks
1 –12 Rev 10.41
This page left blank intentionally.
Rev 10.41 2 –1
Layer 1 (Physical Layer) Troubleshooting and
Problem Resolution Module 2
In this module, various layer 1 problems will be discussed.
The technologies include:
Cable / Link problems
Link Errors
Troubleshooting HP Networks
2 –2 Rev 10.41
“It’s the cable”
Figure 2.1: Cables
Some of the most common Layer 1 problems can be isolated to the cable.
Common physical layer problems:
Bad Cables can be terminated improperly or have physical breaks in one or
more conductors, etc.
Mis-wired cables can be terminated in the wrong order. A common symptom
here is that a cable works with 10 or 100Mbps links but not 1 Gig links because
of the extra conductors required for Gigabit. It is also common to have fiber
links mis-wired so that transmit is connected to transmit and receive is connected
to receive.
Interference is mostly a problem with unshielded copper cables. This can be
due to running data cable alongside power cable.
Wrong cable types could be using a CAT3 cable with a Gigabit link or a
multimode fiber cable with transceivers that require single mode, etc.
Layer 1 (Physical Layer) Troubleshooting and Problem Resolution
Rev 10.41 2 –3
Physical Layer Symptoms These are some common symptoms of layer 1 issues:
No link
Link on one end only
Errors on link
To troubleshoot these issues, the switches port counters and event logs can be very
useful.
A-Series commands
display interface <INT-ID>
display interface brief
or
display brief interface
display logbuffer reverse
E-Series commands
show interfaces <INT-ID>
show interfaces brief
log –r
Here are some examples of these commands.
[4800G]display interface GigabitEthernet 1/0/2
GigabitEthernet1/0/2 current state: UP
IP Packet Frame Type: PKTFMT_ETHNT_2, Hardware Address: 0022-
5782-fec2
Description: GigabitEthernet1/0/2 Interface
Loopback is not set
Media type is twisted pair
Port hardware type is 1000_BASE_T
1000Mbps-speed mode, full-duplex mode
Link speed type is autonegotiation, link duplex type is
autonegotiation
Flow-control is not enabled
The Maximum Frame Length is 1522
Broadcast MAX-pps: 3000
Unicast MAX-ratio: 100%
Multicast MAX-ratio: 100%
Forbid jumbo frame to pass
PVID: 1
Mdi type: auto
Link delay is 0(sec)
Port link-type: access
Tagged VLAN ID : none
Untagged VLAN ID : 1
Port priority: 0
Peak value of input: 279 bytes/sec, at 2000-04-26 12:09:54
Peak value of output: 78 bytes/sec, at 2000-04-26 12:09:59
Last 300 seconds input: 1 packets/sec 115 bytes/sec 0%
Troubleshooting HP Networks
2 –4 Rev 10.41
Last 300 seconds output: 0 packets/sec 78 bytes/sec 0%
Input (total): 916 packets, 136158 bytes
186 unicasts, 79 broadcasts, 651 multicasts
Input (normal): 916 packets, - bytes
186 unicasts, 79 broadcasts, 651 multicasts
Input: 0 input errors, 0 runts, 0 giants, 0 throttles
0 CRC, 0 frame, - overruns, 0 aborts
- ignored, - parity errors
Output (total): 199 packets, 35587 bytes
146 unicasts, 10 broadcasts, 43 multicasts, 0 pauses
Output (normal): 199 packets, - bytes
146 unicasts, 10 broadcasts, 43 multicasts, 0 pauses
Output: 0 output errors, - underruns, - buffer failures
0 aborts, 0 deferred, 0 collisions, 0 late collisions
0 lost carrier, - no carrier
[4800G]display brief interface
The brief information of interface(s) under route mode:
Interface Link Protocol-link Protocol type Main IP
NULL0 UP UP(spoofing) NULL --
Vlan1 UP UP ETHERNET 16.1.1.50
The brief information of interface(s) under bridge mode:
Interface Link Speed Duplex Link-type PVID
GE1/0/1 DOWN auto auto access 1
GE1/0/2 UP 1G(a) full(a) access 1
GE1/0/3 DOWN auto auto access 1
---- More ----
[4800G]display logbuffer reverse
Logging buffer configuration and contents:enabled
Allowed max buffer size : 1024
Actual buffer size : 512
Channel number : 4 , Channel name : logbuffer
Dropped messages : 0
Overwritten messages : 0
Current messages : 166
%Apr 26 13:54:59:803 2000 4800G LLDP/2/CREREM:Port
GigabitEthernet1/0/2 (IfIndex 9437185):Created new neighbor,
chassis ID: 001c-2e96-8900, port ID: 1.
%Apr 26 13:54:58:908 2000 4800G MSTP/2/PFWD:Instance 0's
GigabitEthernet1/0/2 has been set to forwarding state!
%Apr 26 13:54:58:907 2000 4800G IFNET/4/UPDOWN:
Line protocol on the interface Vlan-interface1 is UP
%Apr 26 13:54:58:907 2000 4800G IFNET/4/LINK UPDOWN:
Vlan-interface1: link status is UP
%Apr 26 13:54:58:873 2000 4800G IFNET/4/LINK UPDOWN:
GigabitEthernet1/0/2: link status is UP
%Apr 26 13:54:56:209 2000 4800G IFNET/4/UPDOWN:
Line protocol on the interface Vlan-interface1 is DOWN
---- More ----
Layer 1 (Physical Layer) Troubleshooting and Problem Resolution
Rev 10.41 2 –5
E3500yl# show interfaces 23
Status and Counters - Port Counters for port 23
Name :
MAC Address : 001c2e-968929
Link Status : Up
Totals (Since boot or last clear) :
Bytes Rx : 1,821,092 Bytes Tx :
304,614
Unicast Rx : 1626 Unicast Tx : 1938
Bcast/Mcast Rx : 10,253 Bcast/Mcast Tx : 503
Errors (Since boot or last clear) :
FCS Rx : 0 Drops Tx : 0
Alignment Rx : 0 Collisions Tx : 0
Runts Rx : 0 Late Colln Tx : 0
Giants Rx : 0 Excessive Colln : 0
Total Rx Errors : 0 Deferred Tx : 0
Others (Since boot or last clear) :
Discard Rx : 0 Out Queue Len : 0
Unknown Protos : 0
Rates (5 minute weighted average) :
Total Rx (bps) : 5,001,008 Total Tx (bps) : 3,010,520
Unicast Rx (Pkts/sec) : 0 Unicast Tx (Pkts/sec) : 0
B/Mcast Rx (Pkts/sec) : 0 B/Mcast Tx (Pkts/sec) : 0
Utilization Rx : 00.50 % Utilization Tx : 00.30 %
E3500yl# show interfaces brief
Status and Counters - Port Status
| Intrusion MDI Flow Bcast
Port Type | Alert Enabled Status Mode Mode Ctrl Limit
----- ----- + ------ ------- ------ ------- ----- ---- -----
1 1000 | No Yes Up 1000FDx MDI off 0
2 1000 | No Yes Down 1000FDx Auto off 0
3 1000 | No Yes Down 1000FDx Auto off 0
4 1000 | No Yes Down 1000FDx Auto off 0
5 1000 | No Yes Down 1000FDx Auto off 0
6 1000 | No Yes Down 1000FDx Auto off 0
-- MORE --, next page: Space, next line: Enter, quit: Control-C
E3500yl# log -r
Keys: W=Warning I=Information
M=Major D=Debug E=Error
---- Reverse event Log listing: Events Since Boot ----
I 10/22/10 17:52:38 00561 ports: port 1 Applying Power to PD.
I 10/22/10 17:52:38 00560 ports: port 1 PD Detected.
I 10/22/10 17:52:36 00076 ports: port 1 is now on-line
I 10/22/10 17:52:35 00565 ports: port 1 PD Removed.
I 10/22/10 17:52:34 00561 ports: port 1 Applying Power to PD.
I 10/22/10 17:52:34 00560 ports: port 1 PD Detected.
I 10/22/10 17:52:31 00565 ports: port 1 PD Removed.
I 10/22/10 17:52:30 00077 ports: port 1 is now off-line
-- MORE --, next page: Space, next line: Enter, quit: Control-C
Troubleshooting HP Networks
2 –6 Rev 10.41
Troubleshooting no link
Step 1: Determine if one or two fibers are in use. BX (bi-directional) transceivers use
only one fiber for both transmit and receive. There are two "flavors" of BX
transceiver. One is a "D" (downstream), the other is a "U" (upstream). You must
connect a "D" to a "U". You cannot connect a "D" to a "D", and you cannot
connect a "U" to a "U".
Is this a BX transceiver link?
Action: If BX, try using the other "flavor" (D or U). Or try a connection to a nearby
device, ensuring D connects to U.
Step 2: Roll (swap) transmit and receive fibers at only one place; for BX ensure "D"
connects to"U".
Does link come up?
Step 3: If no link after rolling the fibers, try connecting to a nearby device with
crossover fiber.
NOTE: Fiber must be "crossover", meaning transmit at one end connects to receive
at the far end. Many fiber patchcords are mis-labeled. Do not rely on color-coding
of strain relief, or "A" and "B" labels on the patchcord, to determine if patchcord is
crossover. (Those can be wrong.) Instead, use manufacturer's lettering on outside of
fiber to identify which strand is which. With connector nub facing up on each end,
and with each connector pointing the same direction, be sure lettering is on left at
one end, and on right at other end, as shown here:
With both connectors facing
same direction, crossover fiber
has lettering on left fiber at one
end, and lettering on right fiber
at other end.
Does link come up?
Action: If no link occurs using crossover fiber to nearby device with known-good
transceiver, then validate with physical inspection that this is a genuine HP
transceiver.
Layer 1 (Physical Layer) Troubleshooting and Problem Resolution
Rev 10.41 2 –7
Troubleshooting Errors on link
HP switches keep per-port statistics (counters) that help us diagnose problems on the
link or on the network. In addition to "normal" errors like an occasional bad packet
received (with incorrect FCS/CRC for example), HP switches alert users to abnormal
or "excessive" errors.
"Excessive" errors and FFI
FFI (Find, Fix, Inform) is a feature of HP switches that informs the user when the switch detects a
large number of errors in a short period of time, with specific parameters defined by the HP
Switch Lab. The feature was originally called "Fault-finder", and is a good indicator of
problems on the link or network. Here are the FFI messages and explanations from the "Help"
text in the menu-based event log. Description is what the switch detected. Possible causes are
documented, as are user Actions to resolve the problem.
Too many undersized/giant packets
Description: A device on this port is transmitting packets shorter than 64 bytes or
longer than 1518 bytes (longer than 1522 bytes if tagged), with valid CRCs.
Possible Causes: A misconfigured NIC or a malfunctioning NIC, NIC driver, or
transceiver.
Actions:
a. Check the NIC for a misconfiguration.
b. Update the NIC driver software.
c. Replace the malfunctioning NIC or transceiver.
d. Check for a short-circuit in the cable path connected to this port.
Excessive jabbering
Description: A device on this port is incessantly transmitting packets ("jabbering" is
detected as oversized packets with CRC errors).
Possible Causes: A misconfigured NIC, or a malfunctioning NIC or transceiver. It
could also be caused by a short-circuit in the network cable path.
Actions:
a. Check the NIC for a misconfiguration.
b. Update the NIC driver software.
c. Replace the NIC or transceiver.
d. Check for a short-circuit in the cable path connected to this port.
Excessive CRC/alignment errors
Description: A high percentage of data errors was detected on this port.
Possible Causes: Faulty cabling or topology, half/full duplex mismatch, a
misconfigured NIC, or a malfunctioning NIC, NIC driver, or transceiver.
Troubleshooting HP Networks
2 –8 Rev 10.41
Actions:
a. If this port is 100Base-T, make sure the cable, connectors, punch-down
blocks, and patch panels connecting to the port are Category 5 or better.
Verify the correctness of the installation using a Category 5 test device.
b. Check the directly-connected device for mismatches in half/full duplex
operation (half duplex on the switch and full duplex on the connected
device, or the reverse).
c. Update the NIC driver software.
d. Verify that the network topology conforms to IEEE 802.3 standards.
e. Replace or relocate the cable. Also check wiring closet components,
transceivers, and NICs for proper operation.
Excessive late collisions
Description: Late collisions (collisions detected after transmitting ~64 bytes) were
detected on this port.
Possible Causes: An overextended LAN topology, half/full duplex mismatch, or a
misconfigured or faulty device connected to the port.
Actions:
a. Verify that the network topology conforms to IEEE 802.3 standards. Insert
bridges or switches, if needed, to extend the network topology.
b. Check the directly-connected device for mismatches in half/full duplex
operation (half duplex on the switch and full duplex on the connected
device).
c. If this port is 100Base-T, make sure the cable connecting to that port is
Category 5 or better.
d. Check for faulty cabling, transceivers, and NICs.
High collision or drop rate
Description: A large number of collisions or packet drops have occurred on the
port.
Possible Causes: An extremely high level of traffic on this port, half/full duplex
mismatch, a misconfigured or malfunctioning NIC or transceiver on a device
connected to this port, or a topology loop in the network.
Actions:
a. Use a network monitoring device or application to determine the traffic
levels on the affected segment. If needed, consider subdividing that
segment with switches or bridges, or moving high-traffic devices to their
own switch ports.
Layer 1 (Physical Layer) Troubleshooting and Problem Resolution
Rev 10.41 2 –9
b. Check the directly-connected device for mismatches in half/full duplex
operation (half duplex on the switch and full duplex on the connected
device).
c. Check for a misconfigured NIC or transceiver (such as a transceiver
configured for "loopback test" or "SQE test").
d. Verify that there are no topology loops in your network. If not enabled,
you may also enable spanning.
Excessive broadcasts
Description: An excessively high rate of broadcast packets was received on the
port. This degrades the performance of all devices connected to this switch.
Possible Causes: This is usually caused by a network topology loop, but can also be
due to a malfunctioning device, NIC, NIC driver, or software application.
Actions:
a. Verify that there are no topology loops in your network.
b. Find and correct any malfunctioning devices or NICs on the segment.
c. Find and correct any malfunctioning applications on devices on the
segment.
Troubleshooting HP Networks
2 –10 Rev 10.41
This page left blank intentionally.
Rev 10.41 3 –1
Layer 2 (Data Link Layer) Troubleshooting and
Problem Resolution Module 3
In this module, various layer 2 technologies will be reviewed and common problems
will be discussed.
The technologies include:
Layer 2 switching
VLANs
Link Aggregation
Spanning Tree
IRF
Troubleshooting HP Networks
3 –2 Rev 10.41
Switching
Figure 3.1: Switching
Today’s switches forward frames in two ways. They flood frame and they switch
frames. Frames are flooded if their destination is unknown. That is, the destination
doesn’t have an entry in the MAC address table. This is also the biggest difference
between hubs and switches. Hubs do not maintain a MAC address table.
When the destination address is known, then a frame is only forwarded towards that
destination. This has the effect of reducing traffic on a network because traffic is not
sent out on all links.
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –3
VLANs Virtual LAN – A logical broadcast domain
VLANS are used to divide a network segment to smaller sub networks to :
Reduce the overhead of layer 2 broadcast.
Increase security.
Improve management of network infrastructure
VLANs are created through software configuration.
Type of VLANS
Port-based VLANs
MAC address-based VLANs
Protocol-based VLANs
IP-subnet-based VLANs
Policy-based VLANs
A virtual LAN (VLAN) is a collection of network nodes that are logically grouped
together to form a separate broadcast domain. A VLAN has the same general
attributes as a physical LAN, but it allows all nodes for a particular VLAN to be
grouped together, regardless of physical location. One advantage of using VLANs is
design flexibility.
VLANs allow individual users to be grouped based on business needs.
Connectivity within a VLAN is established and maintained through software
configuration. The list above is a partial list of supported VLAN types.
A-Series switches also support Voice VLANs and policy-based VLANs, which are
used with 802.1X authentication. This security technology is covered in the
Accredited Systems Engineer (ASE) certification track.
Troubleshooting HP Networks
3 –4 Rev 10.41
Switch VLAN port types Access ports:
Belong to one VLAN – Port is untagged
Trunk ports:
Carry multiple VLANs on a single physical link
VLANs are 802.1Q tagged
The native VLAN is untagged
Hybrid ports:
Belongs to multiple VLANs
Multiple VLANs can be untagged and tagged
Typically used for IP phone connection
Also in conjunction with protocol VLANs, IP subnet VLANs
A-Series switches
By default, VLAN 1 is the native VLAN. To define a trunk:
interface gi 1/0/1
port link-type trunk
port trunk permit [all | vlan ids]
port trunk pvid vlan [id] (Defines Native VLAN.)
In this case, VLAN 1 will be tagged if still carried. The undo port trunk permit vlan 1
command undoes VLAN 1 assignment.
Control plane info, including BPDU and LLDP frames, is sent untagged. To configure
multiple ports, define port groups:
[switch] port-group manual [port-group-name]
[switch] group-member [port names]
[switch] port link-type [trunk | hybrid | access]
Access ports are ports that belong to a single VLAN and the traffic is sent and
received untagged. There are two methods to define access ports.
Add access ports to VLAN for PCs
[SW-A]vlan 100
[SW-A-vlan100]port gigabitethernet 1/0/1 to gig 1/0/20
OR in interface configuration mode, set interface as an access port in VLAN 100
[SW-A]interface gi 1/0/1
[SW-A-GigabitEthernet1/0/1]port link-type access
[SW-A-GigabitEthernet1/0/1]port access vlan 100
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –5
Use these commands to view VLAN membership.
display vlan [vid]
display vlan all
Hybrid Ports
Hybrid ports are used mostly for IP phones. Hybrid ports can be assigned to multiple
VLAN as tagged or untagged.
To set Hybrid ports using a port group:
[SW]port-group manual phones-1
[SW-port-group-manual-phones-1]group-member gi 1/0/11 to gi
1/0/20
[SW-port-group-manual-phones-1]port link-type hybrid
To set Data VLAN 100 as the native VLAN:
[SW-port-group-manual-phones-1]port hybrid PVID 100
Note: Hybrid port is still part of VLAN 1. Then Remove hybrid port from VLAN 1
[SW-port-group-manual-phones-1]undo port hybrid vlan 1 untagged
To set VLAN 200 as voice VLAN:
[SW-port-group-manual-phones-1]voice vlan 200 enable
This makes VLAN tagged on port and auto-QoS if Phone SNMP OUI Is detected.
Voice VLAN command will dynamically:
Allocate the voice vlan as a tagged vlan with auto qos if a predefined phone
SNMP OUI is detected.
Add OUI with voice OUI command at system view.
Hybrid ports can be set as untagged in one or more VLANs. Here is an example of
configuration on a hybrid port to use with protocol VLAN:
[SWA]vlan 2
[SWA-vlan2]Description IP and ARP VLAN
[SWA-vlan2]protocol-vlan mode ethernetii etype 0800
[SWA-vlan2]protocol-vlan mode ethernetii etype 0806
[SWA-vlan2]vlan 3
[SWA-vlan2]Description Novell IPX VLAN
[SWA-vlan3]protocol-vlan ipx llc
[SWA-vlan3]interface gigabit 1/1/1
[SWA-gigabit1/1/1]description Access port Separate IP and IPX
traffic
[SWA-gigabit1/1/1]port link-type hybrid
[SWA-gigabit1/1/1]undo port hybrid vlan 1
[SWA-gigabit1/1/1]port hybrid vlan 2 3 untagged
Troubleshooting HP Networks
3 –6 Rev 10.41
[SWA-gigabit1/1/1]port hybrid protocol-vlan vlan 2 all
[SWA-gigabit1/1/1]port hybrid protocol-vlan vlan 3 all
[SWA-vlan3]interface gigabit 1/1/23
Hybrid ports can be assigned to multiple VLAN as tagged or untagged.
To set Hybrid ports using a port group
[SW]port-group manual phones-1
[SW-port-group-manual-phones-1]group-member gi 1/0/11 to gi
1/0/20
[SW-port-group-manual-phones-1]port link-type hybrid
Ts set Data VLAN 100 as the native VLAN
[SW-port-group-manual-phones-1]port hybrid PVID 100
Note
Hybrid port is still part of VLAN 1. Then Remove hybrid port from VLAN 1.
[SW-port-group-manual-phones-1]undo port hybrid vlan 1 untagged
To set VLAN 200 as voice VLAN
[SW-port-group-manual-phones-1]voice vlan 200 enable
This makes VLAN tagged on port and auto-qos if Phone SNMP OUI Is detected.
[SWA-gigabit1/1/23]description Trunk port Separate IP and IPX
traffic
[SWA-gigabit1/1/23]port link-type trunk
[SWA-gigabit1/1/23]port trunk permit vlan 2 3
Trunk Ports
On trunk 802.1Q ports: one VLAN at most is untagged, all other VLANs are tagged
To configure the trunk interfaces & allow the VLANs:
[SW-A]interface gi 1/0/23
[SW-A-GigabitEthernet1/0/23]port link-type trunk
[SW-A-GigabitEthernet1/0/23]port trunk permit vlan 100 200
[SW-A]interface gi 1/0/24
[SW-A-GigabitEthernet1/0/24]port link-type trunk
[SW-A-GigabitEthernet1/0/24]port trunk permit vlan all
List trunk ports:
[SW-A]display port trunk
Interface PVID VLAN passing
GE1/0/23 1 1, 100, 200
GE1/0/24 1 1, 100, 200
On edge switches you can set the uplinks as trunk ports carrying all VLANs:
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –7
port link-type trunk
port trunk permit vlan all
Note
Do not confuse ―trunk‖ ports with the link aggregation ports that are called trunk
ports on HP E-Series switches.
On distribution/core switches, set exactly what VLANs should be carried on
downlinks to edge switches:
port link-type trunk
port trunk permit vlan 100 200
Note
VLAN 1 is set by default.
To change the native VLAN to VLAN 99
[SW-A-GigabitEthernet1/0/23]port trunk PVID 99
This forces the interface to be tagged on VLAN 1. If VLAN 1 is not desired on port,
remove it
[SW-A-GigabitEthernet1/0/23]undo port trunk permit vlan 1
List trunk ports
[SW-A]display port trunk
Interface PVID VLAN passing
GE1/0/23 99 99, 100, 200
GE1/0/24 99 99, 100, 200
VLAN 1 is not necessary on A-Series switches. For example: BPDUs for STP, LLDP or
LACP are sent untagged whatever is the setup of VLANs on the link. BPDUs are
accepted by a receiving switch because their destination MAC address matches the
list of MAC addresses on the ports. In other words, because the protocols (LLDP, STP,
LACP) are enabled on port and global levels.
Troubleshooting HP Networks
3 –8 Rev 10.41
E-Series switches
E-Series switches do not use the same terminology as the A-Series. On E-Series
devices, VLAN membership is configured from the VLAN context with the tagged
and untagged commands. A port can be considered to be a VLAN trunk port if it is
assigned to more than one VLAN. Similarly, a port can be considered to be an
access port if it is only assigned to one VLAN for untagged traffic.
To configure a port to be an untagged member of a VLAN (access port):
E-Series(config)# vlan 100
E-Series(vlan-100)# untagged a1-a12
To configure a port to be a tagged member of a VLAN (trunk port):
E-Series(config)# vlan 100
E-Series(vlan-100)# tagged a1-a12
E-Series(vlan-100)# vlan 200
E-Series(vlan-200)# tagged a1-a12
E-Series(vlan-100)# vlan 5
E-Series(vlan-5)# untagged a1-a12 (This is optional)
To configure a port to be a voice VLAN:
E-Series(config)# vlan 100
E-Series(vlan-100)# voice
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –9
Link Aggregation Link aggregation is called trunking on HP E-Series switches.
E-Series switches support two trunking methods:
HP Port Trunking—HP has supported port trunking since its first offering of
switches in the mid-1990s. The original HP port trunking technology remains an
option on ProCurve switches. HP port trunking is the default on E-Series switches.
For proper trunk operation, all links in the same trunk group must have the same
speed, duplex, and flow control
Link Aggregation Control Protocol (LACP)—The IEEE standard for link
aggregation. HP’s implementation of LACP supports both active and passive
configuration of trunking.
These link-aggregation methods impose a similar set of requirements and restrictions.
However, LACP imposes an additional restriction—the links must operate in full-
duplex mode. This is rarely a concern because trunks consist of point-to-point links
between switches, and these links will usually negotiate up to full duplex operation.
HP port trunking does not have this requirement.
Both methods for port trunking share one important limitation in the area of load
sharing—they are static methods. They do not adjust to reflect traffic volume on the
links or evaluate an individual conversation to determine which link would be best at
a given moment. Instead, all methods distribute the conversations evenly across all
links with the expectation that the load generally is balanced. The benefits of
trunking are always best realized in the presence of many source and destination
points on each side of the trunk.
Configuring Port Trunking on E-Series Devices
To enable static port trunking from the CLI, you use the trunk command. At the global
configuration level, issue the trunk command followed by a list of the ports that will
be aggregated, a name for the trunk, and the type of trunk (HP trunk or LACP). The
ports need not be contiguous, although the example above shows four contiguous
ports. A list of ports is separated by commas, for example: trunk a1,a7,b1,b24 trk1
LACP.
Note
The 2500 series switches support only one trunk. If the trunk is statically defined,
it will be named ―Trk1.‖
The trunk configuration must be performed on both sides of the trunk before the
redundant links are connected.
Edge_1(config)# trunk ?
[ethernet] PORT-LIST Specify the ports that are to be
added to/removed from a trunk.
Edge_1(config)# trunk c1,c2 ?
Troubleshooting HP Networks
3 –10 Rev 10.41
trk1 Trunk group 1
trk2 Trunk group 2
...
Edge_1(config)# trunk c1,c2 trk1 ?
trunk Do not use any protocol to create or maintain
the trunk.
lacp Use IEEE 802.1ad Link Aggregation protocol.
<cr>
Edge_1(config)# trunk c1,c2 trk1 lacp
The trunk command is used to create an HP port trunk or LACP port trunk
trk1, trk2, etc. are fixed label names for trunks
On the 8100fl series, trunks are referred to as Link Aggregation Groups
Configuring Link Aggregation on A-Series Devices
Static Link Aggregation
1. Create VLAN 10 and aggregate interface 1, and assign the aggregate interface
to VLAN 10.
<DeviceA> system-view
[DeviceA] vlan 10
[DeviceA-vlan10] quit
[DeviceA] interface bridge-aggregation 1
[DeviceA-Bridge-Aggregation1] port access vlan 10
[DeviceA-Bridge-Aggregation1] quit
2. Assign ports GE4/0/1 through GE4/0/3 to link aggregation group 1 and
VLAN 10 one at a time.
[DeviceA] interface gigabitethernet 4/0/1
[DeviceA-Gigabitethernet4/0/1] port link-aggregation group 1
[DeviceA-Gigabitethernet4/0/1] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
[DeviceA-Gigabitethernet4/0/1] quit
[DeviceA] interface gigabitethernet 4/0/2
[DeviceA-Gigabitethernet4/0/2] port link-aggregation group 1
[DeviceA-Gigabitethernet4/0/2] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –11
[DeviceA-Gigabitethernet4/0/2] quit
[DeviceA] interface gigabitethernet 4/0/3
[DeviceA-Gigabitethernet4/0/3] port link-aggregation group 1
[DeviceA-Gigabitethernet4/0/3] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
[DeviceA-Gigabitethernet4/0/3] quit
3. Configure Device A to perform load sharing based on source and destination
MAC addresses for link aggregation groups.
[DeviceA] link-aggregation load-sharing mode source-mac
destination-mac
Dynamic Link Aggregation
1. Create VLAN 10 and aggregate interface Bridge-aggregation 1, configure the
link aggregation mode as dynamic, and assign the aggregate interface to
VLAN 10.
<DeviceA> system-view
[DeviceA] vlan 10
[DeviceA-vlan10] quit
[DeviceA] interface bridge-aggregation 1
[DeviceA-Bridge-Aggregation1] link-aggregation mode dynamic
[DeviceA-Bridge-Aggregation1] port access vlan 10
[DeviceA-Bridge-Aggregation1] quit
2. Assign ports GE4/0/1 through GE4/0/3 to link aggregation group 1 and
VLAN 10 one at a time.
[DeviceA] interface gigabitethernet 4/0/1
[DeviceA-Gigabitethernet4/0/1] port link-aggregation group 1
[DeviceA-Gigabitethernet4/0/1] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
[DeviceA-Gigabitethernet4/0/1] quit
[DeviceA] interface gigabitethernet 4/0/2
[DeviceA-Gigabitethernet4/0/2] port link-aggregation group 1
[DeviceA-Gigabitethernet4/0/2] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
[DeviceA-Gigabitethernet4/0/2] quit
Troubleshooting HP Networks
3 –12 Rev 10.41
[DeviceA] interface gigabitethernet 4/0/3
[DeviceA-Gigabitethernet4/0/3] port link-aggregation group 1
[DeviceA-Gigabitethernet4/0/3] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
[DeviceA-Gigabitethernet4/0/3] quit
3. Configure Device A to perform load sharing based on source and destination
MAC addresses for link aggregation groups.
[DeviceA] link-aggregation load-sharing mode source-mac
destination-mac
Load Sharing Mode
1. Create VLAN 10.
<DeviceA> system-view
[DeviceA] vlan 10
[DeviceA-vlan10] quit
2. Create aggregate interface Bridge-aggregation 1, configure the source MAC-
based load sharing mode for the link aggregation group, and assign the
aggregate interface to VLAN 10.
[DeviceA] interface bridge-aggregation 1
[DeviceA-Bridge-Aggregation1] link-aggregation load-sharing mode
source-mac
[DeviceA-Bridge-Aggregation1] port access vlan 10
[DeviceA-Bridge-Aggregation1] quit
3. Assign ports GE4/0/1 and GE4/0/2 to link aggregation group 1 and VLAN
10.
[DeviceA] interface gigabitethernet 4/0/1
[DeviceA-Gigabitethernet4/0/1] port link-aggregation group 1
[DeviceA-Gigabitethernet4/0/1] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
[DeviceA-Gigabitethernet4/0/1] quit
[DeviceA] interface gigabitethernet 4/0/2
[DeviceA-Gigabitethernet4/0/2] port link-aggregation group 1
[DeviceA-Gigabitethernet4/0/2] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –13
[DeviceA-Gigabitethernet4/0/2] quit
4. Create aggregate interface Bridge-aggregation 2, configure the destination
MAC-based load sharing mode for the link aggregation group, and assign the
aggregate interface to VLAN 10.
[DeviceA] interface bridge-aggregation 2
[DeviceA-Bridge-Aggregation2] link-aggregation load-sharing mode
destination-mac
[DeviceA-Bridge-Aggregation2] port access vlan 10
[DeviceA-Bridge-Aggregation2] quit
5. Assign ports GE4/0/3 and GE4/0/4 to link aggregation group 2 and VLAN
10.
[DeviceA] interface gigabitethernet 4/0/3
[DeviceA-Gigabitethernet4/0/3] port link-aggregation group 2
[DeviceA-Gigabitethernet4/0/3] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
[DeviceA-Gigabitethernet4/0/3] quit
[DeviceA] interface gigabitethernet 4/0/4
[DeviceA-Gigabitethernet4/0/4] port link-aggregation group 2
[DeviceA-Gigabitethernet4/0/4] port access vlan 10
Warning: This port is a member of the link aggregation group. If
configuration of the whole group is required to be modified, please
configure it under the aggregation interface view. Otherwise, this
operation may interrupt network traffic.Continue?[Y/N]: y
[DeviceA-Gigabitethernet4/0/4] quit
Troubleshooting HP Networks
3 –14 Rev 10.41
LACP – Link Aggregation Control Protocol Link Aggregation Control Protocol (LACP) is another option for creating ―port trunk
groups‖ on HP switches. LACP is defined by the IEEE standard 802.3ad. LACP was
standardized to allow a switch to automatically recognize coterminous, full duplex,
same-speed links between itself and another LACP-compliant switch.
Although LACP can automatically recognize links that are capable of aggregation,
the activation of an LACP trunk requires some configuration. You can’t simply connect
four links between the same two switches and expect them to act as a trunk.
When using dynamic LACP, you must define the trunk on one side, which is known
as the ―active‖ side. The active side sends Bridge Protocol Data Units (BPDUs) across
every link that has LACP defined statically.
Although a complete description of the fields in the BPDU is beyond the scope of this
course, a few BPDU fields relevant to dynamic operation are worth noting. They are:
A system identifier, which is the switch’s MAC address.
A priority value, which is a permutation of the MAC address.
A port identifier, which contains a port number.
When a switch receives BPDUs through multiple ―passive‖ LACP ports that have the
same system identifier, it knows that those ports are linked to the same switch. If the
links are the same speed, the switch sends BPDUs to the ―active‖ partners on the
other side of the links, and the two switches agree to load share across the group of
links. Passive LACP ports only ―speak‖ when ―spoken‖ to; a passive LACP port sends
BPDUs only after it has received BPDUs from a connected switch.
Configurable LACP States HP switches offer three possible options for LACP configuration:
Passive
Active
Disabled - (default state)
LACP is configured on a per-port basis. When a port is configured for a passive
LACP state, it will be blocked for approximately five seconds when the switch is
initialized. This is appropriate for ports that are linked to active LACP partners
because it provides the ports with time to discover the LACP topology before
forwarding any traffic. However, this delay can be unacceptable for normal switch
operation.
Consequently, HP recommends that LACP remain in the default state of disabled for
all ports that will not participate in dynamic link aggregation.
If you define a trunk using the trunk command described earlier in this module, the
no lacp command is automatically executed and included in the configuration for
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –15
the ports specified in the trunk command’s port list. Static and dynamic port trunking
cannot be simultaneously active on the same port.
Finally, is the case of 802.1X (Port-Based Access Control) being configured on a Port.
To maintain security, LACP is not allowed on ports configured for 802.1X
authenticator operation. If you configure port security on a port on which LACP
(active or passive) is configured, the switch removes the LACP configuration, displays
a notice that LACP is disabled on the port(s), and enables 802.1X on that port.
Static vs. Dynamic Link Aggregation One important advantage of dynamic link aggregation is its ability to recognize and
use trunk standby links. When two switches detect more than four coterminous, same
speed links, they aggregate the four links with the lowest port numbers. The
remaining links are used as standby links.
While dynamic LACP is the only way to set up standby links in a trunk, its
disadvantage is that in certain circumstances it can give you less control.
The primary disadvantage of static link aggregation is its lack of support for standby
links. Switches configured for static link aggregation cannot automatically detect new
members of the trunk group and, therefore, cannot use standby links.
On the other hand, static aggregation enables administrators to retain more control
of the operation of the trunk ports.
Troubleshooting HP Networks
3 –16 Rev 10.41
Spanning Tree
Figure 3.1: Spanning tree
Multiple Spanning Tree Protocol (MSTP) enables the configuration of VLAN-aware
Spanning Tree topologies. As described in IEEE 802.1S, multiple spanning trees
allow frames assigned to different VLANs to follow different data routes within
administratively established regions of the network.
In this way, MSTP enables the configuration of Multiple Spanning Trees within a
physical topology, which provides significant improvement in the utilization of
redundant links. Furthermore, the standard notes that an MST configuration probably
will provide simple and full connectivity for frames even in the presence of
administrative errors in the allocation of VLANs to Spanning Trees.
MSTP should not be confused with another VLAN-aware Spanning Tree protocol
known as Per VLAN Spanning Tree (PVST). In PVST configurations, a separate
Spanning Tree instance is created for each VLAN. BPDUs are transmitted with tags
that identify the STP instance and VLAN ID to which they belong. While this enables
the use of redundant links if you apply priorities and costs intelligently, it can be a
CPU-intensive process if there are many VLANs.
MSTP, on the other hand, enables the creation of multiple Spanning Tree instances
that are specifically mapped to VLANs. It is not necessary to literally have a one-to-
one correspondence between Spanning Trees and VLANs. In this way, MSTP
combines the best of two extremes—the single Spanning Tree configurations of STP
and RSTP and the Spanning Tree per VLAN configuration of PVST.
MSTP Features
MSTP is the default protocol when Spanning Tree is enabled
MSTP allows for multiple instances of a redundant path for a set of VLANs within
the bridged network
Each Spanning Tree instance has its own Root Bridge
Traffic is distributed across redundant links
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –17
MSTP follows the same basic principles as STP
and RSTP
Compatible and interoperable with STP and RSTP
Emulates STP and RSTP behaviors when encountering switches that do NOT
support MSTP
Because MSTP implements the same basic principles as the earlier Spanning Tree
protocols, it is completely interoperable and compatible with STP and RSTP.
Furthermore, MSTP will emulate STP and RSTP behaviors when encountering devices
that do not support MSTP.
MSTP is the latest iteration of Spanning Tree, and is the default Spanning Tree
protocol on most switches. Check the release notes or manuals for a specific switch to
determine its default.
Comparing RSTP, PVST and MSTP
Table 2.1: Comparing RSTP, PVST and MSTP
Before the release of the MSTP standard, the only IEEE-standardized way to combine
VLANs and Spanning Tree was to resolve loops within the topology without regard to
VLAN configuration.
Cisco Systems Inc. developed PVST—and later PVST+—to enable the configuration of
VLAN-aware Spanning Trees. PVST enables administrators to configure Bridge and
Port Priority settings and path costs so that any two paths between a pair of switches
can both be used. With PVST enabled, some Spanning Tree instances will take one
path while other instances take another path. However, each of the Spanning Tree
instances is separately configured, which results in more overhead than the simpler
Troubleshooting HP Networks
3 –18 Rev 10.41
RSTP solution. Furthermore, the scalability of PVST is limited because of the increased
CPU utilization described earlier in this module.
MSTP, on the other hand, enables the configuration of fewer Spanning Tree
instances, typically between 1 and 16, with each VLAN mapped to the appropriate
instance.
Spanning Tree for Instance 1
Figure 3.3: Multiple spanning tree (1)
With MSTP, Spanning Tree instances are associated with VLAN IDs, not with
individual links. Because a separate Root Bridge is elected for each MST instance,
each instance uses a different set of links as the active path.
As with STP and RSTP, backup—or Blocking State—ports are not used in the primary
active path, but they enable the quick restoration of connectivity in the event of link
failure.
In the graphic above, Edge_1 was elected as the Root Bridge for MST Instance 1,
which resulted in the topology shown. Instance 1 includes VLANs 2 to 10. The next
slide illustrates the Spanning Tree topology for MST Instance 2.
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –19
Spanning Tree for Instance 2
Figure 3.4: Multiple spanning tree (2)
In the diagram above, Edge_2 has been elected as the Root Bridge for MST Instance
2. Instance 1 includes VLANs 11 to 20.
Because of this election, the state of the physical links is different than in MST
Instance 1, shown on the previous slide.
MST Regions
A group of switches that collectively define multiple Spanning Tree instances is
known as an MST region
Each switch can belong to only one region
All switches in a region must have identical configuration attributes:
Alphanumeric configuration name
Configuration revision number
Associations between VLANs and Spanning-Tree instances
A switch defines a region boundary if it receives BPDUs from:
A switch with different configuration attributes, or
An STP or RSTP switch
MST Instances Within a Single Switch
When MST is initially enabled, the default conditions are as follows:
Troubleshooting HP Networks
3 –20 Rev 10.41
Each switch defines its MAC address as its configuration name and ―0‖ as
its configuration revision number
All of the VLANs defined on a switch belong to the Internal Spanning Tree
(IST) instance
To cause the switch to interact correctly with other switches in the MST region,
you must define common configuration attributes
Any VLAN not explicitly mapped to a user-defined instance remains associated
with the IST
VLAN 1 is often associated with the IST
Immediately after MSTP is enabled, all the VLANs configured on a switch are part of
the Internal Spanning Tree (IST), which is an RSTP instance that exists within the MST
region. As you add new instances and associate them with VLANs, the VLANs are
removed from the IST. However, the IST remains in place, even if no VLANs are
explicitly mapped to it.
In most cases, user-defined VLANs are associated with user-defined instances
configured identically on all switches in the MST region. The default VLAN (VLAN ID
1) remains associated with the IST. This provides an important benefit: if the VLAN-to-
instance mappings are misconfigured, you can still access the switch because the
IST’s association with VLAN 1 ensures that connectivity is not completely disrupted.
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –21
Basic IRF Concepts
Figure 3.5: IRF concepts
The devices that form an IRF virtual device are called IRF member devices. A member
device assumes the role of master or slave. An IRF stack contains only one master,
which manages the IRF virtual device. All other members operate as slaves and as
backups for the master. When the master fails, the IRF virtual device automatically
elects a new master from one of the slaves. Master and slaves are selected through
the role election mechanism. The details of the role election mechanism will be
covered later in this module.
A logical IRF port is a logical port dedicated to the internal connection of an IRF
virtual device. These ports cannot act as access, trunk or hybrid ports. An IRF port is
effective only when it is bound to a physical IRF port.
Physical ports used for connecting members of an IRF virtual device are called
physical IRF ports. Typically, an Ethernet port or optical port forwards frames to the
network. When a physical port is bound to an IRF port, it acts as a physical IRF port
and forwards data traffic such as IRF-related negotiation frames and data traffic
among members.
As shown in the figure above, an IRF stack can have a daisy chain topology or a
ring topology. A ring connection is more reliable than the daisy chain connection. In
a daisy chain topology, the failure of one link can cause the IRF virtual device to
partition into two independent IRF virtual devices, which can disrupt connectivity as
well as IRF functioning. The failure of a link in a ring connection results in a daisy
chain connection, and does not affect IRF services.
Troubleshooting HP Networks
3 –22 Rev 10.41
IRF application scenario: Increasing port density
Figure 3.6: IRF increases port density
IRF provides a simple, cost-effective solution to the issues that arise when use
population exceeds the available network ports. With IRF deployed, you can add
new members to your virtual IRF device, adding port density with minimal
configuration of the new switches.
IRF application scenario: Expanding system processing
capabilities
Figure 3.7: IRF expands system processing capabilities
When the forwarding capability of the core switch cannot satisfy users’ needs, you
can add a switch to form an IRF stacking system with the original core switch. If the
forwarding capability of one switch is 64 Mpps, the forwarding capability of the
whole stack system is 128 Mbps after another switch is added. Note that this
increases the forwarding capability of the entire stacking system, not a single switch.
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –23
IRF application scenario: Expanding bandwidth
Figure 3.8: IRF expands bandwidth
You can increase the uplink bandwidth of an edge switch by adding another switch
to form a stacking system with the existing edge switch. You can configure multiple
physical links of the member devices as an aggregation group to increase the
bandwidth of the link to the core switch. In the IRF configuration in the above Figure ,
four links (two from each switch) are aggregated to double the bandwidth from the
edge to the core. Adding a second edge switch without IRF would add more
throughput to the core, but the bandwidth would be divided between the edge
switches and their corresponding clients. To the core switch, the number of edge
switches does not change. The original edge switch will back up the current
configurations to the newly added switch in batches, having minimal effect on
network planning and configuration.
How IRF simplifies networks
Figure 3.9: IRF simplifies networks (1)
Troubleshooting HP Networks
3 –24 Rev 10.41
This network topology provides redundant links between the edge and the
distribution layer. MSTP is required to prevent loops introduced by these redundant
links.
VRRP is a protocol for providing router redundancy. For each of the two segments in
the configuration shown, one router in the distribution layer acts as the master and
does the actual routing and the other acts as a backup. If the master fails, the
backup can take over the routing. In enterprise networks, VRRP is often combined to
add Layer 3 redundancy to the Layer 2 redundancy provided by MSTP.
Figure 3.10: IRF simplifies networks (2)
In this solution, all four of the distribution layer switches are combined into one IRF
stack. All of the switches have the same routing table and can route packets received
from the edge switches. The IRF master will run the routing protocol for the entire
virtual device.
When configured as an IRF stack, the distribution layer switches now act as a single
virtual switch. Loops can still occur, however between an edge switch and the IRF
virtual switch. In order to retain the redundant links between the edge and
distribution layers, the redundant links can be combined in a link aggregation,
creating a single logical link that spans two physical devices in the IRF virtual switch.
Advantages of this topology The IRF topology is simpler to configure and maintain
than the MSTP/VRRP solution. In the IRF implementation, the virtual switch is
configured as if it were a single device. If the same switches were running MSTP and
VRRP, each switch would need a distinctly different configuration to ensure the correct
election of MSTP Root Bridge and VRRP Master. Furthermore, each switch would
need to be configured separately for all routing and switching functions.
Architecture: Operational Planes
Plane Functions
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –25
Management
Management interfaces (console, Telnet, SNMP, FTP,
TFTP, etc.) Internal/hardware monitoring: temperature, fan status,
module and power management, etc. File system including: Configuration File
Control
Layer 2 protocols: LACP, RSTP, MSTP Layer 3 Protocols: RIP, OSFP, BGP, ISIS, etc. Routing Table ACLs and QoS Policies
Forwarding FIB (Forwarding Information Base) and Local ACLs and
QoS Policies Frame/packet forwarding and handling
Modern Switches and Routers segregate their functions into different groups called
―operational planes‖ or simply ―planes‖.
Operational Planes in Standalone Switches
Figure 3.11: Operational planes in standalone switches
Modern Switches and Routers segregate their functions into different groups called
operational planes or simply planes.
The most common planes are:
Control Plane: this group includes all internal monitoring and control functions
related to power, temperature, and hardware state in general.
Management plane: this functional group is where the user interface is located
and where and all protocols run, for example STP in Layer 2 and OSPF in layer
3.
It is in this plane that the routing table is built.
Functions in this plane are software based to allow for upgrades.
Forwarding Plane: this group of functions includes L2 and L3 forwarding, packet
filtering and QoS policies.
Troubleshooting HP Networks
3 –26 Rev 10.41
It is in this plane that the routing table is actually used.
Functions in this plane are hardware based because of speed requirements.
Operational Planes in IRFv2
Figure 3.12: Operational planes in IRFv2
In stackable switches, the distribution of these planes is simple: a general purpose
CPU runs the management and control planes and one or two ASICs are in charge
of actual packet processing and forwarding.
In the case of chassis, the management and control plane are centralized in SRPUs
(Switching and Routing Processing Units) and the forwarding plane is distributed in
two or more LPUs (Line Processing Units). All chassis have the option of installing two
SRPUs for redundancy.
Operational Planes in IRFv2
When connecting several units to form an IRF, the management and control planes of
one of the units become active and those of the other units stay in standby.
In the case of chassis, today only two of them can be connected in an IRF. If each
has 2 SRPUs, one of these SRPUs is going to become active and the other three will
stay in standby.
In other words, an IRF system acts like a chassis with centralized management and
control planes and a distributed forwarding plane.
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –27
IRF-ports
Figure 3.13: IRF ports
To build an IRF-stack its member devices must be connected. This connection requires
the configuration of IRF-ports. An IRF-port is a logical entity composed of one or more
standard 10GbE ports. In other words, physical 10 GbE ports are bound to an IRF-
port.
By allowing the configuration of standard 10GbE ports as IRF ports, HP offers the
possibility of having:
Local IRF-stacks, in which all members are in the same room
Geographically distributed IRF-stacks
Important: IRF-port 1 can only be connected to IRF-port 2 of the next device in the
IRF-stack.
By allowing the configuration of regular 10GbE ports as IRF ports, H3C offers the
possibility of having:
Local IRF systems, in which all members are in the same room and
Geographically distributed IRF systems, for Data Center redundancy.
Local connections can be built using inexpensive copper cables:
with CX4 and XFP ports, CX4 local connection cables can be used
with SFP+ ports special IRF cables can be used
In both cases, cables of 50, 100 and 300cm are available.
For geographically distributed IRFs, the 10GbE technology required will depend on
the distance.
IRF Member ID
Devices forming an IRF-stack must have a different IRF Member ID. This number
is equivalent to the slot number in a chassis.
Troubleshooting HP Networks
3 –28 Rev 10.41
Switches A5120 and A5500 support dynamic Member ID allocation: when there
is a ―member-id collision‖ one of the devices changes it Member ID
automatically.
In all other A-Series switches the Member ID must be configured manually. This
step is the first step required when building an IRF-stack
By default: IRF Member ID = 1
Layer 2 (Data Link Layer) Troubleshooting and Problem Resolution
Rev 10.41 3 –29
Lab 4: VLAN Switching Lab 4 is design to ensure you can use a structured troubleshooting methodology to
resolve VLAN switching problems. There are three trouble tickets in this lab. Refer to
your lab guide for instructions on how to do this lab.
Troubleshooting HP Networks
3 –30 Rev 10.41
This page left blank intentionally.
Rev 10.41 4 –1
Layer 3 (Network Layer) Troubleshooting and
Problem Resolution Module 4
In this module, various layer 3 technologies will be reviewed and common problems
will be discussed.
The technologies include:
IPv4 Routing and Addressing
Inter-VLAN Routing
VRRP
OSPF
iBGP/eBGP
NAT
Troubleshooting HP Networks
4 –2 Rev 10.41
Forwarding between VLANs
Figure 4.1: forwarding between VLANs
As is shown in the example above, IP address 10.1.2.1 with the 24-bit mask
(255.255.255.0) defines a range of local IP addresses between 10.1.2.0 and
10.1.2.255. When using this mask, the first 24 bits of the IP address are recognized
as the "network" portion; the addresses of all the hosts in this range have the same
value in the network portion.
Layer 3 forwarding - host to router
Figure 4.2: Layer 3 forwarding
The router has traditionally been a tool for interconnecting networks. As a layer 3
device, it uses layer 3 information to make forwarding decisions and requires that
Layer 3 (Network Layer) Troubleshooting and Problem Resolution
Rev 10.41 4 –3
each interface leads to a different network. The diagram above illustrates layer 3
forwarding.
When Host 1 wants to talk to Host 2, it first determines whether Host 2 is local to its
own network. Host 1 uses its own IP address and mask to determine the range of
addresses that are local. In the example above, Host 2 is not in the same address
range as Host 1. The local range of Host 1 is 10.1.2.0 – 10.1.2.255.
Since the intended destination is remote, Host 1 sends the traffic to the MAC address
of its configured default gateway, which is a local router interface. All traffic
destined for address ranges other than the local network are directed toward the
default gateway. While Host 1 maintains an ARP cache that contains information
about local hosts, including the default gateway, it has no knowledge of layer 2
addresses on the other side of the router.
Layer 3 forwarding – router to host
Figure 4.3: Forwarding router to host
A router is not transparent to end stations; IP hosts are configured with a local
router's address as a default gateway and they send to the router all traffic destined
for hosts on other networks or subnetworks.
The router performs a lookup operation on the packet's destination IP address
against the entries in a routing table or cache. A successful lookup returns an
outbound interface.
The router performs an ARP cache lookup operation to resolve the layer 2 address of
the destination IP host. In the slide below, the destination host is on a network that is
directly connected to the router. If the destination network is not directly attached to
the router, it sends the packet to another router that leads toward the destination
network.
Troubleshooting HP Networks
4 –4 Rev 10.41
The router encapsulates the outbound IP datagram in a new layer 2 header and
forwards it to Host 2. Unlike the switched frame, which is forwarded without
modification, a routed frame is always changed by the router.
E-Series – Enable routing between VLANs
Figure 4.4: Routing between VLANs
To forward IP traffic between VLANs on the HP 5400zl switch, you need to add the
global configuration level command: ip routing. When you enable routing, the IP
addresses that are defined within the context of the VLANs are used as router
interfaces that provide default gateway service for end stations
The members of the VLANs may be tagged and/or untagged ports. Note that in the
diagram above two of the ports on the routing switch, ports C1 and C2, lead to
switches that support two port-based VLANs. Although these VLANs completely
overlap from the perspective of the 5400zl switch, they have two different IP
addresses; each VLAN has its own IP address that is within the range of the hosts in
that VLAN.
Also note that two ports on the switch, ports C3 and C4, lead to layer 2 switches
whose ports are all within the same VLAN. Although there are multiple physical
ports within this VLAN, there is only one IP address assigned to the group of ports.
All of the hosts within the address range 10.1.4.0/24 will use the same IP address
(10.1.4.1) as their default gateway.
Also note that, although none of the layer 2 switches have active ports in VLAN 1,
they do have an IP address within VLAN 1 for management purposes. The 2524
switches use VLAN 1 for management by default, called the primary VLAN.
A-Series devices have routing enabled by default.
Layer 3 (Network Layer) Troubleshooting and Problem Resolution
Rev 10.41 4 –5
VRRP Basics
Figure 4.5: VRRP basics
Basic default gateway redundancy operation
Common goals for default gateway redundancy methods:
Enable continuity for off-network communication despite the failure of the
primary default gateway
Provide for automatic failover from primary to backup default gateway
within typical session timeout intervals
Common technologies and implementation methods:
Routers use shared IP address (virtual address or interface on one router)
that is the default gateway address for hosts
Backup router takes over forwarding if Master router fails or is unavailable
VRRP: automatic failover for default gateway
Virtual Router Redundancy Protocol (VRRP) provides automatic failover for default
gateways
Specified in IETF RFC 3768
Enables load sharing in designs that coordinate VRRP and MSTP
Provides industry standard for default gateway provisioning
Implemented on all HP Networking E-Series ProVision ASIC switches
VRRP terminology review
A virtual router consists of a set of router interfaces on the same network that
share:
A virtual router identifier (VRID)
A virtual IP address
One router in the group becomes the VRRP Master; other routers are VRRP
Backup(s)
Troubleshooting HP Networks
4 –6 Rev 10.41
The VRRP Master router periodically sends advertisements to a reserved
multicast group address
VRRP Backup routers listen for advertisements and assumes Master role if
necessary
A VRRP router can support many virtual router instances, each with a unique
VRID/IP address combination
Client interacts with virtual router
Figure 4.6: Client interacts with virtual router
Hosts on VRRP-protected networks learn the default gateway’s virtual MAC
address from the Master’s via ARP request
Hosts send all off-network traffic to the local virtual MAC address without
knowing it is not a physical address
Automatic failover
Figure 4.7: Automatic fallover
If the Owner fails, the non-Owner (backup) begins forwarding traffic addressed
to the VRID 2 virtual MAC address (same as the Router 1 virtual MAC address)
Host does not require any configuration changes or session restarts
Host is unaware that a different router is forwarding its off-network traffic
Layer 3 (Network Layer) Troubleshooting and Problem Resolution
Rev 10.41 4 –7
OSPF Basics Benefits
Offers faster convergence than RIP
Scales to meet the needs of very large intranets
Characteristics
OSPF routers advertise the state of connected links
Flood advertisements to neighbors, who flood to other neighbors
Depends on router adjacency, formal relationship used to share routing
information
Intelligent path selection based on bandwidth-sensitive link costs
Divide large domain into smaller areas to enhance efficiency
Careful design can avoid router overload
As described in IP Routing Foundations, OSPF is a sophisticated routing protocol
designed to scale to meet the needs of very large enterprise networks. OSPF offers
several important advantages over the older Routing Information Protocol (RIP),
including faster convergence times as well as scalability.
OSPF uses hierarchical areas to enhance efficiency. By making sound decisions
when defining area borders, network designers can develop routing hierarchies that
scale readily without placing undue load on the routers.
This module will describe the design, deployment, and configuration of OSPF
networking using the E-Series ProVision ASIC switches.
Figure 4.8: OSPF basics
OSPF provides a hierarchical routing structure based on multiple areas
Backbone area (Area 0) required
Troubleshooting HP Networks
4 –8 Rev 10.41
Other area types include stub and NSSA
Router roles:
Area Border Router (ABR)
Autonomous System Boundary Router (ASBR)
As described in IP Routing Foundations, OSPF provides a hierarchical routing
structure that can scale to meet enterprise needs. The graphic, adapted from IRF,
illustrates some basic elements of the OSPF topology.
For more detail, consult IRF.
Enabling OSPF
Figure 4.9: Enabling OSPF
Before enabling OSPF on an IP router, it is advisable to statically define a Router ID.
If no Router ID is configured, the switch will assign one automatically. On the E-
Series ProVision ASIC switches, the choice of ID will depend on other configuration
items. Five possible cases are:
1. A single loopback interface and multiple VLANs with addresses
The loopback interface will be used as ROUTER ID.
2. A single loopback interface with multiple IP addresses
The lowest loopback IP address will be used as Router ID.
3. Multiple loopback interfaces with multiple IP addresses
The lowest loopback number and lowest loopback IP address will be used as
Router ID.
4. Multiple VLANs with a single IP Address in each VLAN
The IP address of the VLAN that becomes active first will be used as a Router ID.
Typically, on E-Series switches, the lowest number VLAN becomes active first.
Consequently, if an address is defined in VLAN 1, it will become the Router ID.
Enabling OSPF
Rev. 10.417
E5406_A
10.1.65.0/30 10.1.67.3024
5406zl_A(config)# router ospf
5406zl_A(ospf)# area 0
Enable OSPF on each VLAN and the loopback interface; area ID defaults to Area 0
Enable OSPF and create Area 0
5406zl_A(ospf)# vlan 10
5406zl_A(vlan-10)# ip ospf [area 0]
5406zl_A(vlan-10)# ip ospf passive
5406zl_A(vlan-10)# vlan 30
5406zl_A(vlan-30)# ip ospf
5406zl_A(vlan-30)# ip ospf passive
5406zl_A(vlan-30)# vlan 65
5406zl_A(vlan-65)# ip ospf
5406zl_A(vlan-65)# vlan 67
5406zl_A(vlan-67)# ip ospf
5406zl_A(vlan-67)# interface loopback 0
5406zl_A(lo-0)# ip ospf all
5406zl_A(config)# ip router-id 10.1.0.3 Define Router ID
Student VLAN 30
Server VLAN 10
Optionally, define stub networks as ―passive‖
Layer 3 (Network Layer) Troubleshooting and Problem Resolution
Rev 10.41 4 –9
If VLAN 1 is down, the switch will use the next lowest number VLAN IP address
as the Router ID.
5. Multiple VLANs with multiple IP addresses in each VLAN
The lowest IP address of the first active VLAN will be used as a Router ID. In
most cases, this will be a default VLAN IP address.
After the ID is defined, two separate commands are required to enable OSPF
globally on the E-Series ProVision ASIC switches. In the first, you simply enable OSPF
by issuing the router ospf command. In the second, you define at least one area.
To form adjacencies, which are fundamental to OSPF operation, two OSPF routers
must agree on an area ID, among other items.
Note that the configuration for the loopback interface must include an argument
specifying which IP addresses will be included in OSPF advertisements. In the
example on the previous page, ―all‖ indicates that all addresses will be included.
Alternatively, the administrator could specify any address configured on the interface
as this argument.
On the E-Series ProVision ASIC switches, configuration of OSPF at the global and
interface level is dynamic. Enabling OSPF on an interface may cause the router to:
1. Begin sending Hello packets through this interface in an effort to establish
adjacencies.
2. Include the network address range associated with this interface in its Router
LSA.
To minimize OSPF processing overhead, interfaces with no neighboring routers, such
as VLANs 10 and 30 in the example on the previous page, may be defined as
―passive.‖ The router does not send Hello messages over a passive interface, which
means it can never form an adjacency and will never send Link State Updates over
this type of interface.
Troubleshooting HP Networks
4 –10 Rev 10.41
Verifying OSPF status
Figure 4.10
After assigning each IP interface to an OSPF area, you can verify the status of
configured OSPF interfaces by issuing the show ip ospf interface command. In
the example shown in the figure above, only the backbone area has been defined,
and all interfaces are associated with the backbone area.
All of these interfaces were configured with default settings for authentication type,
cost, and priority. OSPF interfaces 10.1.10.1/24 and 10.1.30.1/24 were defined as
passive. The ―State‖ column indicates the relationship each OSPF interface has with
neighboring routers. Note that the passive interfaces have the Designated Router
state. The interfaces assume this role even though the router does not expect to find
neighbors on these networks.
This router has a neighbor on the network 10.1.65.0/30, which is indicated in the
output from the OSPF neighbor table. The entry in this table shows the neighbor’s
Router ID, its IP address on the network it shares with E5406_A, and the state of the
neighbor relationship. In this case, the neighbor is the Backup DR of the network
10.1.65.0/30. The next slide will provide more detail on the OSPF neighbors table.
Verifying OSPF Status
– View status of OSPF interfaces
– View state of the router’s neighbors
Rev. 10.418
5406zl_A(config)# show ip ospf interface
OSPF Interface Status
IP Address Status Area ID State Auth-type Cost Pri Passive
----------- --------- --------- ------ --------- ---- --- -------
10.1.0.3 enabled backbone LOOP none 1 1 no
10.1.10.1 enabled backbone DR none 1 1 yes
10.1.30.1 enabled backbone DR none 1 1 yes
10.1.65.2 enabled backbone DR none 1 1 no
10.1.67.2 enabled backbone DR none 1 1 no
5406zl_A(config)# show ip ospf neighbor
OSPF Neighbor Information
Router ID Pri IP Address NbIfState State Rxmt QLen Events
----------- ---- ---------- --------- ------ ---------- ------
10.1.0.1 1 10.1.65.1 BDR FULL 0 6
Layer 3 (Network Layer) Troubleshooting and Problem Resolution
Rev 10.41 4 –11
Viewing OSPF neighbor states
Figure 4.11: OSPF neighbor states
The figure on the previous page showed how information from the OSPF interface
and neighbor tables can be combined to learn the state the router interfaces on a
given network. In the figure above, the neighbor table from a different router,
E8212_A, which has three neighbors. Because all of E8212_A’s neighbors have
Router IDs that are higher than E8212_A’s Router ID, which is 10.1.0.1, all three
neighbors have assumed the role of Designated Router on their respective networks.
If you were to view the OSPF interface table, you would see that E8212_A has the
Backup DR state for the three networks that support its full adjacencies.
As shown, the neighbor table identifies each adjacent router by its Router ID and the
IP address on the interface where the adjacency has formed. The table also
indicates each neighbor’s priority and state. Use the OSPF neighbor table to
troubleshoot routing problems that may arise from the failure to form an adjacency.
Viewing OSPF neighbor states
– E8212_A has full adjacency with one
neighbor on each of the following
networks:
• 10.1.64.0/30
• 10.1.65.0/30
• 10.1.68.0/30
– With equal interface priorities, the OSPF
router with the highest router ID becomes
the Designated Router
Rev. 10.419
E8212_A(config)# show ip ospf neighbor
OSPF Neighbor Information
Router ID Pri IP Address NbIfState State Rxmt QLen Events
--------------- --- --------------- --------- -------- --------- ----------
10.1.0.2 1 10.1.64.2 DR FULL 0 6
10.1.0.3 1 10.1.65.2 DR FULL 0 6
10.1.0.4 1 10.1.68.2 DR FULL 0 7
10.1.65.0/30
E5406_A10.1.0.3
E5406_B10.1.0.4
E8212_A10.1.0.1
E8212_B10.1.0.2
10.1.64.0/30
10.1.67.0/30 10.1.68.0/30
10.1.66.0/30
Troubleshooting HP Networks
4 –12 Rev 10.41
External and internal Border Gateway Protocol
(BGP) BGP uses the Transmission Control Protocol (TCP) as its transport protocol, using port
179 for establishing connections. Running over a reliable transport protocol
eliminates the need for BGP to implement update fragmentation, retransmission,
acknowledgment, and sequencing.
The Internet is organized in a multitude of administratively independent networks
called domains or Autonomous Systems (AS). For example, an AS can be an Internet
Service Provider (ISP), a University campus or a corporate network.
The Border Gateway Protocol is an inter-Autonomous System routing protocol. The
primary function of a BGP speaking system is to exchange network reachability
information with other BGP systems. This network reachability information includes
information on the list of Autonomous Systems (ASs) that reachability information
traverses. This information is sufficient to construct a graph of AS connectivity from
which routing loops may be pruned and some policy decisions at the AS level may
be enforced
The route to each destination is called the AS path, and the additional route
information is included in path attributes. BGP uses the AS path and the path
attributes to completely determine the network topology, detect and eliminate routing
loops and it can enforce administrative preferences and routing policy decisions.
Contrasting eBGP and iBGP
BGP-4 provides a new set of mechanisms for supporting CIDR. These mechanisms
include support for advertising an IP prefix and they eliminate the concept of network
"class" within BGP. BGP-4 also introduces mechanisms which allow aggregation of
routes, including aggregation of AS paths.
Once BGP speakers are connected they exchange messages to start a BGP session
with a neighbor. This initial message identifies the sender’s AS number and BGP
identifier. Dependent upon whether the two speakers are in the same AS or different
will govern the session type. There are two basic session types for BGP, interior and
exterior.
While there are many similarities between exterior and interior BGP, the most
important difference is that the BGP speakers in an interior BGP peer session are in
the same AS. Interior BGP is used within a transit AS, as is shown in the diagram
below.
Layer 3 (Network Layer) Troubleshooting and Problem Resolution
Rev 10.41 4 –13
Figure 4-12: Contrasting eBGP and iBGP
Note that BGP routers at the "edge" of a domain will support both interior BGP
peers and exterior BGP peers.
BGP messages and route selection
Routers send open messages to each other to open or establish a BGP connection.
The two routers must first establish a TCP connection between them. After which the
sending of the Open Message is bidirectional.
Routers send Open Messages out and wait until they receive an Open Message from
their peer before continuing. Once the BGP peer is established, routers can
exchange routing information. This routing information is contained in Update
Messages
Once the connection has been established, the routers send incremental updates that
include summarized address ranges and AS numbers. (Messages vary somewhat
dependent upon whether they are between interior or exterior BGP speakers.) They
also send ―keepalives‖ to maintain the session. The router builds a graph or table of
the destinations and the attributes. BGP uses the AS or Autonomous System number
to select the shortest path to route data and avoid routing loops.
The two routers use UPDATE messages to add new routes, replace existing routes,
withdraw invalid routes, and communicate attributes.
BGP Notification Messages are an error message. The router selects the error type,
and puts it into the Notification Message and sends it to the peer. It then tears down
the peer connection
Notification Messages consist of multiple pieces, including the BGP header, error
code, error sub-code, and data that describes the error. This is important as it helps
the Notification Message recipient router to troubleshoot BGP peering problems
Troubleshooting HP Networks
4 –14 Rev 10.41
Network Address Translation (NAT)
Figure 4.13: Network address translation (NAT)
Network Address Translation (NAT) was originally created as a solution to the limited
number of public IP addresses. Internet Protocol version 4 (IPv4) uses four octets (32
bits) of address space—which does not provide enough IP addresses for the current
demand—and IPv6 is not yet widely implemented. NAT can provide an alternative
to obtaining a large block of registered addresses. With NAT implemented on the
network, a company does not need a public IP address for each of its computers.
NAT uses a device (a router, firewall, or computer) as an agent between the trusted
network and the untrusted network. When a packet destined for the untrusted
network reaches this device, the sender’s private IP address is translated into either
the company’s one public IP address or one of a limited range of such addresses
assigned to that company.
NAT also provides security: you give away nothing about your company’s internal
network if you use NAT when communicating with untrusted networks. The NAT-
enabled device adds an entry to its address translation table that maps the internal
address it replaced with the new public IP address. When the destination computer
sends a reply packet back through the router, the router uses the table to identify the
original internal IP address and sends the reply back to the appropriate computer on
the trusted network.
The following sections discuss the various types of NAT technology available. These
include single IP address translation, static NAT, dynamic NAT, Port Address
Translation (PAT), and NAT Traversal (NAT T).
Layer 3 (Network Layer) Troubleshooting and Problem Resolution
Rev 10.41 4 –15
Single IP Address Translation
Figure 4.12: Single address translation
Single IP address translation allows one public IP address to be used by a full IP
network. In this version of NAT, the available port numbers of the NAT-enabled
gateway (router) are assigned to different private IP addresses. This allows multiple
simultaneous TCP/IP sessions to occur using only the router’s public IP address.
How It Works
When an internal computer sends a packet (containing the source IP address, source
port, destination IP address, and destination port), the packet must travel through the
NAT-enabled router. At this point, the router rewrites the packet header so that it
contains the router’s public IP address instead of the source IP address. The router
then encapsulates the package to send to its destination.
When the router rewrites the packet, it adds an entry into the address translation
table that maps the internal address it replaced with its own public IP address. When
the destination computer sends a reply packet back through the router, the router
identifies its original internal IP address from the address translation table and sends
the reply back to the appropriate computer. The above figure illustrates this process.
Troubleshooting HP Networks
4 –16 Rev 10.41
Static and Dynamic NAT
Figure 4.13: Static and dynamic NAT
Static NAT maps an internal IP address to a public IP address on a one-to-one basis.
That is, static NAT will always assign a particular computer the same public IP
address. For example, it will always assign the computer with IP address
192.168.45.10 the public IP address 213.18.121.110.
Dynamic NAT maps an internal IP address to a public IP address from a range of
public addresses assigned to that company. A computer on the trusted network is
dynamically assigned a random IP address depending on which addresses are
available at a given time. For example, NAT can assign a computer public IP
address 213.18.121.110 one time and then assign that same computer IP address
213.18.121.116 the next time that computer tries to send a packet to the untrusted
network.
Static NAT is particularly useful when a device needs to be accessible from outside
the network. Conversely, implementing dynamic NAT automatically creates a firewall
of sorts between a company’s internal network and untrusted networks: NAT only
allows connections that originate from the trusted network. Essentially, this means
that a computer in an untrusted network cannot connect to a computer in the trusted
network unless the trusted host initiates contact first.
Layer 3 (Network Layer) Troubleshooting and Problem Resolution
Rev 10.41 4 –17
Lab 5: Layer 3 Practice and Tools Lab 5 is design to ensure you can use the troubleshooting methodology and
troubleshooting tools. There are three trouble tickets in this lab. Refer to your lab
guide for instructions on how to do these labs.
Troubleshooting HP Networks
4 –18 Rev 10.41
Lab 6: OSPF Routing Issues Lab 6 is designed to ensure you can use a structured troubleshooting methodology to
resolve OSPF routing problems. There is one trouble ticket in this lab. Refer to your
lab guide for instructions on how to do this lab.
Layer 3 (Network Layer) Troubleshooting and Problem Resolution
Rev 10.41 4 –19
Lab 7: Addressing Issues Lab 7 is design to ensure you can use a structured troubleshooting methodology to
resolve IP addressing problems. There is one trouble ticket in this lab. Refer to your
lab guide for instructions on how to do this lab.
Troubleshooting HP Networks
4 –20 Rev 10.41
Lab 8: Inter-VLAN and Routing Lab 8 is design to ensure you can use a structured troubleshooting methodology to
resolve inter-VLAN routing issues. There is one trouble ticket in this lab. Refer to your
lab guide for instructions on how to do this lab.
Rev 10.41 5 –1
Layer 4 (Transport Layer) Troubleshooting and
Problem Resolution Module 5
This module focuses on troubleshooting at the transport layer 4. Upper layer
protocols such as TCP, UDP, HTTP, FTP and Telnet run on top of the IP layer 3.
Figure 5.1: The 5 layer IETF model
In the course the five Layer IETF model is used to describe a layered approach to
networking. The TCP/IP model consists of four Layers. Even though there are some
architectural differences, both models have interchangeable transport and network
layers and their operation is based upon packet-switched technology.
Troubleshooting HP Networks
5 –2 Rev 10.41
Troubleshooting TCP/UDP The Host-to-Host (Transport) Layer contains two protocols; Transmission Control
Protocol (TCP) and User Datagram Protocol (UDP). TCP and UDP are used to
transmit datagrams.
Figure 5.2: Contrasting TCP and UDP
Below is a description of major differences between TCP and UDP.
Reliable/Connection-Oriented
TCP is a connection-oriented protocol. When a file or message send it will get
delivered unless connections fails. If connection lost, the server will request the
lost part. There is no corruption while transferring a message.
Unreliable/connectionless
UDP is connectionless protocol. When you a send a datagram or message, you
don't know if it'll get there, it could get lost on the way. There may be
corruption while transferring a message
Ordered
Each message is sent with a sequence number, so that even if they arrive out of
order, they can be reassembled in the correct order.
Not Ordered
If you send two messages out, and they arrive out of order, the application itself
would be responsible for reassembly in the proper order.
Heavyweight
When the low level parts of the TCP "stream" are lost, resend requests have to
be sent, and all the out of sequence parts have to be put back together, so
requires a bit of work to piece together.
Lightweight
No ordering of messages, no tracking connections, etc. This means it's a lot
quicker, and the network card / OS have to do very little work to translate the
data back from the packets.
Streaming
Layer 4 (Transport Layer) Troubleshooting and Problem Resolution
Rev 10.41 5 –3
Data is read as a "stream," with nothing distinguishing where one packet ends
and another begins.
Datagram
Packets are sent individually and are guaranteed to be whole if they arrive.
The TCP header occupies quite a large space in the Ethernet frame.
Figure 5.3: TCP message segment format
Source Port: 16 bits - The source port number.
Destination Port: 16 bits - The destination port number.
Sequence Number: The sequence number of first data octet in the segment
(except when SYN is present). If SYN is present the sequence number is the
initial sequence number (ISN) and the first data octet is ISN+1.
Acknowledgment Number: If the ACK control bit is set this field contains the
value of the next sequence number the sender of the segment is expecting to
receive. Once a connection is established this is always sent.
Data Offset: The number of 32 bit words in the TCP Header. This indicates
where the data begins. The TCP header (even one including options) is an
integral number of 32 bits long.
Reserved: 6 bits - Reserved for future use. Must be zero.
Flags: 6 bits and contains:
URG: Urgent Pointer field significant
ACK: Acknowledgment field significant
PSH: Push Function
RST: Reset the connection
SYN: Synchronize sequence numbers
FIN: No more data from sender
Troubleshooting HP Networks
5 –4 Rev 10.41
Window: 16 bits - The number of data octets beginning with the one indicated
in the acknowledgment field which the sender of this segment is willing to
accept.
Checksum: 16 bits
The TCP Length: The TCP header length plus the data length in octets (this is not
an explicitly transmitted quantity, but is computed).
Urgent Pointer: This field communicates the current value of the urgent pointer as
a positive offset from the sequence number in this segment. The urgent pointer
points to the sequence number of the octet following the urgent data. This field
is only be interpreted in segments with the URG control bit set.
Options: Options may occupy space at the end of the TCP header and are a
multiple of 8 bits in length. All options are included in the checksum. An option
may begin on any octet boundary. There are two cases for the format of an
option:
A single octet of option-kind.
An octet of option-kind, an octet of option-length, and the actual option-
data octets.
The option-length counts the two octets of option-kind and option-length as well
as the option-data octets. Note that the list of options may be shorter than the
data offset field might imply.
Data: variable - The actual user data is included after the end of the header
To troubleshoot TCP and UDP it is often necessary to analyze TCP segments using a
network analyzer tool such as Wireshark. The TCP Packet capture shown in the
figure below is a request-response message sequence carried over TCP.
Notice the fields discussed above: Source Port, Destination Port, Sequence number,
Window size, Flags, Checksum and options.
Layer 4 (Transport Layer) Troubleshooting and Problem Resolution
Rev 10.41 5 –5
Figure 5.4: TCP packet capture
UDP does not ensure that the data bytes sent will arrive at the other site. Thus, UDP
imposes less network overhead than TCP.
Source Port: The 16-bit port number of the process that originated the UDP
message on the source device. This will normally be an ephemeral (client) port
number for a request sent by a client to a server, or a well-known/registered
(server) port number for a reply sent by a server to a client.
Destination Port: The port number of the process that is the ultimate intended
recipient of the message on the destination device. This will usually be a well-
known/registered (server) port number for a client request, or an ephemeral
(client) port number for a server reply.
Length: The length of the entire UDP datagram, including both header and Data
fields.
Checksum: An optional checksum computed over the entire UDP datagram plus
a special “pseudo header” of fields. See below for more information.
Data: The encapsulated higher-layer message to be sent.
Troubleshooting HP Networks
5 –6 Rev 10.41
Figure 5.5: UDP message segment format
Below is a picture of a packet capture of the UDP section of the Ethernet frame. Note
that the UDP packet capture shows the Source port, Destination port, Length and
Checksum
Figure 5.6: UDP packet capture
Layer 4 (Transport Layer) Troubleshooting and Problem Resolution
Rev 10.41 5 –7
Firewalls Layer 4 protocols are subject to packet filters and firewalls. It is possible to have IP
connectivity between the network components but certain packets are unable to
traverse between a source and destination address. These types of connectivity
issues may cause by problems with:
Firewalls
Packet filters
Servers
Authentication and authorization
Application software interoperability
Operating system interoperability
In this section we are going to look at troubleshooting firewall and packet filter
issues.
Firewall configurations
You have many options when deciding where or how to implement your firewall. The
configuration typically includes a combination of routers, gateways, and servers on
the edge of a trusted network. Firewalls can be configured in (but are not limited to)
the following architectures shown in the picture below.
Figure 5.7: Firewall configurations
Troubleshooting HP Networks
5 –8 Rev 10.41
Denying or permitting packets
A firewall is a collection of components configured to enforce a specific access
control policy between your internal (trusted) network and any other (untrusted)
network. As the above figure shows, a firewall protects your company’s internal
network from the Internet.
A firewall filters incoming and outgoing packets to ensure only authorized packets
pass. You must set up a clearly defined security policy that delineates authorized
traffic. For example, you can configure rules in which the firewall drops packets from
specific untrusted servers that you identify by IP address.
Essentially, you can use one of two principles when implementing rules for your
company’s firewall:
Deny everything except that which is explicitly permitted
Permit everything except that which is explicitly denied
Layer 4 (Transport Layer) Troubleshooting and Problem Resolution
Rev 10.41 5 –9
Firewall types Firewalls fall into one or more of the following categories:
Packet-filtering firewall:
Must establish a predefined table of rules against which a packet-filtering
firewall compares the full association of the packets.
Must specify which packets should be accepted and which denied.
Can create rules that will drop packets from specific untrusted servers, which you
identify by IP address.
Can also create rules that permit particular types of connections (such as FTP
connections) only if they are using the appropriate trusted servers (such as the
FTP server).
Circuit-level gateway
Acts as a proxy server to establish a circuit with the internal computers.
All outgoing packets from the trusted clients appear to have the proxy server’s
source IP.
After a connection is established, the circuit-level gateway simply copies and
forwards packets back and forth without filtering them further.
Application-level gateway
Acts as a proxy server between a trusted client and an untrusted host.
Only accept packets generated by services they are designed to copy, forward,
and filter.
For example, only a telnet proxy can copy, forward, and filter telnet traffic.
Stateful-inspection firewall
Combines all of the above.
Filtering all incoming and outgoing packets based on source and destination IP
addresses and port numbers.
Ensuring packets in a session are appropriateEvaluates the contents of each packet
up through the application layer and ensures that these contents match the rules
in your company’s network security policy.
Algorithms compare packets against known bit patterns of authorized packets.
Troubleshooting HP Networks
5 –10 Rev 10.41
Table 5.1: Contrasting firewall types
Few firewalls belong in only one of these categories, and fewer still exactly match the
definition for any one category. These categories, however, do reflect the key
capabilities that differentiate one firewall from another.
Figure 5.8: Stateful-inspection firewalls
In a specific firewall implementation, the various types can be combined to create
complex, sophisticated solutions. For example, a dual-homed host can be either a
circuit-level gateway or an application-level gateway. A screened subnet includes at
least two packet-filtering firewalls.
Layer 4 (Transport Layer) Troubleshooting and Problem Resolution
Rev 10.41 5 –11
Network address translator (NAT) There are various types of NAT technology available. These include
Single IP address translation
Static NAT and dynamic NAT
Port Address Translation (PAT)
NAT Traversal (NAT T)
Network address translation (NAT) was discussed in an earlier module. This module
extends this discussion to include Port Address Translation (PAT).
Often, a company’s global address pool does not contain enough public IP
addresses to ensure all hosts in the trusted network can be mapped to an Internet
address when they need to be. In this situation, the company should implement Port
Address Translation (PAT). PAT maps each host in the trusted network to a global IP
address and also to a unique TCP or UDP port number on the NAT-enabled router.
In this way, PAT can map the same global IP address to a number of private IP
addresses; it uses the unique port number to distinguish between them.
Figure 5.9: Port address translation (PAT)
The router stores the original IP address and port against the new IP address and
port in the address translation table. When the destination computer on the
untrusted network sends a reply packet back through the router, the router identifies
the recipient on the trusted network using the address translation table and routes the
packet appropriately.
Enabling PAT NAT
PAT must be enables when you configure NAT, that is:
Configure a basic or advanced ACL for each range of private addresses for
which you want to provide NAT.
Configure a pool for each consecutive range of Internet addresses to which you
want NAT to be able to map the private addresses specified in the ACLs. Each
pool must contain a range with no gaps. If your Internet address space has
Troubleshooting HP Networks
5 –12 Rev 10.41
gaps, configure separate pools for each consecutive range within the address
space.
Associate a range of private addresses (specified in a basic or advanced ACL)
with a pool.
Enable the Port Address Translation feature if you have more private addresses
that might need NAT than the Internet address pools contain.
Enable outbound NAT on the interface connected to global addresses. The following
commands configure a basic ACL for the private subnet 10.10.10.x/24, then enable
inside NAT for the subnet. This example has Port Address Translation Enabled.
# acl number 2001
rule permit source 10.10.10.0 0.0.0.255
#
nat address-group 1 209.157.1.2 209.157.1.254
)# interface Serial 5/0
nat outbound 2001 address-group 1
Rev 10.41 6 –1
Layer 5 (Application Layer) Troubleshooting and
Problem Resolution Module 6
In this module, the common issues around troubleshooting application layer 5 issues
will be reviewed and common problems will be discussed. The most common
application layer problems evolve around QoS. The focus of this module is therefore
around QoS.
Troubleshooting HP Networks
6 –2 Rev 10.41
QoS process flow QoS attributes include:
QoS parameters
Congestion
QoS mechanisms
Switch QoS configurations
Traffic handling techniques
QoS policies
QoS parameters
The reason that networks exist is to enable users to access and run their
applications. Applications include web browsing, file transfers, video streaming,
email exchange, and voice conversations. These applications have different
Quality of Service requirements, where Quality of Service defines the level of
service that the application requires from the network. Quality of Service (QoS)
parameters may include minimum data rates, packet error rates, jitter and latency.
When making a QoS scheme, a network administrator must consider the
characteristics of various applications to balance the interests of diversified users
and fully utilize network resources.
In addition, enterprises today are experiencing increased voice and video traffic
over their networks, and many have fully migrated their voice traffic from a
separate PBX network to run over their IP networks. Voice and video have different
network requirements such that the voice and video quality will be perceived
acceptable by the users:
Video and voice are both sensitive to jitter.
The variation in intervals between the arrival of packets.
Can cause dead spots in real-time transmission.
Voice is sensitive to delay, sometimes called “latency”.
Relates to the amount of time that passes between the sending of a
transmission and its arrival at the receiving station.
Switches and routers can be configured to support these QoS needs.
Congestion
When the rate at which traffic arrives at a device exceeds the rate that the
devicecan forward the traffic on a specific interface then congestion occurs. As
such the interface that forwards packets is a basic network resource.
TCP applications such as web browsing can tolerate congestion
Layer 5 (Application Layer) Troubleshooting and Problem Resolution
Rev 10.41 6 –3
Acknowledgement and flow-control mechanisms
Lost packets retransmitted
Back off procedure when congestion is detected
UDP applications such as voice and video are more susceptible
No acknowledgement or flow control at the transport layer
Applications may provide acknowledgement and flow control
No back off when congestion is detected
Single application might monopolize link
Queuing processes
Congestion management uses queuing and scheduling algorithms to classify and
sort traffic leaving a port. Each queuing algorithm addresses a particular network
traffic problem, and has a different impact on bandwidth resource assignment,
delay, and jitter. Queue scheduling processes packets by priority, and
preferentially forwards high-priority packets. Queuing processes include:
Strict Priority (SP) queuing.
SP queuing is specially designed for mission-critical applications, which must be
served first to reduce response delays when congestion occurs. SP queuing
classifies eight queues on an A-Series switch port into eight classes, numbered 7
to 0 in descending priority order.
SP queuing schedules the eight queues strictly according to the descending
order of priority. It sends packets in the queue with the highest priority first.
When the queue with the highest priority is empty, it sends packets in the queue
with the second highest priority, and so on. Thus, you can assign mission-critical
packets to the high priority queue to ensure that they are always served first and
common service packets to the low priority queues and transmitted when the
high priority queues are empty.
The disadvantage of SP queuing is that packets in the lower priority queues
cannot be transmitted if there are packets in the higher priority queues. This may
cause lower priority traffic to be starved and never be transmitedt.
Weighted Round Robin (WRR) queuing
WRR queuing schedules all the queues in turn to ensure that each can be served
for a certain time. Assume there are eight output queues on a port. WRR assigns
each queue a weight value (represented by w7, w6, w5, w4, w3, w2, w1, or
w0) to decide the proportion of resources assigned to the queue. On a 100
Mbps port, you can configure the weight values of WRR queuing to 5, 5, 3, 3,
1, 1 1, and 1 (corresponding to w7, w6, w5, w4, w3, w2, w1, and w0
respectively). In this way, the queue with the lowest priority is assured, thus
avoiding the disadvantage of SP queuing that packets in low-priority queues
may fail to be served for a long time.
Troubleshooting HP Networks
6 –4 Rev 10.41
Another advantage of WRR queuing is that while the queues are scheduled in
turn, the service time for each queue is not fixed, that is, if a queue is empty, the
next queue will be scheduled immediately. This improves bandwidth resource
use efficiency.
Weighted Fair Queuing (WFQ)
The only difference between WFQ and WRR is that: WRR schedules certain
number of packets from a queue in each cycle of scheduling, while WFQ
schedules certain number of bytes from a queue in each cycle of scheduling.
Additionally, WFQ can work with the minimum guaranteed bandwidth
mechanism. You can configure a minimum guaranteed bandwidth for each
WFQ queue, so that each WFQ queue is guaranteed of the bandwidth when
congestion occurs. The assignable bandwidth (total bandwidth – the sum of the
minimum guaranteed bandwidth for each queue) is allocated to queues based
on queue priority.
Because WFQ can balance delay and jitter among congested flows, it can be
applied in certain special scenarios. For example, WFQ is used for the assured
forwarding (AF) services of the Resource Reservation Protocol (RSVP). In Generic
Traffic Shaping (GTS), WFQ schedules buffered packets.
SP+WRR queuing.
By assigning some queues on the port to the SP scheduling group and the others
to the WRR scheduling group (group 1), you implement SP + WRR queue
scheduling on the port. Packets in the SP scheduling group are scheduled
preferentially. When the SP scheduling group is empty, packets in the WRR
scheduling group are scheduled. Queues in the SP scheduling group are
scheduled with the SP queue scheduling algorithm. Queues in the WRR
scheduling group are scheduled with WRR.
QoS mechanisms
QoS mechanisms enable network administrators to manage the use of network
resources, enabling mission critical applications to receive priority access to
network resources over lower priority traffic.
Traffic arriving at the device is separated into flows via a process referred to as
“Classification.”
Classification
Recognize traffic that should be prioritized
Assign an internal traffic class (internal forwarding priority)
The device maps priority values to its internal queues and forwards appropriately.
If transmitting host does not mark its own traffic, devices can apply policies to
inbound traffic
Marking
Layer 5 (Application Layer) Troubleshooting and Problem Resolution
Rev 10.41 6 –5
Indicates within the header how traffic should be handled – for the
benefit of other devices
Layer 2 marking: IEEE 802.1p
Layer 3 marking: IP Precedence or Differentiated Services Code
Point (DSCP)
Scheduling algorithms determine the packets and the rate of the packets that will
be forwarded on the interface.
Scheduling / traffic shaping
Place traffic in queues based on traffic class
Allocate sufficient percentage of outbound bandwidth for high priority
traffic
Figure 6.1: QoS mechanism
Switch QoS configuration
As such when configuring the switch to provide QoS to application traffic, you
must configure the QoS interface parameters including:
Classifications
Behavior
Number of queues
Traffic filtering
You can filter in or filter out a class of traffic by associating the class with a traffic
filtering action. For example, you can filter packets sourced from a specific IP
Troubleshooting HP Networks
6 –6 Rev 10.41
address according to network status. By using ACL rules configured with a time
range for traffic classification, you can implement time-based traffic filtering.
Class of Service (CoS) is:
The process of classifying traffic based on:
Layer 2: IEEE 802.1p
Layer 3: IP Precedence or DSCP
A classification method only
A tool used by scheduling (queuing) mechanisms to limit delay
To illustrate traffic filtering, below is an example configuration for a host connected
to interface GigabitEthernet 1/0/1 of the switch. The requirement is to configure
traffic filtering to filter the packets whose TCP source port number is 21 received on
the interface
# Create advanced ACL 3000, and configure a rule to match packets whose
source TCP port number is 21.
[DeviceA-acl-basic-3000] rule 0 permit tcp source-port eq 21
[DeviceA-acl-basic-3000] quit
# Create a class named classifier_1, and use ACL 3000 as the match criterion in
the class.
[DeviceA] traffic classifier classifier_1
[DeviceA-classifier-classifier_1] if-match acl 3000
[DeviceA-classifier-classifier_1] quit
# Create a behavior named behavior_1, and configure the traffic filtering action
for the behavior to
drop packets.
[DeviceA] traffic behavior behavior_1
[DeviceA-behavior-behavior_1] filter deny
[DeviceA-behavior-behavior_1] quit
# Create a policy named policy, and associate class classifier_1 with behavior
behavior_1 in the policy
[DeviceA] qos policy policy
[DeviceA-qospolicy-policy] classifier classifier_1 behavior behavior_1
[DeviceA-qospolicy-policy] quit
# Apply the policy named policy to the incoming traffic on interface
GigabitEthernet 1/0/1.
[DeviceA] interface gigabitethernet 1/0/1
Layer 5 (Application Layer) Troubleshooting and Problem Resolution
Rev 10.41 6 –7
[DeviceA-GigabitEthernet1/0/1] qos apply policy policy inbound
Troubleshooting HP Networks
6 –8 Rev 10.41
802.1p traffic prioritization Traffic handling techniques generally involve the host sending time-sensitive traffic
modifies bits in packet header. Examples of protocols that do this include:
DSCP or IP Precedence: Priority field within IP datagram header
IEEE 802.1p: Priority field within 802.1Q tag
802.1p is a layer 2 marking that is used in many LANs. 802.1p defines a field in
the MAC Ethernet header that carries one of eight priority values as shown in the
picture below.
IEEE 802.1p reserves a three-bit field in the 802.1Q tag
Some end stations set priorities for their traffic
Figure 6.2: 802.1p priority tag
Switches can retain or modify markers for prioritized traffic forwarded over tagged
links.
The table below provides an example of parameters that can be configured in a E-
Series switch.
Minimum percentages shown below are configurable per port
If all waiting traffic has the same priority level (e.g. normal) in a given time
period, 100% of the bandwidth is given to that traffic.
Layer 5 (Application Layer) Troubleshooting and Problem Resolution
Rev 10.41 6 –9
Table 6.1: Illustration of 802.1p switch settings
Configuring QoS policy
Switches can act as QoS policies enforcement points (PEP) to control access. PEPs
determine whether traffic can be admitted.
Figure 6.3: Configuring QoS
A QoS policy can be applied to:
An interface, the policy takes effect on the traffic sent or received on the
interface.
Troubleshooting HP Networks
6 –10 Rev 10.41
A user profile, the policy takes effect on the traffic sent or received by the online
users of the user profile.
A VLAN, the policy takes effect on the traffic sent or received on all ports in the
VLAN.
Globally, the policy takes effect on the traffic sent or received on all ports.
Default QoS example
The picture below shows an example of the E-Series switch supporting both a data
VLAN and a voice VLAN.
Figure 6.4: Default QoS on A-Series 3500 switches
Layer 5 (Application Layer) Troubleshooting and Problem Resolution
Rev 10.41 6 –11
Traffic marking by an end station Many IP phones mark their traffic for high-priority handling. In this illustration:
1. Phone marks priority level in IEEE 802.1Q tag
2. The edge switch
a. Classifies traffic based on priority marker in tag
b. Schedules packet for delivery by placing it in queue associated with
traffic class
Figure 6.5: IP phone illustration
Display the QoS policy applied to VLAN
Below is an example of how to display the parameters of VLAN 2.
# show qos vlan-policy vlan 2
VLAN priorities
VLAN ID Apply rule | DSCP Priority
------- ----------- + ------ -----------
1 No-override | No-override
500 No-override | No-override
Troubleshooting HP Networks
6 –12 Rev 10.41
Retaining priority between VLANs Continuing the previous example:
3. The core switch classifies traffic based on priority marker in tag
4. The core switch
a. Marks priority in 802.1p field of outbound packet‟s 802.1Q tag
b. Schedules packet for delivery by placing it in appropriate queue
c. Classifies and schedule delivery
Figure 6.6: Continuation of IP phone illustration
Configuring port priority
Below is an example of how port priorities can be set per VLAN or per interface.
# vlan 500
(vlan-500)# qos priority
(vlan-500)# qos
dscp Specify DSCP policy to use.
priority Specify priority to use.
(vlan-500)# qos priority
0
1
2
3
Layer 5 (Application Layer) Troubleshooting and Problem Resolution
Rev 10.41 6 –13
4
5
6
7
# interface A1
(eth-A1)# qos priority
(eth-A1)# qos
dscp Specify DSCP policy to use.
priority Specify priority to use.
(eth-A1)# qos priority
0
1
2
3
4
5
6
7
Troubleshooting HP Networks
6 –14 Rev 10.41
Normal priority data traffic This last illustration show the normal priority data traffic. In this example the edge
switch uplink (port 50) is a tagged member of VLAN 10; 802.1p field in tag contains
“0” value.
The steps are:
1. The user‟s data traffic is sent untagged, with no priority marker
2. The edge switch
a. Classifies the traffic as „normal‟
b. Then marks “0” value in 802.1p field of outbound packet‟s 802.1Q tag
c. Schedules packet for delivery, assigning it to the queue associated with
“normal” traffic
Figure 6.7: Normal priority data traffic
Layer 5 (Application Layer) Troubleshooting and Problem Resolution
Rev 10.41 6 –15
Lab 10: Quality of Service Lab 10 is designed to ensure you can use a structured troubleshooting methodology
to resolve Quality of Service problems. There is one trouble ticket in this lab. Refer to
your lab guide for instructions on how to do this lab.
Troubleshooting HP Networks
6 –16 Rev 10.41
This page left blank intentionally.
Rev 10.41 7 –1
Troubleshooting an End-to-End Complex,
Integrated Multi-Protocol Network Module 7
This module brings all the lessons from the previous modules and challenges you to
resolve a complex multi-protocol problem.
Stable network operations are critical to most enterprises. Failure of the network
results in productivity and revenue losses. Troubleshooting multiprotocol networks
can be complex and formidable, however following a structured approach diagnosis
and resolution can help resolve problems quickly and effectively.
In this lab you will solve a trouble ticket that has several problems. To do this lab,
you should use a structured approach to troubleshooting and document your steps.
Troubleshooting HP Networks
7 –2 Rev 10.41
Lab 11: Final lab Lab 11 is designed to ensure you can use a structured troubleshooting methodology
to resolve problems at multiple protocol layers. There is one trouble ticket in this lab
that contains several problems. Refer to your lab guide for instructions on how to do
this lab.
To learn more about HP Networking, visit
www.hp.com/networking
© 2010 Hewlett-Packard Development Company, L.P. The information contained herein is
subject to change without notice. The only warranties for HP products and services are set forth
in the express warranty statements accompanying such products and services. Nothing herein
should be construed as constituting an additional warranty. HP shall not be liable for technical
or editorial errors or omissions contained herein.