Solving Real-World Problems with Wireshark

Top-Down vs. Bottom-Up Approach

By Emil Prysak

IntroductionNOC (Network Operations Center) Engineer at Weight

WatchersResolving various networking problemsWireshark used frequentlyUsing top-down and bottom-up approaches depending on

the given symptomsTwo examples:

IP Address Loss Bottom-Up ApproachRightFax Email Delay Top-Down Approach

Overview of OSI ModelApplication Layer – file transfer, email, network

management, softwarePresentation Layer – data representation, encryption,

protocol conversionSession Layer – managing connections between

applicationsTransport Layer – error recovery, flow controlNetwork Layer – routing, switching, packet sequencing, IP

addressingData Link Layer – frame synchronization, error control,

physical (MAC) addressingPhysical Layer – cabling, bit stream, electric signal

Troubleshooting Approaches for Networking Problems

Top-Down Approach Bottom-Up ApproachFrom Application to

PhysicalSoftware HardwareWorks best when

problem is isolated to one user or device

From Physical to Application

Hardware SoftwareWorks best when

problem is affecting multiple users or devices

WiresharkOpen source packet analyzerAble to sniff out various types of network

trafficCan use custom display and capture filters

Can also use Perl compatible regular expressions

Helpful when using both top-down and bottom-up approaches

Version used: 1.8.1

IP Address LossSome users are losing network connectivity“ipconfig/release” – “ipconfig/renew” gives

temporary fix, problem reoccurs when computer restarts

Obtaining different addresses eachBottom-up approach will be used

Involves physical connectivityMultiple users affected

IP Address Loss:List of Components InvolvedCore router

wwjer-ro6509-1 (Cisco Catalyst 6500 series, IOS)Core switches

Wwjer-sw6509-1 (Cisco Catalyst 6500 series, CatOS)Wwjer-sw6509-2 (Cisco Catalyst 6500 series, CatOS)

DHCP server “LICNAAD01” 10.80.40.34 Wireshark server “LICPSHARK01” 10.80.40.41

Version 1.8.1Multiple end users

Jack C-345 IPs: 192.168.0.136 and 10.80.110.50

Jack C-336 IPs: 192.168.0.137 and 10.80.110.126

IP Address Loss: First Steps Needed to wait until issue was confirmed by more users All affected users appeared to be on vlan 110 Catalyst Switched Port Analyzer (SPAN)

Allows certain type of traffic to be picked up by designated sniffing server (LICPSHARK01)

set span 110 4/17 session 1

IP Address Loss: Wireshark TraceBoth interfaces used

for capture, encompassing users on both switches

Filter needed for DHCP Packets:Udp port 67 or udp

port 68

IP Address Loss: Wireshark Trace

IP Address Loss: Wireshark Trace

D-Link Wireless Router

IP Address Loss: Finding The Wireless RouterRouter was not always on, had to wait until

another user was affectedFind router with switching commands

Show cam 00:15:E9:F3:EF:B0Router was found at jack E-649DHCP was enabled, and giving off addresses

before DHCP server could do soRouter was removed and problem resolved

RightFax Email DelayRightFax

client software by OpenTextreceives faxed documents, converts to email,

and sends to destination mailboxRightFax emails delayed by hoursTallyFax1 reported the issue firstRightFax runs on separate server (NYCPFAX)Top-Down approach will be used

Related to RightFax applicationReported by one user

RightFax Email Delay: Sample MailFax received by server at 3:05: PMEmail received by recipient at 7:38 PM Approximately 4.5 hours of delay

RightFax Email Delay: List of Components InvolvedField Users sending faxesRightFax server (NYCPFAX) 10.40.40.46SMTP server (NYAPSMTP03) 10.75.10.6Office 365 Cloud

RightFax Email Delay: First Steps Ideally, configuration of RightFax would be checked for problems. However, RightFax is an unfamiliar program, and not much was

discovered. Saw that outgoing server was NYAPSMTP03 (Application Layer) SMTP logging was enabled to find a root cause

RightFax Email Delay: SMTP Log Results Initial thoughts: HRBenefits mailbox was full, so no emails were getting

through. Not valid: HRbenefits mailbox was using only 2.12 GB of 25 GB space

allocated

RightFax Email Delay: Wireshark TraceJump to Network LayerSet up Wireshark trace on NYAPSMTP03

Capture Filter: tcp port 25Display Filter: smtp.rsp.parameter matches

“Mailbox full“

RightFax Email Delay: ResultsError message “452 Mailbox Full” was

coming from 10.75.10.6 (NYAPSMTP03)Further research found that the C:\inetpub\

mailroot\Drop directory was full of unsent emails

Destination email address was incomplete, and default was not valid.

Once quota was hit, emails were bounced back to NYCPFAX, clogging queue on fax server.

RightFax Email Delay: Results

RightFax Email Delay: SolutionWhen WW switched from MS Exchange to Office365, email

aliasing was not considered.With local Exchange server, “TALLYFAX1” could translate

to TALLYFAX1@weightwatchers.comBy sending to the Office365 using SMTP Relay, the

destination email address was considered invalid, and not pick up from C:\inetpub\mailroot\Drop directory

Notification address had to be changed to full email address for each user

Queue of emails in Drop directory on NYAPSMTP03 and NYCPFAX had to be cleared out

After change was made, all faxes and notifications were sent on time.

ConclusionWireshark is extremely useful for

troubleshooting network issues from both types of approachesNetwork Layer is in the middleMakes it equally accessible from both ends

Given more experience, a Divide and Conquer approach may be more beneficialSelect any layer desired

If layer has no issues, check layer above it. If issues were found, check layer below it. The lowest layer with errors is the culprit.