TROUBLESHOOTING

Network infrastructure maintenance

and troubleshooting

Talking to the customer

• Customer: – I have written a letter in Word but I can’t print

it out because the mouse is stuck• Help desk:

– If your mouse is stuck how could you write the letter?

• Customer: – The computer is working fine but the printers

error lamp flash because the mouse is stuck• Help desk: ????

Network maintenance

• Hardware and software installation and configuration• Troubleshooting problem reports• Monitoring and tuning network performance• Planning for network expansion• Documenting the network and any changes made to the

network• Ensuring compliance with legal regulations and

corporate policies• Securing the network against internal and external

threats

Network maintenance goals

• Reduced network downtime– Discovering and preventing problems before they happen

• More cost-effectiveness– Performance monitoring and capacity planning allows decision

makers to make efficient budgeting decisions.

• Better alignment with business objectives– Time and resources are allocated to processes based on their

importance to the business

• Higher network security– Continued attention to network security and prevention

techniques including monitoring

Proactive Versus Reactive

• Structured tasks (Proactive)– Performed as a predefined plan– Often a maintenance policy/model is used– Planning for changes and the future

• Interrupt driven tasks (Reactive)– Resolving issues as they are reported– Often little or no planning

Maintenance models

• Many companies follow a well-known maintenance model– Trouble tickets– Up-to-date documentation– All changes described– Knowledge database

Model Explanation

ITIL IT Infrastructure Library

FCAPS Fault, Configuration, Accounting, Performance and Security management

PPDIOO Prepare, Plan, Design, Implement, Operate and Optimize

ITIL

• IT Infrastructure Library• Set of practices for IT service management 

– (ITSM) • Focuses on aligning ITSM with the needs of

business• Current version ITIL version 3• ITIL certification required at some companies• Developed by UK government• Widely used as formalized way of doing ITSM

FCAPS

• Developed by ISO for telecommunication network management

• Fault management– Recognize, isolate, correct and log faults

• Configuration management– Gather and store configurations of devices

• Accounting management– Billing management

• Performance management– Monitor efficiency of network and prepare for the future

• Security management– Controlling access to the network (authentication)

PPDIOO

• PPDIOO– Developed by Cisco for network and service

management– Network lifecycle

Troubleshooting

• Using a structured approach

Step Description

Collect information A problem report often lacks sufficient information

Examine collected information For example comparing to baseline information. (If available)

Eliminate potential causes Based on knowledge of network and collected information - start to eliminate causes.

Hypothesize underlying cause After eliminating causes hypothesize the most likely cause of the problem.

Verify hypothesis Test if the hypothesize resolve the problem

Troubleshooting

(1) Problem report (2) Collect information

(3) Examine information (4) Eliminate potential causes

(5) Hypothesize underlying cause (6) Verify hypothesize

Structured Approach

(1) Problem report (2) Collect information

(3) Examine information (4) Eliminate potential causes

(5) Hypothesize underlying cause (6) Verify hypothesize

Shoot from the hip

Troubleshooting

• Different approaches– Top-down

• Start with the application (Internet explorer)

– Bottom-up• Start with the physical (Link established to network)

– Divide and conquer• Is it possible to ping. If yes then … if no then …

Layer 3: Network

Layer 2: Data link

Layer 1: Physical

Layer 6: Presentation

Layer 5: Session

Layer 4: Transport

Layer 7: Application

Bot

tom

up

Layer 3: Network

Layer 2: Data link

Layer 1: Physical

Layer 6: Presentation

Layer 5: Session

Layer 4: Transport

Layer 7: Application

Top

dow

n

Layer 3: Network

Layer 2: Data link

Layer 1: Physical

Layer 6: Presentation

Layer 5: Session

Layer 4: Transport

Layer 7: Application

Div

ide

and

conq

uer

Troubleshooting

• Follow the path of traffic– Step 1: Is the switch switching the packet– Step 2: Can the Router be pinged from the PC– Step 3: Is the Router routing the packet – Step 4: Is the switch switching the packet– Step 5: Is the server receiving the packet

Step 1 Step 2 Step 3 Step 4

Troubleshooting

• Component swapping

Switch 1Labtop A

Switch 1Labtop A

Switch 1Labtop B

Switch 2Labtop A

Swap Cable

Swap Switch port

Swap Labtop

Swap Switch

Port 1

Port 2

Port 1

Port 1

T-shoot example

• Problem report from user at client 3:– I have no connection to the Internet

Server 1DNS

DHCP

Server 2FIL PRINT

1

192.168.0.2

Building 2

Client 2

192.168.200.0/24

1

R1

Internet

DHCP172.30.0.0/30

1 2

Building 1

192.168.0.3

DHCP

DHCP

SW1

Client 1

254

SW2

Building 3

192.168.210.0/241

6

DHCP

SW3

2

R3R2

Client 3

172.30.0.4/305

Printer

DHCP

Client 4

T-shoot example

• Very short problem report.– Could be almost anything!!

• Collect information– Network drawing good documentation– Try and talk to the user – if available

• Are other users experiencing symptoms?– In this case it is after business hours – so no info

– We can’t find more info before we visit the customer's premises• Where should we go??

T-Shoot example

Server 1DNS

DHCP

Server 2FIL PRINT

1

192.168.0.2

Building 2

Client 2

192.168.200.0/24

1

R1

Internet

DHCP172.30.0.0/30

1 2

Building 1

192.168.0.3

DHCP

DHCP

SW1

Client 1

254

SW2

Building 3

192.168.210.0/241

6

DHCP

SW3

R3R2

Client 3

172.30.0.4/305

Printer

DHCP

Client 4

We have to go to building 3Client 3 and verify the problem

For example by opening the browser

and connecting to google.com andsearch for applepie

T-Shoot example

• Follow the path of the traffic– A chain of events.

• Analyze prerequisites– The PC needs the following prerequisites to

be successfully connected to the Internet1. A valid IP address from the DHCP server

2. A valid IP address for a working DNS server

3. Connection to the Internet through the network

• Examine each prerequisite for itself– Simplify T-shooting

The DHCP chain

Server 1DNS

DHCP

Server 2FIL PRINT

1

192.168.0.2

Building 2

Client 2

192.168.200.0/24

1

R1

Internet

DHCP172.30.0.0/30

1 2

Building 1

192.168.0.3

DHCP

DHCP

SW1

Client 1

254

SW2

Building 3

192.168.210.0/241

6

DHCP

SW3

R3R2

Client 3

172.30.0.4/305

Printer

DHCP

Client 4

DHCP discover packet sent from client

DHCP discover packet Received by Router

And relayed to 192.168.0.2 R3 routes packet To R2

R2 routes packet To R1

R1 routes packet To server 1

DHCP server find a freeIP address in the

192.168.210.0/24 poolDHCP server sends DHCP Offer back to the client

Vi R3 DHCP relay

DHCP chain tools

• IPCONFIG /ALL– See detailed IP configuration on host

• IPCONFIG /RELEASE– Release IP address leased from DHCP server

• IPCONFIG /RENEW– Request DHCP lease from DHCP server

• PING 192.168.0.2– Is it possible to reach the DHCP server

• TRACERT –d 192.168.0.2– How far – in routers – will the packets go

DHCP chain tools

• Logon to the DHCP server and check– Is the DHCP service running– Does a scope exist for 192.168.210.0/24

• Is the scope active• Are the any free IP addresses for leasing

DHCP chain tools

• Are the host configured as a DHCP client• Are the DHCP relay on R3 configured

correctly• If the host has no IP address and it’s

impossible to lease one– It is possible to temporarily give the host an IP

address (Possible IP address conflict !!! )– For example IP Address 192.168.210.201

Subnet mask 255.255.255.0

Default gateway 192.168.210.1

DNS server 192.168.0.2

The DNS chain

Server 1DNS

DHCP

Server 2FIL PRINT

1

192.168.0.2

Building 2

Client 2

192.168.200.0/24

1

R1

Internet

DHCP172.30.0.0/30

1 2

Building 1

192.168.0.3

DHCP

DHCP

SW1

Client 1

254

SW2

Building 3

192.168.210.0/241

6

DHCP

SW3

R3R2

Client 3

172.30.0.4/305

Printer

DHCP

Client 4

DNS request sent to

DNS serverIe. www.ascom.no

DNS Server tryes toResolve

www.asom.noIf the DNS can’t resolvewww.ascom.no it will ask

It’s forwarder or a DNS root-server

(Depending on configuration)

Answer from Internet returns

DNS server returns answer to client

DNS chain tools

• NSLOOKUP– Test DNS server resolver

• If known test DNS servers own zone-records– For example NSLOOKUP server1.ascom.local

• Logon to the DNS server– Check DNS service running– Check Internet connectivity from DNS server

• Ping 8.8.8.8

– Check forwarder and if root-server IP addresses are configured properly

The Internet chain

Server 1DNS

DHCP

Server 2FIL PRINT

1

192.168.0.2

Building 2

Client 2

192.168.200.0/24

1

R1

Internet

DHCP172.30.0.0/30

1 2

Building 1

192.168.0.3

DHCP

DHCP

SW1

Client 1

254

SW2

Building 3

192.168.210.0/241

6

DHCP

SW3

R3R2

Client 3

172.30.0.4/305

Printer

DHCP

Client 4

Clints sends HTTP: request Packet

to www.ascom.no’s IP Address

Answer from www.ascom.no

The Internet chain

• PING to the internet– For example: PING 8.8.8.8

• TRACERT to the internet– For example: TRACERT –d 8.8.8.8

• Check routing tables on routers– The routers has to know all routes on the way– Check default route 0.0.0.0/0

• Is NAT/PAT working properly on R1

T-shoot Conclusion

• While testing one chain after another– The problem will be isolated and we focus on

one chain– Dissect the suspected chain and find the

problem using• Divide and conquer• Bottom-up• Top-down• Shoot from the hip• Swapping devices

• It could of course be a misconfigured browser

Thank you for participating