voip switch monitoring and traffic management

2

VOIP Switch Monitoring and Traffic Management

Habib Madani Syed Khurram

3333

Agenda

• Overview of Softswitch• Voip Softswitch Call Processing Overview• Protocols• Call Flows• Softswitch Operations• Trend analysis• Summary

4444

Agenda


5555

Overview of Softswitch

• Network Overview• Network Configuration• Broadband Local Integrated Services Solutions

6666

Network Overview

• The BTS 10200 call agent offers telephony services over IP and ATM networks.

• BTS 10200 provides call control intelligence for establishing, maintaining, routing, and terminating voice calls through the IP or ATM network via media gateways, while seamlessly operating with the PSTN

• Supports Class 5 level services, such as 911 emergency, call forwarding and caller ID

• Also provides support for messaging and announcements

7777

IP

Network Configuration

ANNOUNCE-MENT

SERVER

AS5300

AS5300

PSTNCO

3660

ISDNPBX

ISDNPBX

AS5300

UBRCISCO2600

CASPSAP911

Voice Mail

SERVER

SS7STP

SS7STP

D-Channel Backhaul

MGCPSIP

RUDPSS7 Links

T1/Analog Lines

BTS

10200

Another BTS 10200

or Call Agent

8888

BLISS-T1 Signaling Interfaces

ISDN ISDN BackhaulBackhaul

FCPFCP

CACAFeature Feature ServersServers

Announcement Announcement ServerServer

Trunking Trunking GatewayGateway

PRIPRIPBXPBXCisco

SIPSIP Voice MailVoice MailServerServer

IMTIMT

COCO

Network Management Network Management & Operational Support & Operational Support

Systems Systems SNMP, CORBA, TELNET, SNMP, CORBA, TELNET,

FTP, CLI, HTTPFTP, CLI, HTTP

IP NetworkIP Network

CACA

SIP-TSIP-T

SS7SS7 STP

9999

Agenda


10101010

VOIP Softwitch Call Processing Overview

• ISUP trunking• Emergency Services• Trunking gateway for operator services• Voice mail services• Announcement services

11111111

SS7 Links

IMTs

MGX

IAD

C4/C5

BTSSTP

ISUP Trunking Gateway

• Routes offnet calls to ILEC or IXC over Inter-machine Trunks (IMTs)

• High port density • Interconnects SS7 links b/w IP

Transfer Point (ITP) and C4/5 via STPs

• BTS should be provisioned trunks of the same trunk group across multiple TGWs for redundancy

• Sigtran used to backhaul ISUP to BTS CA for call control

RTP

MGCPMGCP

V

SCTP

12121212

MF/CAS Trunks

IAD

BTSBTS

RTP

PrimaryPSAP

E911Tandem

Trunking Gateway for 911 Services

• Uses Feature Group D, Operator Service (OS) signaling protocol

• TGW requires support of MGCP CAS “MO” package as described in “draft-foster-mgcp-cas-packages-00.txt”

• 2 types of 911s: enhanced 911 and basic 911B911 has PSAP (Public Safety Answer Point)

connect directly to TGW E911 has PSAP connect TGW via tandem switch

• 911 requires “keep the circuit up even if the caller hangs up”

• 911 trunks recommended connected to multiple TGWs for redundancy

Selective Routing Database

AlternatePSAP

Automatic Location ID

MGCPMGCP

MGXV

13131313

MF/CASTrunks

IAD

BTSBTS

RTP

Operator Postions (OPs)

TandemSwitch

Trunking Gateway for Operator Services

• Uses Feature Group D, Operator Service (OS) Signaling protocol

• TGW requires support of MGCP CAS “MO” package as described in “draft-foster-mgcp-cas-packages-00.txt”

• BTS sends Preferred Carrier Info to tandem switch to determine appropriate operation position

• OS does not have “keep the circuit up even if the caller hangs up” requirement as 911

• OS trunks recommended connected to multiple TGWs for redundancy

MGCPMGCP

MGXV

14141414

LDAPIMAP

SMTP

Directory Server

MessageServer

SIP

UC App. Server

IAD

BTS

Application Services

Backend Services

RTPMGX

Voice Mail Server

Voice Mail Server

• Provides voice messaging capabilities• Components

UC Applications ServerVM SW resides. Terminates calls, records and replays

messages, and interacts with backend serversDirectory Server

Stores subscriber profiles and information about which greetings are active and where greetings are located

Message ServerStores and retrieves personal greetings, subscriber

messages, and distribution lists2

MGCP

MGCP

V

15151515

BTS

IAD

AS

Announcement Server

• Instructed by BTS with MGCP to play announcement RTP to ingress MG (MTA or MGX)

• Audio files are stored in AS• MGCP package options for AS:

Script (MG requires scripting language support such as TCL)

Announcement Server• Deploy multiple AS for redundancy

This # has been disconnected.Please check your # again.

RTP

MGX

MGCPMGCPV

16161616

Agenda


17171717

Protocols

• SIP Methods• MGCP Implementation• MGCP Commands• MGCP Modes• ISDN• Channel Associated Signaling (CAS)• CAS - PSAP/911

18181818

SIP Methods

•Consists of Requests and Responses•Requests (unless mentioned, each has a response)

- REGISTER UA registers with Registration Server) - INVITE (request from a UA to initiate a call) - ACK (confirms receipt of a final response to INVITE) - BYE (sent by either side to end a call) - CANCEL (sent to end a call not yet connected) - OPTIONS (sent to query capabilities)

•Messages contain SIP Headers and Body. Body might be SDP or an attachment or some other application

**UA=User Agent (end device)**

19191919

MGCP Implementation

• Communication between the BTS 10200 call agent and the media gateway (MGW) is done via MGCP

• MGCP uses a sequence of commands and mandatory acknowledgements

• Commands contain a requested verb (action to be performed by endpoint) and additional parameters

20202020

MGCP Commands

• NotificationRequest-issued by CA instructing the MGW to watch for specific events, such as hook actions or DTMF tones on a specified endpoint (RQNT)

• Notify-used by the MGW to inform the CA when the requested events occur (NTFY)

• CreateConnection-used by the CA to create a connection that terminates at an endpoint inside the MGW (CRCX)

• ModifyConnection - used by the CA to change parameters associated with a previously established connection (MDCX)

• DeleteConnection - used by CA to delete an existing connection or by MGW when an existing connection can no longer be sustained (DLCX)

21212121

More MGCP Commands

• AuditEndpoint - used by CA to audit the status of the endpoint (AUEP)• AuditConnection – used by CA to retrieve the parameters attached to a

connection (AUCX)• RestartInProgress - used by MGW to notify the CA when the gateway

or a group of endpoints on the gateway are taken out of service or being placed back in service (RSIP)

22222222

ISDN

• Backhaul- Terminology for sending messages between CA - PBX through the AGW

• ISDN-Q.931- Normal application layer messages sent to the CA over IP

• RUDP- Cisco proprietary protocol that makes UDP Reliable

ISDN-Q.931 ISDN-Q.931

UDPUDP

IPIP

Backhaul

RUDPRUDP

23232323

ISDN Network Diagram

RUDP – Signaling and Call Setup/Teardown on the D-Channel

MGCP – Voice, data, or video on the B-Channels

IP

ISDNPBX

ISDNPBX

AS5300D-Channel Backhaul

MGCP

BTS

10200

24242424

Channel Associated Signaling (CAS)

• Inband signaling made up of tones carried on the same circuit as the call they are setting up

• Implemented via MGCP to support PBX connectivity• Dual Tone Multi-frequency (DTMF) refers to the signaling (tones)

generated when you touch a button on a push button pad• MGCP supports all the DTMF/MF

(signaling types: DTMF loopstart, DTMF groupstart, DTMF imstart, DTMF winkstart, MF FGD, MF imstart, MF winkstart,)

• CAS is required to:Support PBX interconnect and incoming CAS trunk interconnectsSupport Barge-In and Busy-Line Verification operator services (operator interrupt services) – Not supported release 1.0Support PSAP/911 services

25252525

CAS - PSAP/911

• 911 services require support for MF signaling• PSAP operators must be able to hold the line even if caller

goes on-hook

26262626

Agenda


27272727

Call Flow

• CA to CA Call Flow – SIP• RG to RG• TG - RG Using ISDN - MGCP Signaling• PBX/ISDN PRI to RG• Barge-In/Busy Line Verification• BLV/BLI Call Flow

28282828

CA to CA Call Flow - SIP

CA-2 PSTN2CA-1PSTN1

1. IAM2. Invite

3. IAM4. 100 Trying

5. ACM6. 183 Progress

7. ACM8. ANM

9. 200 OK10. ANM

12. Talking

13. REL14. RELC

15. Bye16. REL17. RELC

18. 200 OK19. ACK

11. ACK

29292929

BTS

10200RG-o RG-t

1. Off-hook

5. digits

2. NTFY (O:hd) 200 OK

3. RQNT (R:hu (N))

200 OK

4. RQNT (R:hu, hf, [0-9:#*T](D), S:dl)

200 OK

6. NTFY (O:9,1,6,3,4,2,1,2,1,2,T)

200 OK

7. RQNT (R:hu, hf)

200 OK

916-342-1206 916-342-1212

8. CRCX (M:recvonly)

200 OK orig. SDP

Hey Call Agent, I’m going off hook

Hey endpoint, let me know if your subscriber hangs up

Hey endpoint, let me know if your subscriber hangs up, hook-flash or dials digits. Oh and signal dial tone to your subscriber

Hey Call Agent, I am letting you know that I have collected digits

Sure, but let me know if your subscriber hangs up or hook-flash

I am going to create a backwards audio path to you in case some in-band info is played by an endpoint.Endpoint ACKs with his SDP info

Onnet Call – RGW to RGW

30303030

Onnet Call – RGW to RGWBTS

10200RG-o RG-t

200 OK with term. SDP

10. MDCX (M:recv only, term. SDP) 200 OK

916-342-1212

9. CRCX (M:sendrecv orig. SDP)

11. RQNT (R:hd S:rg)

200 OK

12. Ringing

14. Ring Back 15. Off-hook

16. NTFY (O:hd)

200 OK

17. RQNT (R:hu, hf)

200 OK

13. RQNT (R:hu S:rt)

200 OK

Hey endpoint 2, you have a caller. Creating bi-directional path to you with orig. SDP info.

Hey end endpoint 1, I am sending you the term. endpoint SDP information

Hey, ring the phone.

Hey, play a ring back tone, the termination is ringing too.

Hey call agent, my subscriber answered the phone

Ok, but let me know if he hangs up

31313131

Onnet Call – RGW to RGWBTS

10200RG-o RG-t

18. MDCX (M:sendrecv)200 OK

916-342-1212

19. RQNT (R:hu, hf)

Bi-Directional Voice Path

200 OK

200 OK

20. On-hook

21. NTFY (O:hu)

23. DLCX

250 Connection Deleted

22. RQNT (R:hd)

200 OK

I am making your connection bi-directional so you can talk to your buddy and hear him Hey endpoint, let

me know if your subscriber hangs up or hook-flashes

Hey call agent, my subscriber went on-hook.

OK, but let me know if your subscriber goes back off-hook.

Ok I am deleting the connection to you because the originating subscriber is now on-hook.

Conversation:Aunt Pearl tells Sally about her new groovy hairstyle

32323232

Onnet Call – RGW to RGW

BTS

10200RG-o RG-t

916-342-1212

25. RQNT (R:hu)

24. DLCX

250 Connection Deleted

26. RQNT (R:hd)

200 OK

200 OK

28. NTFY (O:hu)

29. RQNT (R:hd)

200 OK

200 OK

Deleting your connection because you went on-hook Let me know if

your subscriber goes on-hook.

Let me know if your subscriber goes off-hook

27. On-hook

Hey my subscriber went on-hook

Ok, but let me know if your subscriber goes off-hook.

33333333

TG - RG Using ISDN - MGCP Signaling

endpoint/[email protected]

IP

MGCP/IP

endpoint/[email protected]

ISDNBackhaul

PRI

User 1 User 2

RG-2

PBXMGCP/IP

BTS

10200

TG-2

34343434

TG-3EO/

User 2PBX

User 1 TG-1 RG-2User 3

PBX/ISDN PRI to RG

SETUP

CRCX(M:recvonly)

14. Off-hook

ACK (SDP1) CRCX

(M:sendrecv, SDP1)ACK

(SDP2)

13. Ring back tone

Alert

SETUPBackhaul

CALL PROC

MDCX(M:recvonly

SDP2)ACK

12. Ringing

RQNT(R:hd, S:rg, rbk(xxx))10. ACK

BTS

10200

CALL PROC

Aler

t

35353535

BTS

10200TG-3User 3EO/

User 2PBX

User 1 TG-1 RG-2

22. CONN ACK

19. MDCX(M:sendrecv)

20. ACK

16. OK

15. NTFY(O:hd)

21. CONN

18. ACK

17. RQNT(R:hu)

23. Bearer Connection Established

PBX/ISDN PRI to RG (cont'd)

36363636

Barge-In/Busy Line Verification

• Permits operators to establish a connection to a customers line to verify a busy condition

• Operator access is provided over dedicated facilities• Facilities connect directly to a switchboard or via a

switched network accessed by remote operator systems• The trunks may use reverse battery loop or E&M lead

supervision with multi-frequency (MF) or dial pulse (DP) signaling

37373737

BTS

10200

BLV/BLI Call Flow

Caller CB -TG/RG OperatorA-RG TG Conversation

seizeNTFY (O:MS/sup)

ACK wink-start

RQNT (R:MS/inf,MS/rel)

CRCX (M: inactive, SDPc)

digits

ACK(SDPc)

NTFY(O:MS/inf(digits))

CRCX (M:recvonly)

ACK

ACK(I: BLV-2, SDPa)MDCX (M:reconly,S:MS/ans, SDPa)

ACK answerMDCX (M: confrnce,I:BLV-1)

ACK

Operator reportsline is busy

MDCX (M: confrnce,I:BLV-2)MDCX(M:sendrec)

ACK

Conversationa

3-way call activeb

38383838

Agenda

• Overview of Softswitch• Voip softswitch Call Processing Overview• Protocols• Call Flows• Softswitch Operations• Trend analysis• Summary

39393939

Softswitch Operations

• Network management and Performance Counters

• Network Performance- Voice Quality

• Defining jitter, packet loss and latency

• VOIP Switch Performance Counters

• Performance Counters Flow

• Types of Counter on CISCO BTS

• Performance counter monitoring

40404040

Network management and Performance Counters

• One of the key Network Management aspects is monitoring Performance counters or Performance Pegs.

• Performance counter collection and reportingTypically in NMS/EMS and NE the Performance data is collected as reports. Performance counters are collected in various time buckets, these buckets keep historic and pseudo-real time data. The pseudo-real time buckets can be reset for immediate trouble shooting.These reports are also periodically dumped to disk as flat files. These files are then pulled off to a data store to perform Data mining.

• Common Service Provider (SP) usagePreemptive trend analysis for capacity planningService Level Agreements (SLA)Quality of service monitoring and network trouble shooting.

41414141

Network management and Performance Counters cont ..

• How do we measure network performance?• VOIP network Key Measurements are based of Call Success Rate , Voice Quality and

Voice Mail access% Ineffective Attempts

Network issues: IP backbone partially down, DNS servers partially down, voicemail trunk congestion, HFC/Cable plant capacity.No Channels available for Off-net PSTN calls.

% Dropped CallsOR IP backbone completely down, total outageCall Processing Failure at the PSTN, signalling link is down or the bearer trunks are down.

• Thus all these factors are deterministic of VOIP network performance and they need to be effectively monitored.

42424242

Network Performance- Voice Quality

• It is dictated by Mean Opinion Score or MOS in short.• How listeners perceive voice quality.• Key Factors affecting Voice Quality for VOIP network:

JitterDelay in packet lossLatency

43434343

Defining jitter, packet loss and latency

• Delay is the time taken from point-to-point in a network. Delay can be measured in either one-way or round-trip delay. VoIP typically tolerates delays up to 150 ms before the quality of the call is unacceptable

• Jitter is the variation in delay over time from point-to-point. If the delay of transmissions varies too widely in a VoIP call, the call quality is greatly degraded. VOIP Network compensates for this by having jitter buffers.

• Packet loss is losing packets along the data path, which severely degrades the voice quality.

44444444

Performance Counters and VOIP Switch Vendors

• Current Performance Counter Availability Currently Counters are available through private Interfaces which capture the VOIP call segments.ISUP counters for PSTN signaling, SIP counters , MGCP counters for trunk gateways, general Call processing counters and QOS counters.

• Industry Standard for VOIP monitoringTo Monitor VOIP Performance, Standard collection and polling mechanisms should available through SNMP/MIBS, CORBA/IDL, CMIP/Q3.

• Alerting based of the Performance Counters The VOIP switch vendors need to implement configurable thresholds mechanisms, acting as a high/low/variable water marks. These watermarks would act as triggers for alarms and events, allowing real time monitoring of the System.

• There is a lack of composite monitoring standard

It would dictate guidelines for Performance counters, collection mechanism Alert trigger and generations.

45454545

Performance Counters Flow

Provider Backbone

HFC Plant

LNP

STP

PSTN

Signaling GW

MGV

CMS/SoftSwitch

MGC

DOCSISDOCSIS

CMTS

VMCONFSRV

ANNSRV

Media Servers

CALEA

NCS EMTA

NCS EMTA

CM

NCS MTA

Dqos Counters

MGCP Counters

SIP Counters

ISUP Counters

Dqos Counters

46464646

Case Study leveraging counters available on CISCO BTS

• CISCO BTS offers a wide set of performance counters through its private interfaces SNMP MIB being one of them.

• The following set of BTS counters capture system health across various VOIP call segments:

ISDN User Part (SS7/PSTN) signaling protocol related information.MGCP signaling protocol related information.SIP Interface Adapter related information Call Processing specific information Trunk Group usage informationDynamic Quality of Service related information

47474747

Agenda

• Overview of Softswitch• Voip softswitch Call Processing Overview• Protocols• Call Flows• Softswitch Operations• Trend analysis• Summary

48484848

Case Study for Trend Analysis

• Trend Analysis and Visual Monitoring of Performance Counters• Architecture of DDRAW setup • DDRAW Dash Board in depth• Trend Analysis using Call Performance counters• Trend Analysis using MGCP Performance counters• Trend Analysis using ISUP Performance counters• Trend Analysis using Dynamic Qos Performance counters• Trend Analysis using SIP Performance counters• Trend Analysis for PSTN Bearer Trunks

49494949



50505050

Trend Analysis and Visual Monitoring of Performance Counters

• BTS performance data collectionBTS performance data was collected for a USA CABLE MSO.The reports were collected at 15 min. buckets for 24 hour, and dumped to flat files (CSV). 3 months worth of these data was collected.

• Pull dataThe data was ftped over to a linux server which had Perl, Round Robin Database(RRD) and DRRAW(cgi) installed on them.

• RRD updatePerl was used to parse the CSV files and RRD was updated with 3 months of cable MSO performance data.

51515151

Trend Analysis and Visual Monitoring of Performance Counters (cont.)

• DDRAW was customized to create a Dashboard for some of these counters.

• RRD and DRRAW we have a pseudo real-time display of the performance stats.This Dashboard displays past 28 hours, 1 week, month and year of data.

• It can be used to create a system wide view of the VOIP call flow, do capacity planning, and keep on top of SLA.

52525252

Architecture of DDRAW setup

BTS 01BTS 01

BTS 02BTS 02

BTS 03BTS 03RRD

Perl ScriptParsing

Parse Data

Populate Data

Client View

HTTP

Pull Data

Pull PM DataLinux ServerClient

Client

MGCP

CallpISUP

DRRAWCGI

DRRAW CGI

Apache

53535353



54545454

Dash Board in depth

• Call Stats Dash Board display, captures, the number of originating call attempts of all types, call attempts, call originating failures and call success on the reporting BTS.

• DQOS Dash actually looks ok, which reflects the CMTS leg, it shows that the Gate SET attempts are equal to the Gate SET Successes.

• MGCP Dash shows number of mgcp attempt success, fail or abandon

55555555

Dash Board in depth cont ..

• ISUP Dash shows SS7 signaling pattern, which includes number IAM, ANM and REL message

• SIP Dash shows number of SIP messages going thru the switch

• Trunk Dash shows utilization of trunk and overflows- Which may help in capacity planning

56565656

Dash Board

57575757

Dash Board cont.

58585858



59595959

Trend Analysis using Call Performance Counters

• It can easily be seen that there is a pattern.• Where the peaks represent the busy hour. Any

deviations from these patterns represents anomalies that would need to be investigated by the Service Provider.

• Also we see a clear gap between the Success and the Call originations, this gap indicates that we are losing calls.

• It could be a result of hang-ups, busy dial, or network problems.

60606060

Trend Analysis using Call Stat counters – Example

61616161

Trend Analysis using Call Stat counters – Example cont.

62626262



63636363

Trend Analysis using MGCP Performance counters

• It can help monitor any performance issues in the network just by reviewing ave mgcp attempts and failures.

• Information can be segregated by gateway which could be related to trunking gateway or announcement server.

64646464

Trend Analysis using MGCP Performance counters – Example

65656565

Trend Analysis using MGCP Performance counters – Example cont.

66666666



67676767

Trend Analysis using ISUP Performance counters

• It can identify issues related to PSTN signaling, or any ss7 link problems.

• The Dashboard example is showing IAM,ANM,REL counters being monitored.

• Any anomaly related to PSTN network/usage related to incoming/outgoing PSTN calls would be visible.

As an example drop in IAM would clearly indicate the call originations are having issues.

68686868

Trend Analysis using ISUP Performance counters - Example

69696969

Trend Analysis using ISUP Performance counters – Example cont.

70707070



71717171

Trend Analysis using Dynamic Qos Performance counters

• COPS protocolGate Sets attempts, Success and Commits are tracked

• Problem trendNetwork problem related to CMTS are visible through these Gate counters.A difference in Gate Set attempts and Gate Set Success would be a clear indication of CMTS resource allocation issue.A slight deviation from the norm could be an indication of a problem before SP starts experiencing it.

• DQOS parameters of Jitter, Latency and Packet loss are also collected by the BTS and can be monitored in this way.

72727272

Trend Analysis using Dynamic QoS counters – Example

73737373

Trend Analysis using Dynamic QoS counters – Example cont.

74747474



75757575

Trend Analysis using SIP Performance counters

• BTS uses SIP to interact with VOICEMAIL equipment and SIP trunks.

• Dash Board SIP CountersSIP Outgoing messages, SIP Outgoing Success,5xx Errors

• Problem trendsAn increase in retransmits or increase in 5xx errors is a visible indication that Voice mail server or SIP trunks is having issues.

76767676

Trend Analysis using SIP counters – Example

77777777

Trend Analysis using SIP counters – Example cont.

78787878



79797979

Trend Analysis using Trunk Usage Counters

• Dash Board Trunk CountersTrunk total overflow, Incoming Trunk Busy, Outgoing trunk Busy, Total Trunk usage.

• A pattern is seenwe can see that most of the trunk seizers are for outgoing trunks.Incoming trunk seizures are low.Overflow of trunks is very low.

• Problem indicationTotal Trunk Usage goes high, Overflow of trunks goes high are indications of capacity issue.

80808080

Trend Analysis using Trunk usage counters – Example

81818181

Trend Analysis using Trunk usage counters – Example cont.

82828282

Summary

• VOIP switch technology is a new field.• Performance counter aspect of network management can be a

key factor in monitoring the network for issues of equipment malfunction, degradation and capacity.

• To provide seamless customer experience from traditional to softswitch, this will help us identifying issue proactively.

• A monitoring strategy is to use RRD and Drraw, for monitoring the system through pseudo real time graphs at the NOCs.

voip switch monitoring and traffic management

Documents