Module ObjectivesTroubleshoot AudioCodes products using the relevant tools from the variety of available utilities.

Correctly report a problem to AudioCodes support department

Upon completion of this module you will be able to perform the following tasks:

Troubleshooting Guide LinesUnderstanding the problem

Collecting data

What are the expected results?

What are the actual results?

Understanding the Problem

Collecting DataTest setup (hardware, call direction, etc.)Uploaded ini file Syslog (decent syslog server without missing messages)Unfiltered wiresharkAdvanced (per request):PSTN traces PSTN problemsDSP traces problems related to voice quality, modem\fax, tones, DTMF etc.

The ini FileMust be added to every support request

Can be downloaded from the Web Interface (Advanced Configuration-> Configuration File)

There is no need to use the Complete ini file; all information can be obtained from the regular ini file

Includes information on the software version, networking parameters and the configuration of the gateway

When a parameter doesnt appear in the ini file it indicates that it received a default value

AgendaSyslog

Network monitors (Wireshark)

Debug Recording

Command Shell

CPT Wizard

DSP Core Dump

Chapter I

AC Syslog

AC SyslogQ: What does ACSyslog application is used for?A: The ACSyslog application is used to collect the trace messages that are sent by the gateway and send them as txt file.

Syslog Definition:The Syslog is a collection of error messages, warning, and/or other system messages that are sent to a central server using UDP port 514.Syslog protocol is defined in the IETF RFC 3164 standard.

AC Syslog ConfigurationSet Log File Name

Enabling Syslog ServerNote: Enter the IP address of your Syslog server (PC where the Syslog installed)

[Syslog]SyslogServerIP = 10.2.0.136EnableSyslog = 1SyslogServerPort = 514

Enable SIP TracesAlways set the debug level to 5 if debug traces are neededGwDebugLevel = 5Remember to turn off after troubleshooting.

Via Web InterfaceProtocol ManagementAdvanced ConfigurationMGCP Debug Level = High

Via ini FileWhen enabling MGCP/MEGACO traces via ini file the traces are available from startup.

EnableSyslog = 1SyslogServerIP = MGCPDebugMode = 4

Enable MGCP / MEGACO Traces

Online Syslog - WebShould only be used for simple debuggingDoesnt include the IP address of the gatewayMisses messages

Syslog Call ExampleIP addresses are 10.33.45.xx

MP1 (.72) (IP) TP1 (.65) (ISDN) TP2 (.69) (IP) MP2 (.61) Traces\sip_mp-tp-mp.log

Syslog ApplicationsExamples of Syslog servers available as shareware on the Internet:

Kiwi Enterprises: http://www.kiwisyslog.com/The US CMS Server: http://uscms.fnal.gov/hanlon/uscms_server/TriAction Software: http://www.triaction.nl/Products/SyslogDaemon.aspNetal SL4NT 2.1 Syslog Daemon: http://www.netal.comWireshark cyclic buffer

Chapter II

Network Monitors (Wireshark)

Wireshark Network MonitorQ: What is Wireshark application used for?A: Wireshark is a free network protocol analyzer for Linux, Unix and Windows. It allows you to examine data from a live network or from a capture file on disk. You can interactively browse the capture data, viewing summary and detail information for each packet.

Network Monitors ApplicationsUse hub or port mirrorUnfiltteredPlugins The Wireshark plugins were tested with Wireshark version 0.99.4

WinPcap ApplicationWinPcap is an architecture for packet capture and networkanalysis for the Win32 platforms. It includes a kernel-level packet filter, a low-level dynamic link library (packet.dll), and a high-level and system-independent library.

http://winpcap.mirror.ethereal.com/

Wireshark Capturing Packets Main menu: Capture Interfaces Select the network interface that is currently being used by the PC

Wireshark Capturing Packets (Cont.) Select the Display options and press Start

Wireshark FiltersFilter specific addresses or / and protocolFor example: sip and ip.addr==10.33.6.100Select a specific field on Packet Details and apply it as a filter

Wireshark Coloring RulesDifferent colors can be assigned for each protocol for better visualization.The Coloring list is processed until a match is found, therefore place general rules (like TCP and UDP) at the bottom of the list.

Wireshark Generating Call Flow

Visual representation of different callsMain menu: Statistics VoIP Calls

Wireshark Playing RTP Stream

From Wireshark version 0.99.4 Play back G.711 conversations Main menu: Statistics VOIP Calls

Wireshark Analyzing the RTP Data

Wireshark can be used to analyze the RTP data stream and extract the audio from the data packets (only for G.711).Main menu: Statistics->RTP-> Stream Analysis menu item

Wireshark Extracting the Audio

Save the audio payload of the RTP stream to a file. Save Payload as a .pcm fileSelect the forward optionCan be played using programs such as CoolEdit

Wireshark Call ExampleMP1 (.100) (IP) MP2 (.101)

Traces\Basic analog call.pcap

MP1 (.100) (IP) TP (.179) TP (.179) MP2 (.101)

Traces\Basic MP to MP via TP call.pcap

Chapter III

Debug Recording

What is Debug RecordingQ: What is Debug Recording used for?A: The Debug Recording (DR) mechanism is used to capture:

- Media streams (such as RTP and T.38) - Networking streams (such as HTTP) - Signaling traffic (such as SIP, MGCP and ISDN) - Other internal board information (such as DSP Events).

DR is relevant from version 5.2 and is used instead of the obsolete mechanisms that were used prior to this version to collect DSP / TDM and PSTN traces.

The Main Advantages of DR

Can be used to record all IP traffic that is sent / received from / by the gateway

Can be used to record the actual messages that are exchanged between the gateway and the PBX/PSTN switch

Can be used to record the actual voice signal that arrives from the TDM (before it enters the DSP)

Can be used to record network traffic in environments in which hub / port mirroring isnt available

Can be used to record internal traffic between two endpoints on the same gateway

Collecting DR Messages

The client that is used to capture the DR packets is the open source Wireshark application.

AudioCodes proprietary plugins (supplied in the software kit) must be placed in the 'plugin' folder of the installed Wireshark version (typically, c:\Program Files\WireShark\plugins\xxx\, where xxx is the installed version).

The plugins were tested with Wireshark version 0.99.4

Use the following filter ACDR or udp.port==925 to view the DR messages that are sent by the gateway.

Activating the DR

Start a CLI management session (via the FAE page)

At the prompt, type DR to access the DebugRecording directory.

At the prompt, type STOP to terminate all active recordings, if any.

At the prompt, type RTR ALL to remove all previous recording rules.

At the prompt, type RT ALL to remove all DR targets (i.e., client IP addresses) from the list.

At the prompt, type AIT to define the IP address of the PC (running Wireshark) to which the gateway sends its debug packets.

Record Media Streams

Records: Internal DSP packets and eventsPCM - voice signal that arrives from the TDM (before it enters the DSP)RTP, RTCP, T.38syslog

GW is sending DR packets to external IPIP

Record Media Streams (cont.)

AddNextCallTrace records the next media calls. AddTrunkBchannelTrace records media calls according to trunk and B-channel. AddChannelIdTrace records media calls according to CID. To record all media traffic (including PCM) of the next call enter: ANCT ALL-WITH-PCM 1 At the prompt, type START.

Note that currently PCM record on MP is available only on version 5.0See Trace Files: DR on internal MP call

Record Media Streams (cont.)

RTP/PCMUse filter rtp and acdr

Record PSTN MessagesRecords ISDN, CAS or SS7 messages that are exchanged between the gateway and the PSTN/PBX switch.

First enable PSTN traces; via cmd shell enter: pstn PstnCOmmon PstnSetTraceLevel For example, to enable PSTN traces on the first trunk: PstnSetTraceLevel 0 -1 1

For SS7 only, set the Trace Level of the SNs and Links tables to 1.

At the prompt, type AddPstnSignalingTrace . For example, APST ISDN.

At the prompt, type START. See Trace Files: PSTN Trace with DR

Record PSTN Messages (cont.)ISDN call flow can be viewed using the filter: acdr and q931

Record PSTN Messages (cont.)The trace can also be converted to a graph: Statistics->VoIP Calls

Record Control MessagesRecords SIP, MEGACO, MGCP or TPNCP messages.

AddIPControlTrace TracePoint ControlType

Trace point (not relevant to SIP): Net2Host = Inbound non-media traffic Host2Net = Outbound non-media traffic

ControlType: SIP, MEGACO, MGCP or TPNCP

At the prompt, type START.

Record IP Traffic

Used to record IP streams that are not associated with media or control. Such as HTTP and SCTP.

Format: AddIPTrafficTrace TracePoint PDUType SourcePort DestPort

Trace Point: Net2Host or Host2NetPDUType: UDP, TCP, ICMP or IPType# Any other protocol type (as defined by http://www.iana.org/assignments/protocol-numbers). A = All traffic types.SourcePort: datagrams source port number (or A for all ports).DestPort: datagrams destination port number (or A for all ports).

To record SCTP traffic: aiptt n2h 132 a a aiptt h2n 132 a a

DR General Notes

PSTN, DSP, Control and IP recording can be performed simultaneously

All DR rules are deleted after the gateway is reset.

Save the information that was captured in Wireshark and send it to AudioCodes.

DR recording must only be used for debugging purposes. On normal operation (if not requested differently from AudioCodes support) DR should be disabled.

To stop the DR, enter the command STOP in the command shell

Chapter IV

Command Shell

Command ShellQ: What is Command Shell application used for?A: The Command Shell is primarily used for debugging purposes and to obtain various information about the gateway.For example, to check PSTN clocks synchronization.

Accessing the Command Shell via Web

Type the ip_address /FAE in the address fields of your web browser.

Accessing the Command Shell via TelnetEnable the Embedded Telnet Server: When using the ini file, set the parameter TelnetServerEnable to 1 (standard mode) or 2 (SSL mode). When using the Embedded Web Server, set the parameter Embedded Telnet Server to Enable (Unsecured) or Enable Secured (SSL) and save the changes so they are available after a power fail.

Reset the gateway.

Use a standard Telnet application to connect to the gateways Embedded Telnet Server. Note that if the Telnet server is set to SSL mode, a special Telnet client is required on your PC to connect to the Telnet interface over a secured connection.

Login using the same username (default Admin) and password (default Admin) you use for the Embedded Web Servers Administrator level.

Accessing the Command Shell via RS-232Can be used when the gateway cannot be accessed via IP

Connect the RS-232 port to your PC

Use a serial communication software (e.g., HyperTerminalTM) with the relevant communications port settings (according to the gateway).

Commands Shell and Passwords There are three groups of commands:

Commands that are always available

Advanced commands. To enter this mode type: AdvancedMode. Use 'ADVancedmode OFF to return to normal mode.

Commands that are locked by password. The password is calculated by AudioCodes FAE from the IP address of the gateway. To enter this mode type: PW xxxxx Were xxxxx is the password. Use PW OFF to return to normal mode.

Command Shell Syntax Command line syntax is: [Command Name]

Enter h at the CLI prompt for help on global commands and enter h for information on a specific command.

Use two consecutive dots (i.e., ..) to access a higher directory level.

You can use the upper case of each command / directory as a shortcut. For example, enter CONF instead of CONFiguration and GPD instead of GetParameterDescription.

How to Get the Gateway's ExceptionsWhen the gateway resets itself during operation without an explicit command from the user, an exception is burned to Flash. This exception can be viewed in the syslog server immediately after the gateway starts up and can be retrieved from the command shell using the following procedure:Access the FAE page (xx.xx.xx.xx/FAE)On the left hand menu press Cmd ShellIn the Command Line type the following commandsBSPEXCEIPH

PSTN Clock SynchronizationSeveral problems (such as, echo, fax and modem failures) can be caused if the E1/T1 clocks arent synchronized.To verify if the clocks are synchronized, use PSTN Performance Monitoring (PM).Note that PM is enabled by default.Access the command shellIn the Command Line type the following commandsAdvancedModeCd PSTNCd PhysicalPstnGetPerformanceMonitoring trunknumber 0 send the results to AudioCodes (for example, PstnGetPerformanceMonitoring 1 0)

Chapter V

Call Progress Tones Wizard

What is the CPT FileThe Call Progress Tones (CPT) configuration file is a binary file(with the extension dat) which contains the definitions of the Call Progress Tones (levels and frequencies) that aredetected / generated by the gateway.

Most common tones are:Dial toneRingback toneBusy toneReorder Tone

Tones_File Example.ini

CPT WizardQ: What is CPTWizard used for?A: The Call Progress Tones wizard helps detect the Call Progress Tones generated by your PBX (or telephone exchange).The software automatically creates a basic Call Progress Tones: ini file. Note: The version of the CPTWizard must match the software version of the gateway.

Getting Started with the CPT WizardTo use this wizard:

Connect the MediaPack/FXO gateway to your PBX with two physical phone lines

To install the CPTWizard, copy the files from the supplied installation kit to any folder on your PC.

The FXO gateway must be configured with factory-default settings and shouldnt be used for phone calls during the operation of the wizard.

CPT Wizard- Initial Settings

Execute the CPTWizard.exe file.Enter the IP address of the MP-11x/FXO.Select the gateways ports that are connected to your PBX, and specify the phone number of each extension.Insert Invalid phone number field, that generates a fast busy.

CPT Wizard - Automatic Mode Press the Start Automatic Configuration button. The wizard starts the Call Progress Tones detection sequence (the operation takes approximately 60 seconds to complete)

CPT Wizard - Automatic Mode (Cont.) Sets port 1 offhook, listens to the dial tone.

Sets port 1 and port 2 offhook, dials the number of port 2, listens to the busy tone.

Sets port 1 offhook, dials the number of port 2, listens to the Ringback tone.

Sets port 1 offhook, dials an invalid number, listens to the reorder tone.

CPT Wizard - Recording ScreenThe wizard then analyzes the recorded Call Progress Tones.4 Call Progress Tones are saved as standard A-law PCM at 8000.

CPT Wizard - Manual Mode Manual mode enables you to record and analyze tones in addition to those tones analyzed when in automatic mode.

Converting ini to dat Convert the ini (text) file to binary mode using the Dconvert software.

Chapter VI

DSP Core Dump

DSP Core DumpQ: What is a DSP Core Dump?A: A DSP Core Dump is a file that is created after a DSP restart occurs (indicated in the syslog). It is used to debug DSP restart problems.

DSP Core DumpThis option is available:On version 4.8 from baseline F4.80.052.002On version 5.0 from baseline F5.00.026.007 On all 5.2 versions (and higher).

The DSP dump file will be replaced automatically (by another DSP restart event) only after the previous DSP core dump file was downloaded.

To download the DSP Core Dump file:Access the FAE page.Press the Get DSP MON File link, and download the current Core Dump file.

Thank you for your time

**************Our implementation includes 3 objects that define different aspects of the call: Endpoint (EP), Call and Session. In a very brief description:Endpoint handles the actual endpoint, usually same as CIDCall Handles the call from infrastructure pointy of view (mostly common to H.323 and SIP)Session handles the call from the protocol side (H.323 and SIP)Each call has 3 identification numbers EP ID, Call ID and Session ID.Going back to the syslog you probably realized that the messages are indented. The leftmost messages are related to the board side (the telephony) and the rightmost are related to the IP.The leftmost messages are endpoint related messages and the number #5 is the endpoint ID.The next tab is the call related messages so #3 is the callThe next tab are session related messages.*****************************************************