call trace analyser

Outline
Background and Motivation Other Call Analysis and Classification Tools Uses for CACL Data Flow Overview Running the CACL Command Output from CACL CACL Post-Processing Tools CACL Scenario Language

Background and Motivation

Early call drop analysis:
Performed by hand Practical for small samples drive tests
Automated approach to call analysis provides:
Consistent and comparable results Ability to analyze very large numbers of calls
commercial traffic
Large scale analysis provides:
Statistically valid results Ability to predict effects of fixes and enhancements Ability to detect low occurrence events

Other Call Analysis and Classification Tools
Call Drop Analyser CDA (Mark Warren)Initially based on Call Failure Reports as well
as other BSC logging and logActErr Also can now use BTI trace
- Uses early version of CACL Scenario Language Web-based interface to request logs for automated
retrieval and analysis, and results display IMSI Tracking and Reporting Tool (Jat Shocker/Rui Cruz)
- Does real-time tracing for selected IMSI
- Uses CACL for near-real-time classification

Investigation on merging common components

of CDA and CACL ongoing

Uses for CACL
Classify failed calls:
Cause for failure (e.g. Radio Link Failure, RLC, procedure
timeout)
Classify successful calls:
Normal termination
Handover to 2G
User inactivity (PS only)Analyze procedures such as RRC Re-establishment or Cell Change
Order which might occur multiple times per call
- Determine relative frequency of different scenarios
- Gather timing information (e.g. time to execute procedure)
Some types of analysis focus on end of call, others
analysis the call over its entire duration

CACL Dataflow Overview

RNC

Decoded

BSC/BTI
logs per
FMS

(BIG)

Parsed
BSC/BTI
logs per
FMS

(small)

rnclogcat

messages.pl

cacl
output
records

(one per

call)

Post processing
- caclReport
- caclSelect

- sed/grep/wc

- etc

Parsed
BSC/BTI
logs for

all FMS

cat

caclSort

Parsed
logs for

all FMS
sorted by
RNTI

cacl

CACL
Scenario
File

Binary

Logstore

Running the cacl command

cacl s [-d] [-S] [-r rnti] [-x suffix] input-file

where

scenario-file drives the analysis (described later in this talk)

-d is used for debugging scenarios trace output generated

-r is used to filter to a particular RNTI (usually for debugging)

-S is used to specify that input files are pre-sorted (see below)

-x is used to specify file output suffix (see below)

input-files are parsed BTI files to be analyzed
Output appears on stdout, unless x is used in which case an output
file with specified suffix is generated for each input file Input files can be presorted with caclSort. If this is done, use S to
save time by not sorting them again Handles compressed input files (.gz) if extension .gzs assumes they
are presorted and then compressed (-S not needed)

Output from cacl

Strictly speaking, scenario file defines output. But for call drop analysis
as currently being done, output consists of:
One line per call with call scenario and all user and system variables
for that call, in tag=value notation. If call was not classified, BTI trace is listed after one-line record, to
help with manual analysis
MATCHED SCENARIO="User-inactivity" CATEGORY="Non-failure" TAG="NONE" RAB="HS/CONFIRMED" RNTI="001028 START_TIME="11:38:21.338 END_TIME="11:40:25.944" RBCONFIG="COMPLETE" LASTNB="1000063" ECNO="-3@16914" RRCSTATE="FACH" LASTHSPSC="NIL" IMSI="310410086357674" ASU="COMPLETE" TCCONFIG="COMPLETE" HO2GSM="NO" E3A="NIL" DUR="123328"

MATCHED SCENARIO="RNC RLC DCCH failure after 6.5+ sec w/o major activity" CATEGORY="RNC RLC DCCH failure: Standalone or shortly after procedure (except CM)" TAG="NONE" RAB="CS/CONFIRMED" RNTI="001069" START_TIME="17:08:53.515" END_TIME="17:39:14.520" SECMODE="COMPLETE" RBCONFIG="NONE" LASTNB="1000095" ECNO="-7@30360" IMSI="310410093282286" ASU="COMPLETE" CM="NIL" HO2GSM="NO" IRAT="NIL" E2F="ON@1811268" DUR="1820409"

CACL post-processing tools - caclReport
Designed specifically for call drop analysis output Generates a report containing a number of statistics including failures
and non-failures per
- Category and scenario
- Last NodeB involved in call
- IMSI Some stats are repeated for calls where last known EcNo was good Somewhat ad-hoc, new output sections are added as needed Usage:
caclReport input-file

where
input-file(s) contain cacl output records

CACL post-processing tools - caclSelect

caclSelect is used to select specified records and dump specified

fields (based very loosely on SQL SELECT)

caclSelect w regex nw regex f field-list input-file

where

-w (where) is used to specify a regex that must be present in selected
records

-nw (not where) is used to specify a regex that must not be present in
selected records

-f is used to specify a field (variable) list. Must be quoted if multiple
fields are specified. The field name %FILE presents the input file
where the record were found

input-file(s) which contain cacl output records

Output is tab-separated for easy import into excel

CACL post-processing tools
caclSelect (contd)

caclSelect nw Non-failure f RNTI START_TIME ECNO SCENARIO caclOutput

001069 17:08:53.515 -7 RNC RLC DCCH failure after 6.5+ sec w/o major activity

001433 11:52:33.822 -14 ASU: UE RLC DCCH failure waiting for ASUC

001526 17:28:26.931 e2d NB RLF (1 RL) after 4.5+ S w/o major activity

001546 17:24:16.160 -14.5 ASU: NB RLF waiting for ASUC

001785 17:38:39.223 -10 ASU: NB RLF waiting for ASUC

001788 17:35:46.562 -11.5 RNC RLC DCCH failure, no procedures, within 6.5 S of ASUC

001912 12:14:21.512 -21.5 CM: Failure within 4 seconds after cancelling reconfiguration due to NB RLF

caclSelect nw Non-failure f SCENARIO caclOutput | sort | uniq c | sort -rn

12 IRAT timer trelocoverall

11 NB RLF (1 RL) after 4.5+ S w/o major activity

8 RNC RLC DCCH failure after 6.5+ sec w/o major activity

6 CM: NB RLF (1+ RL), no procedures, within 6.5 S of activation

6 ASU: NB RLF waiting for ASUC

5 RNC RLC DCCH failure, no procedures, within 6.5 S of ASUC

3 Unclassified failure with Iu-ReleaseRequest

3 NB RLF (1+ RL), no procedures, within 4.5 S of ASUC

3 CM: NB RLF (1+ RL), no procedures, within 12 S of activation

3 ASU: Timeout or L1/L2 failure waiting for ASUC after adding drift leg

3 ASU: RNC RLC DCCH failure waiting for ASUC

2 NB RLF (2+ RLs) after 4.5+ S w/o major activity

Location of files

On macroni cluster machine (e.g. bldr): /home/umtslogs/tools

Or via:

http://mobility.ih.lucent.com/~erschroe/umtslogs/tools/

Important files:

cacl main command

cacl.pl called from cacl

caclSelect, caclReport post processing tools

scenCsDrop current scenario files for CS drops

scenHsDrop current scenario file for HS drops

caclSort command to sort BTI parsed log files

caclSort outputFile

cat parsed*fms* | caclSort >parsed.sorted

CACL Scenario Language (CSL)

Introduction to CACL Scenario Language
Used to describe patterns in BTI trace, called scenarios A scenario typically represents A particular type of call failure
e.g. RNC DTCH failure within 15 seconds of ASU
A particular variation of a procedure
e.g. Cell Change Order followed by physical channel
error (this does not result in a call drop) A type of call to be excluded from analysis
e.g. a call still in progress at end of log A scenario might also be thought of as a signature
What CACL generally does is scans BTI trace for calls
to determine which scenario(s) they contain

CACL Operation for Call Drop Analysis

Global config

Variables

Scenario 1

Scenario 2

BTI Trace File (Parsed)

Scenario File

for each call (RNTI) in a BTI Trace File
{

for each scenario in the Scenario File

{

if call matches scenario

{

print one line summary: scenario id
and call data

break

}

}

print full call data including BTI trace for
manual analysis of unclassified calls

}

Trace RNTI X

Trace RNTI Y

Typical Scenario File Layout
Global commands
- Restriction on RAB type and RAB state
- Debug output Variable definitions Scenario definitions Scenarios are usually the bulk of the file Order of scenarios can be important when in the default mode
of analyzing a call until a matching scenario is found
- If a call might match multiple scenarios, put the most
specialized one first

CSL: A Sample Scenario

SCENARIO RNC RLC DTCH failure, no procedures, within 15 S of ignored e1X

CATEGORY RNC RLC DTCH failure: Standalone or shortly after some procedure

SCAN_FOR_EVENT CnU:Iu-ReleaseRequest

VAR_STATE_RETRIEVE LASTNB_IURELREQ

REQUIRE_VAR $VAR{RBCONFIG} eq 'NONE' || $VAR{RBCONFIG} eq 'COMPLETE'

REQUIRE_VAR $VAR{ASU} eq 'NONE' || $VAR{ASU} eq 'COMPLETE'

REQUIRE_VAR $VAR{TCCONFIG} eq 'NONE' || $VAR{TCCONFIG} eq 'COMPLETE'

REQUIRE_NEXT_EVENT TpU:RadioBearerFailureIndication \[sapId: .*CNTermRLCSAP

REQUIRE_NEXT_EVENT_BLOCK

REQUIRE UeU:measurementReport \[e1

ALLOW UeU:measurementReport \[e2f

TIME_LIMIT SINCE_LAST_MATCH 15000

END_BLOCK

END_SCENARIO

CSL: Scenarios

CSL Concepts: Scenarios

Describing patterns can be complex:
Events may occur in slightly different orders
> May get response from NodeB and UE in
either order
Events may include aspects of AND and OR
> AND: Require response from UE AND NodeB

> OR: Require RLC failure from UE OR TPU

- Noise events (not applicable to the scenario)

> Measurement reports

> Direct transfer messages

CSL Concepts: Scenario Attributes
Event sequence specification including timing constraints
- This is usually the bulk of the scenario definition State requirements:
- CACL provides some state, such as type of
RAB and its state
- Users can define their own states, such as if a
radio bearer reconfiguration is in progress Event scan direction: For call failure analysis, we usually scan in reverse
time order, starting at Iu-ReleaseRequest For other types of analysis, we may scan in forward
time order if it is more natural

CSL Concepts: Scenarios Attributes (contd)
Scan type: By default, CACL scans until the pattern match succeeds
or fails and then stops processing that scenario As an option, once the pattern match fails or succeeds,
CACL restarts the scenario, until the entire call is analyzed Match type: By default, CACL stops processing a call if it matches
a scenario
> Used for call classification As an option, even if the call matches a scenario,
CACL will continue analysis of the call with the next
scenario
> Used for some types of procedure analysis

CSL Concepts: Scenarios Attributes (contd)
Ignore call if matched
- As an option, stop analysis of calls that match the
scenario (these scenarios are usually placed first in
the file and might be in-progress calls)
Debug options:
- e.g., print call data if call matches scenario

CSL: General Notes on Syntax
Comment lines start with # (leading white space allowed) Blank lines allowed anywhere Commands must be on a single line Commands can be indented
- Encouraged to improve readability
Most strings should NOT be quoted
- Leave one space between command and string

- Use spaces as desired within string

- Ensure there are no trailing spaces (unless intentional)

- Exception to this are expressions used to set variables

Format of BTI Logs for Matching

Parsed logs lines edited slightly to simplify matching:
All white spaces replaced with a single space (e.g. no double spaces) Message type & direction put adjacent to message name separated
by a :
Message types

Nb NodeB (NBAP)Cn Core (RANAP)

Rn RNC (RNSAP)Ue UE (RRC)

Ns - NAS

NsU: Paging Response
TpD:CreateCOChannelRequest
TpU:CreateCOChannelSuccess [Success](ref: 15660)
CnU:InitialUE-Message
CnD:SecurityModeCommand
TpD:ConfigureUEContextCipherPrepareRequest
TpU:ConfigureUEContextCipherPrepareSuccess
UeD:securityModeCommand
UeU:securityModeComplete
TpD:ConfigureUEContextCipherUpdateRequest
TpU:ConfigureUEContextCipherUpdateSuccess
CnU:SecurityModeComplete
CnD:CommonID (310410079762773)

CSL: Matching Scenarios
While matching a call against a scenario, there is a conceptual
cursor in the call trace and in the scenario, as matching is done CACL always processes a scenario from top to bottom. But for
call trace, by default, it starts at the end of the call and moves to
the beginningMost natural for call drop analysis Matching is via perl regular expressions Superset of UNIX regular expresssions Can use expressions such as .* Must escape (with \) special chars such as: [()] A line of BTI trace is known as an event

CSL: SCENARIO Block

SCENARIO

CATEGORY

TAG

END_SCENARIO

SCENARIO UE RLC DTCH Failure

CATEGORY RLC DTCH

END_SCENARIO

Syntax

Example
CATEGORY is optional -- default is scenario name TAG is optional intent is to use for MR/AR numbers

CSL: Simple Pattern Matching Commands

These are used to match a single line of BTI trace

SCAN_FOR_EVENT
Scans through BTI log, looking for specified pattern If pattern not found, scenario does not match Otherwise trace cursor is positioned at next event For a call drop scenario, first command is very often:
SCAN_FOR_EVENT Iu-ReleaseRequest

REQUIRE_NEXT_EVENT
Next BTI event must contain pattern (except for ignored
events), otherwise scenario does not match Trace cursor is positioned at next event

CSL: Complex Pattern Matching Commands

These are used to match a combination of trace messages and/or

specify time constraints


< Block commands>

END_BLOCK

PROHIBIT_NEXT_EVENT_BLOCK

< Block commands>

END_BLOCK

Block commands are used to

- Specify patterns to IGNORE with no analysis

- Specify patterns to REQUIRE to be present

- Specify patterns to PROHIBIT from being present

- Specify patterns to ALLOW to be present

- Specify time constraints

CSL: Primary Match Block Commands
IGNORE REQUIRE
- Note REQUIRE events can occur in any order in the trace
PROHIBIT ALLOW TIME_LIMIT SINCE_LAST_MATCH Limits how long (in BTI log time) to scan for REQUIRED
events -- if time limit is reached, block fails Order of commands within a match block does not matter All commands are optional

CSL: Other Match Block Commands
REQUIRE_ONE
- Only one of the REQUIRE events need to be present - an OR

condition. By default all need to be present an AND
condition
TIME_LIMIT AMONG_REQUIRED_EVENTS Requires REQUIRE events to all occur within a specified time
window

CSL: CACL Logic for Block Commands

WHILE not done
Get next BTI trace event
IF no more events left for call, block FAILS
IF TIME_LIMIT SINCE_LAST_MATCH exceeded, block FAILS
IF event matches IGNORE pattern, go to top of loop
IF event matches REQUIRE pattern
IF TIME_LIMIT AMOUNG_REQUIRED_EVENTS exceeded block FAILS
IF REQUIRE_ONE block SUCCEEDS
Remove event from REQUIRE list
If REQUIRE list empty block SUCCEEDS
IF event matches PROHIBIT pattern, block FAILS
IF event matches ALLOW pattern, go to top of loop
ELSE event is unexpected, block FAILS

CSL: PROHIBIT_NEXT_BLOCK

PROHIBIT_NEXT_BLOCK

IGNORE TpU:MeasurementReportIndication

REQUIRE .


END_BLOCK
Just like REQUIRE_NEXT_BLOCK except block must
fail for scenario to continue processing Example: Require no events for the next 10 seconds,
except for TPU measurement reports

CSL: Scenario Commands for Scan Control
FORWARD_TIME_ORDER
- Process from beginning of call to the end
SCAN_ALL
- Once scenario passes/fails, try again until all call events processed NO_MATCH
- Continue to next scenario regardless of outcome of this one IGNORE_CALL_IF_MATCHED
- If scenario matches, immediately stop processing of this call
without generating any output

CSL: Scenario Commands for Output
DUMP_CALL_INFO_IF_MATCHED
- Dumps one-line info and BTI trace if scenario matches

- Typically used in last scenario (wild card) that handles
unclassified calls
DUMP_SUMMARY
- Dumps one-line info when encountered in scenario
- Typically used for procedure analysis to dump info
whenever a procedure variation is matched (perhaps
multiple times per scenario per call)
DUMP_CALL_INFO
- Like DUMP_SUMMARY but dumps BTI trace also

CSL: Other Scenario Commands
IGNORE_STARTING_HERE
- Ignore events matching pattern from this point on in scenario
- No need to specify pattern in event blocks

CSL: Example Call Drop Scenario (1 of 3)

SCENARIO CM: RNC RLC DCCH failure, no procedures, within 6.5 S of activation

CATEGORY Compressed mode: Failure during or shortly after procedure

TAG AR 1-1391211

SCAN_FOR_EVENT CnU:Iu-ReleaseRequest

VAR_STATE_RETRIEVE LASTNB_IURELREQ

REQUIRE_VAR $VAR{CM} eq 'ON'

CSL: Example Call Drop Scenario (2 of 3)

# Find the failure


# Allow cleanup

ALLOW RadioLinkDeletionRequest

ALLOW RadioLinkDeletionResponse

ALLOW TpD:DeleteUEContextRequest

ALLOW TpU:DeleteUEContextSuccess

# The failure

REQUIRE TpU:RadioBearerFailureIndication.*BSCTermRBSAP

# Allow subsequent L1/L2 failures

ALLOW UeU:cellUpdate

ALLOW RadioLinkFailureIndication

# Allow odds and ends

ALLOW NsD:

ALLOW NsU:

ALLOW DirectTransfer

ALLOW TpD:CreateCOChannelRequest

ALLOW TpU:DeleteCOChannelSuccess

END_BLOCK

CSL: Example Call Drop Scenario (contd)

# Find CM activation


REQUIRE UeD:measurementControl $interRAT activate$

ALLOW CompressedModeCommand

ALLOW RadioLinkReconfigurationCommit

# Allow multiple failures of the same type

ALLOW TpU:RadioBearerFailureIndication

# Allow odds and ends

ALLOW RadioLinkRestoreIndication

ALLOW NsD:

ALLOW NsU:

ALLOW DirectTransfer

ALLOW UeU:measurementReport \[e2d

ALLOW UeD:measurementControl

ALLOW TpD:CreateCOChannelRequest

ALLOW TpU:DeleteCOChannelSuccess


END_BLOCK

END_SCENARIO

CSL: Example Procedure Scenario (1 of 2)

SCENARIO Successful fallback to R99 on CM failure

NO_MATCH

FORWARD_TIME_ORDER

SCAN_ALL

SCAN_FOR_EVENT UeD:measurementControl \(interRAT activate


# Activate fails and we disable compressed mode

REQUIRE UeU:measurementControlFailure \(configurationIncomplete

REQUIRE UeD:radioBearerReconfiguration

REQUIRE CompressedModeCommand

REQUIRE UeU:radioBearerReconfigurationComplete

ALLOW .


END_BLOCK

CSL: Example Procedure
Scenario (2 of 2)


# Reconfigure to R99

REQUIRE RadioLinkReconfigurationPrepareFDD

REQUIRE RadioLinkReconfigurationReady

REQUIRE ConfigureUEContextReconfPSRABFromHSDSCHToDCHRequest

REQUIRE ConfigureUEContextReconfPSRABFromHSDSCHToDCHSuccess

REQUIRE radioBearerReconfiguration

REQUIRE RadioLinkReconfigurationCommit

REQUIRE radioBearerReconfigurationComplete

ALLOW .


END_BLOCK


REQUIRE UeD:measurementControl \(interRAT activate

ALLOW .


END_BLOCK

DUMP_SUMMARY

END_SCENARIO

CSL: Global Commands

CSL: Global Filtering Commands
GLOBAL_REQUIRE_RAB_TYPE GLOBAL_REQUIRE_RAB_STATE
RAB types are CS, PS, HS where HS is HSDPA

RAB states are REQUESTED, CONFIRMED, FAILED

For call drop analysis, restrict calls to those with a confirmed

RAB.

Handling of multi-RAB calls is TBD

CSL: Global Debug Commands
GLOBAL_DUMP_CALL_INFO_AT_START
- If present, dump BTI trace at start of each call analysis.
Used for general debugging
GLOBAL_DUMP_CALL_INFO_IF_MATCHED
- If present, dump call info including BTI trace when call matches.
Usually for debugging to check that matches are valid GLOBAL_DUMP_SUMMARY_ALL_ANALYSED_CALLS
- If present, dump one line summary of all calls after analysis.
Used to get the one-line summaries for call drop work.

CSL: Other Global Commands
GLOBAL_IGNORE_EVENT
- Use to ignore trace throughout processing of calls. Eliminates
needing to handle it in each individual scenario and event block

CSL: Variables

CSL: Variables
Variables can be used to collect state information about a call Useful for
- State checks within a scenario (e.g. ASU active)

- Post processing analysis (variables are included in
one-line summary output for a call)
All variables are global (not part of scenario) Besides user defined varialbles, there are several system defined
variables (e.g. RNTI) User variables should be declared with an initial value
(by default initial values is NIL) Variables can be set explicitly within scenarios, or scan
patterns can be defined Pattern scanning for variables is CPU-intensive; the results
of a scan can be saved and reused in subsequent scenarios Perl syntax is used for setting and comparing variables
- Trivial when constant values are used

CSL: Variable commands

VAR_INIT

- In global section

- Declare variable and set initial value

VAR_SCAN_SET TO on

- In global section

- Multiple allowed per variable

- As log events are scanned, sets variable on the first defined
pattern encountered
VAR_SCAN

- In a scenario

- Starting at current cursor in call, search for patterns
and set variables as specified. Variables only set for
the first matching pattern.


# Get the state of last active set update

VAR_INIT ASU NONE

VAR_SCAN_SET ASU TO 'COMPLETE' ON UeU:activeSetUpdateComplete

VAR_SCAN_SET ASU TO 'INPROG' ON UeD:activeSetUpdate

VAR_SCAN_SET ASU TO 'FAILED' ON UeU:activeSetUpdateFailure

More on variables
Scanning does not affect call trace cursor -- it is only the starting
point Scanning only sets variables equal to their initial value Scan direction is the same as direction of analysis (reverse time
if doing call drop analysis For call drop analysis, scanning is usually done at the point of
failure to determine the state of the call at time of failure
Simple example:


The TO value in VAR commands is a Perl expression:
If a constant value, enclose it in single quotes If a string containing Perl variables, enclose in double quotes If a complex expresssion use no quotes Perl ( ) and $N syntax can be used in pattern TO value
More complex example:

# Get the IMSI

VAR_INIT IMSI NIL

VAR_SCAN_SET IMSI TO "$1" ON CnD:CommonID $(\d+)$

Matches and captures IMSI value from:

CnD:CommonID (310410079762773)


More commands:

VAR_SCAN_SET TO

- Sets variable directly to (without any scanning) in response
on a VAR_SCAN command
VAR_SCAN

- Causes scanning for specified variable (IF it contains init value)

VAR_SET TO

- Sets variable to directly in a scenario

VAR_RESET
- Resets variable to its init value allowing scanning one more

CSL: Checking variables in scenarios
User variables can be accessed using the syntax
$VAR{var name} This can be used in the VAR_REQUIRE command in a scenario
VAR_REQUIRE

Example:

# Verify that a handover is in progress

REQUIRE_VAR $VAR{ASU} eq 'INPROG'

CSL: System variables
Some common information (variables) is maintained automatically These can be accessed using the syntax $SYSVAR{}
- RNTI
- START_TIME
- END_TIME
- LAST_MATCH_TIME
- LAST_MATCH_TIME_MS
- MS_SINCE_LAST_MATCH
- MS_SINCE_START

Example:

# Find and record time of release request

SCAN_FOR_EVENT Iu-ReleaseRequest
VAR_SET REQUEST_TIME $SYSVAR{LAST_MATCH_TIME}

CSL: Scenario Commands to Store/Retrieve Variable Set
VAR_SAVE_STATE
- Saves value of all user-defined variables
VAR_RETRIEVE_STATE
- Retrieves value of all user-defined variables previously saved

Used to avoid having to rescan for to set variables for each scenario

- This is CPU intensive and repetitive
Scan is done once in a dummy scenario and saved
Subsequent scenario retrieves these variable values

Writing Scenarios
Think about the event sequences that can be handled with
one event block
- Can occur in any order
- A single time limit is acceptable Using an ALLOW . command in an event block lets you avoid
having to anticipate any extraneous event that might occur in
a call, but can also allow matching of unintended sequences Use debug options (-d and r) to trace processing as a
scenario is applied to a specific RNTI Look at samples

call trace analyser

Documents