guide to troubleshooting paging failures-20130110-v1.0

Guide to Troubleshooting Paging Failures

Issue 1.0

Date 2012-12-05

HUAWEI TECHNOLOGIES CO., LTD.

Copyright © Huawei Technologies Co., Ltd. 2012. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.

All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.

Huawei Technologies Co., Ltd.

Address: Huawei Industrial Base

Bantian, Longgang

Shenzhen 518129

People's Republic of China

Website: http://www.huawei.com

Email: [email protected]

Issue 01 (2012-12-05) Huawei Proprietary and Confidential Copyright © Huawei Technologies Co.,

Ltd.

i

mailto:[email protected]

http://www.huawei.com/

Guide to Troubleshooting Paging Failures About This Document

About This Document

Author

prepared By

UMTS Maintenance Department Date

2012-10-8

Reviewed By

Zheng Jianhua (employee ID: 00042104)

Qian Jin (employee ID: 00064231)

Zhu Peng (employee ID: 00058079)

Date

2012-10-25

Approved By

Fang Ming (employee ID: 00061579) Date

2012-10-27

Change History

Date Revision version

change Description Author

2012-10-25 0.9 Completed the initial draft Zeng Wei (employee ID: 00161296)

2012-10-30 1.0 Completed the Modification

Zeng Wei (employee ID: 00161296)

KeywordsPaging, PCH congestion, paging delay


Ltd.

ii

Guide to Troubleshooting Paging Failures About This Document

AbstractThis document describes how to troubleshoot paging failure problems by checking and analyzing all rounds in different network operation scenarios, guides frontline personnel to analyze networks, solve paging failure problems, and separate work of network optimization personnel, technical service personnel, and Research & Development (R&D) personnel from each other, and provides fixed actions to discover factors affecting paging success rates. The primary goal of this document is to guide frontline personnel to quickly handle simple problems and effectively feed back difficult problems to R&D personnel, and improve the troubleshooting efficiency.


Ltd.

iii

Guide to Troubleshooting Paging Failures Contents

Contents

About This Document........................................................................ii

1 Overview.......................................................................................1

2 Clarification of Paging Problems.....................................................22.1 Definitions of Paging KPIs in Traffic Statistics.................................................................................................2

2.1.1 KPIs of Paging Success Rates on the CN Side.........................................................................................2

2.1.2 KPIs of Paging Success Rates on the RNC Side......................................................................................2

2.2 Sources and Clarification of Problems..............................................................................................................2

3 Troubleshooting Process for Paging Failures...................................43.1 Common Scenarios of Paging Failures and Possible Causes............................................................................4

3.1.2 Categorizing Common Causes and Corresponding Actions of Paging Failures by Application Scopes. 5

3.1.3 Classifying Paging Failures based on Paging Procedures........................................................................7

3.2 Analysis Report for Paging Problems................................................................................................................8

4 Description of Troubleshooting Procedures for Paging Failures......104.1 Action 1: Confirming the Type of a Paging Failure Procedure........................................................................10

4.1.1 Descriptions............................................................................................................................................10

4.1.2 Data Sources and Processing Tools........................................................................................................10

4.1.3 Results....................................................................................................................................................10

4.2 Action 2: Checking Parameters........................................................................................................................10

4.2.1 Checking Parameter Mapping................................................................................................................10

4.2.2 Checking Reasonableness of Parameter Configurations........................................................................11

4.2.3 Checking Parameter Configuration Modifications (Operation Checking).............................................13

4.3 Action 3: Checking Device and Transmission Faults (Performed by Frontline Engineers)............................14

4.3.1 Checking Alarms....................................................................................................................................14

4.3.2 Checking Transmission Fault Alarms.....................................................................................................15

4.4 Action 4: Checking Paging Channel Congestion.............................................................................................18

4.5 Action 5: Checking Versions............................................................................................................................18

4.5.1 Analyzing RNC and NodeB Version Difference (Performed by R&D Personnel with the Assistance of Frontline Personnel)........................................................................................................................................18

4.5.2 Checking RNC and NodeB Versions for Known Problems (Performed by Frontline Engineers).........19

4.6 Action 6: Checking SPU Loads.......................................................................................................................20

4.6.1 Descriptions............................................................................................................................................20


Ltd.

iv



4.6.3 Steps for Analyzing Data........................................................................................................................20

4.6.4 Results....................................................................................................................................................21

4.6.5 Closed Actions........................................................................................................................................21

4.7 Action 7: Checking RRC Setup Success Rates of Called Parties and Registered Users.................................22

4.7.1 Descriptions............................................................................................................................................22


4.8 Action 8: Checking Coverage (Performed by Network Optimization Engineers)..........................................22

4.8.1 Descriptions............................................................................................................................................22



4.8.4 Results....................................................................................................................................................23

4.8.5 Closed Actions........................................................................................................................................23

4.9 Action 9: Comparing Long-Term Change Trends of Traffic Volumes (Seasonal Changes and Increase in the Number of Users)...................................................................................................................................................24

4.9.1 Descriptions............................................................................................................................................24



4.9.4 Results....................................................................................................................................................24

4.9.5 Closed Actions........................................................................................................................................25

4.10 Action 10: Checking Unexpected Factors (Charging, Weather, and Gatherings)..........................................25

4.10.1 Descriptions..........................................................................................................................................25

4.10.2 Data Sources and Processing Tools......................................................................................................25

4.10.3 Steps for Analyzing Data......................................................................................................................25

4.10.4 Results..................................................................................................................................................26

4.10.5 Closed Actions......................................................................................................................................26

4.11 Action 11: Checking Exceptions on the CN Side..........................................................................................26

4.11.1 Descriptions..........................................................................................................................................26



4.11.4 Results...................................................................................................................................................27

4.11.5 Closed Actions......................................................................................................................................27

4.12 Action 11: Analyzing UE Exceptions and Trace............................................................................................27

4.12.1 Descriptions..........................................................................................................................................27



4.12.4 Results..................................................................................................................................................28

4.12.5 Closed Actions......................................................................................................................................28

5 Cases of Required Actions for Paging Problems.............................295.1 Cases of Parameter Checking..........................................................................................................................29

5.2 Cases of Paging Success Rates Caused by UE Exceptions.............................................................................30

5.3 Cases of Paging Success Rate Deterioration Caused by Incorrect Configurations on the CN Side................30


Ltd.

v



Ltd.

vi

Guide to Troubleshooting Paging Failures 4 Overview

1 Overview

This document focuses on analyzing and resolving paging failures during the initial optimization of new and swapped networks and guides frontline personnel to rapidly analyze and optimize networks to ensure that paging problems can be rapidly located and solved, improving efficiency. Differences may come with tools, traffic statistics, parameters, and versions. However, analysis methods described in this document are applicable to various base station controller (BSC) versions.


Ltd.

1

Guide to Troubleshooting Paging Failures 4 Clarification of Paging Problems

2 Clarification of Paging Problems

Paging success rate, one of the network performance indicators (KPIs), indicates paging failures. The evaluation for paging success rates is based on paging success rates on the core network (CN). Paging success rates on the UTRAN side are auxiliary counters for locating problems.

2.1 Definitions of Paging KPIs in Traffic Statistics

2.1.1 KPIs of Paging Success Rates on the CN Side 3G Paging Success Rate (%) = (Number of First Paging Responses from Iu Interface + Number of Repeated Paging Responses from Iu Interface) / Number of First Paging to Iu Interface

Paging success rates on the CN side is mainly measured in LAC areas. The preceding formula shows that paging success rate KPIS on the CN side presented to customers are not First Paging Success Rate KPIs.

2.1.2 KPIs of Paging Success Rates on the RNC Side VS.RANAP.Paging.Succ.IdleUE.Rate

= VS.RANAP.Paging.Succ.IdleUE/VS.RANAP.Paging.Att.IdleUE

On the UTRAN side, only this preceding formula is used to measure paging success rates. Paging KPIs are related to two NEs, therefore, confirm KPIs of the two NEs when resolving paging failures. Action 1: confirming the type of a paging failure procedure.

2.2 Sources and Clarification of ProblemsSource: Paging success rates deteriorate due to network swapping, complaints of VIP users, NE upgrade or feature enabling.

The following table lists sources of paging failures.


Ltd.

2

Guide to Troubleshooting Paging Failures 4 Clarification of Paging Problems

Source of Paging Failures

Data Source Analysis Criteria

Unqualified paging success rate of swapped or new networks

KPI mapping relationships between peer vendors and Huawei before and after swapping

Traffic statistics KPIs between peer vendors and Huawei before and after swapping

Paging success rates on the CN side before swapping

1. Analyze KPI mapping accuracy

including denominators and

numerators) and check that KPIs

are proper.

2. Compare traffic statistics for at

least one week before and after

swapping and check whether the

traffic statistics meet related

requirements.

Deterioration of paging success rates of inventory networks

Original traffic statistics KPIs before and after deterioration

1. Analyze impacts from upgrading

and feature enabling scenarios on

paging success rates.

2. Determine deterioration by

comparing the corresponding

days.

3. Compare KPIs three days before

and after deterioration and

exclude impacts of traffic

periods.

VIP users complain about calling difficulties or long delay.

Drive test (DT) data and traced signaling

Signaling analysis shows that paging failures occur, which is due to causes on the wireless network rather than the CN or UEs.


Ltd.

3

Guide to Troubleshooting Paging Failures 4 Troubleshooting Process for Paging Failures

3 Troubleshooting Process for

Paging Failures

3.1 Common Scenarios of Paging Failures and Possible Causes

Paging failures can be categorized based on application scenarios and paging procedures, as described in the following table.

Table 3-1 Paging failures categorized based on application scenarios

Dimension Paging Problem Type

Description Criteria

Application scenario Unqualified paging success rates on swapped networks

Unqualified paging success rates on the CN side peer vendor's swapped networks

Compare traffic statistics within the latest one week before and after swapping.

Unqualified paging success rates on new networks

Unqualified paging success rates in LAC areas of new networks

Observe whether traffic statistics for at least one week meet related requirements.

Deterioration of paging success rates of inventory networks

On the inventory networks with stable KPIs, paging success rates suddenly deteriorate. Paging success rate deterioration includes non-operation deterioration and operation deterioration (version upgrades, parameter modification, cutover, and feature activation).

Compare paging success rates of the corresponding days in a week (three days before and after deterioration).


Ltd.

4


3.1.2 Categorizing Common Causes and Corresponding Actions of Paging Failures by Application Scopes

Paging Success Rates on Swapped Networks Do Not Meet Related Requirements.

Paging success rate KPIs may not be available on the UTRAN side of peer vendors' networks, therefore, take paging success rates on the CN side to verify whether they meet the standard.

After swapping, paging success rates are lower than those on the original networks. Common reasons for this problem are as follows:

1. Algorithm/Parameter mapping problems: including incorrect mappings of RNC-level and site-level parameters. In addition, some algorithms and parameters of Huawei devices are not accurately mapped with those of original networks. This may result in changes of traffic models, calling control processes, resource allocation, wireless coverage, and network traffic distribution, affecting paging performance. (action 2)

2. Hardware and transmission faults: alarms of device board exceptions and transmission faults. (action 3)

3. NE versions: NE versions are not properly selected (an early version has many problems) or version defects result in low paging success rates. (action 5)

4. Network planning: neighboring cell inheritance errors, LAC/RAC area planning differences, and insufficient paging channels over air interfaces result in low paging success rates. (action 4)

5. RF channel problems: Antenna system faults, RF connection faults, antenna match, antenna tilt adjustment, and internal and external interference of the antenna system result in low paging success rates. (action 6)

6. Long-term impact: In the scenarios with a long swapping period of time, seasonal changes in traffic models and KPIs take place. In this case, increased traffic volumes result in air interface capacity limitation, causing paging success rate deterioration. (action 9)

7. Impacts of external unexpected factors: For example, during swapping or after swapping, user actions and traffic distribution greatly change due to external factors such as charging adjustments of operators, holidays, activities, and climates. Therefore, paging success rates after swapping are affected. (action 10)

8. Exceptions on the CN side: In a short period of time, paging success rates may deteriorate suddenly or fluctuate dramatically. After the RAN-side impacts are eliminated, this problem may be caused by exceptions on the CN side, including exceptions in operations and terminal servers (for example, faults of iTunes and BlackBerry servers may increase the number of paging users), and IP or port scanning in networks in the case of allocating IP addresses to UEs by the CN. (action 11)

9. Abnormal UEs and users: Poor compatibility of UEs, business users, and malicious users result in paging failures. (action 11)

Paging Success Rates of New Networks Do Not Meet Related Requirements.

Common causes are as follows:


Ltd.

5


1. Inappropriate network parameter planning: algorithm/parameter planning, network policy planning, and improper neighboring cell and frequency planning result in low paging success rates. (action 2)

2. Hardware and link faults: Devices, board, and transmission faults result in low paging success rates. (action 3)

3. NE version problems: NE versions are not properly selected (an early version has many problems) or precautions actions are not performed, which results in low paging success rates. (action 4)

4. Network planning optimization problems: Inappropriate site planning, networking inheritance policy planning, LAC/RAC planning, air interface resources, and board resources cause coverage problems, affecting paging success rates. (action 5)

5. RF channel problems: Internal and external inference and RF channel connection faults cause uplink and downlink coverage problems, affect paging success rates. (action 6)

6. Impacts of external unexpected factors: For example, during swapping or after swapping, user actions and traffic distribution greatly change due to external factors such as charging adjustments of operators, holidays, activities, and climates. Therefore, paging success rates on new networks are affected. (action 9)

7. Exceptions on the CN side: In a short period of time, paging success rates may deteriorate suddenly or fluctuate dramatically. After the RAN-side impacts are eliminated, this problem may be caused by exceptions on the CN side, including exceptions in operations and terminal servers (for example, faults of iTunes and BlackBerry servers may increase the number of paging users), and IP or port scanning in networks in the case of allocating IP addresses to UEs by the CN. (action 10)

8. Long-term trend impact: Seasonal changes in traffic models and KPIs affect paging success rates to a certain extent. For example, based on historical experience, the time when new networks are deployed, paging success rates are the lowest throughout the year. (action 8)


Paging Success Rates of Inventory Networks Deteriorate


1. Parameter adjustment impacts: Inappropriate algorithms, parameters, network policies, and neighboring adjustment result in paging success rate deterioration. In the long-term network operation, traffic models and distribution gradually change. Original algorithms and parameters do not adapt to the current network status, which results in slow paging success rate deterioration. (action 2)

2. Hardware and link faults: Transmission, device board faults or adjustment result in sudden paging success rate deterioration. (action 3 and action 10)

3. Network planning problems: Inappropriate site planning, network policies affect network structures and coverage, resulting in paging success rate deterioration. In the long-term network operation, traffic models and distribution gradually change and inappropriate site planning, antenna design, and power planning cause coverage and capacity problems, affecting paging success rates. (action 5)

4. RF channel problems: Sudden external interference and internal interference such as antenna system faults and looseness causes sudden paging success rate deterioration. (action 6)

5. Long-term trend impact: Seasonal changes in traffic models and KPIs results in slow fluctuation in paging success rates.


Ltd.

6


6. Impacts of external unexpected factors: For example, during swapping or after swapping, user actions and traffic distribution greatly change due to external factors such as charging adjustments of operators, holidays, activities, and climates. Therefore, paging success rates deteriorate after swapping.

7. Exceptions on the CN side: In a short period of time, paging success rates may deteriorate suddenly or fluctuate dramatically. After the UTRAN-side impacts are eliminated, this problem may be caused by exceptions on the CN side, including exceptions in operations and terminal servers (for example, faults of iTunes and BlackBerry servers may increase the number of paging users), and IP or port scanning in networks in the case of allocating IP addresses to UEs by the CN. (action 10)

8. NE and version changes: NE replacement and version upgrades result in sudden paging success rate deterioration.(action 4)


VIP users Complain about Calling Difficulties or Long Delay.

For the preceding problems, contact frontline personnel to collect related data and feed back these problems to R&D engineers.

3.1.3 Classifying Paging Failures based on Paging Procedures


Ltd.

7


Type 1: Paging Success Rates Drop Dramatically on the CN Side But Those in Idle Mode on the RNC Side Remain the Same.


Paging messages are discarded over the Iu interface.

1. RRC setup success rates of the called party decrease.

2. RRC is successfully set up but no Initial Direct Transfer message is received on the CN.

Type 2: Paging Success Rates on the CN Side and Those in Idle Mode on the RNC Side Drop Dramatically.


1. Paging messages are not sent over air interfaces due to internal problems within the RNC.

2. Paging congestions occur because of air interface capability limitation (VS.RRC.Paging1.Loss.PCHCong.Cell).

3. Paging success rate decrease may also result from weak coverage, paging black hole cells, NodeB faults, and cell out-of-service problems.

4. Registered RRC setup success rates decrease.

5. RRC is successfully set up but LAU/RAU procedure fails.

6. Location upgrade of UEs in idle mode or inter-RAT cell reselection is frequent. UEs in CELL-PCH mode frequently reselect cells.

7. Some UEs receive no paging response messages from the air interfaces in shared networks.

8. Some parameters are improperly set, therefore, UEs cannot respond to paging messages in a certain period.

Type 3: Paging Success Rates are Normal on the CN Side But Those on the UTRAN Side Deteriorate.


1. UEs are not located in areas under the local RNC.

2. The first paging success rate on the CN side decreases but repeated paging success rate increases.

3. The CELL-PCH/FD/EFD feature is enabled on the RNC.

3.2 Analysis Report for Paging ProblemsPaging problems are analyzed and handled based on the preceding troubleshooting procedures. Based on the detailed description of required actions for RRC and RAB paging problems, frontline personnel perform analysis and checking required in 11 required actions and provide an analysis report. If frontline personnel experience any trouble in analyzing and handling paging problems, they can turn to PSEs and R&D personnel for help. The report template is as follows:


Ltd.

8


Provide the analysis report to PSEs and R&D personnel for review and specify analysis conclusion, handling measures and optimization suggestions. Analysis and checking in 11 required actions are described in the following several chapters.


Ltd.

9

Guide to Troubleshooting Paging Failures 4 Description of Troubleshooting Procedures for Paging Failures

4 Description of Troubleshooting

Procedures for Paging Failures

4.1 Action 1: Confirming the Type of a Paging Failure Procedure

4.1.1 DescriptionsLocate paging failures roughly and confirm handling orientation according to paging success rate KPIs on the CN and UTRAN side.

4.1.2 Data Sources and Processing ToolsData sources: traffic statistics on the UTRAN and CN side.

Tools: OMStar and FMA.

4.1.3 Results Verify whether the current paging problem is one of the followings:

1. Type 1: If paging success rates on the CN side decrease but hardly change on the UTRAN side, it is an Iu interface paging problem.

2. Type 2: If paging success rates decrease on both the CN and UTRAN side, it is a Uu interface paging problem.

3. Type 3: Paging success rates hardly change on the CN side but decrease dramatically on the UTRAN side, it is an improper parameter setting problem.


Ltd.

10


4.2 Action 2: Checking Parameters

4.2.1 Checking Parameter Mapping

Descriptions

Scenario: Paging success rates on swapped networks do not meet the requirements. The problems are preceding type 1 and type 2 problems.

Criteria: Check parameter mapping if paging success rates on the CN or UTRAN side after swapping do not meet the requirements.

Data Sources and Processing Tools

Data Source Processing Tool

1 Huawei MML configuration files, peer vendors' MML configuration files, and parameter lists.

Tools provided in parameter mapping table

Steps for Analyzing Data1. Obtain parameters of peer vendors for checking parameter mapping based on Huawei

mapping system or provide the review conclusion of initial-phase parameter mapping.

For details about the parameter mapping process, see xxx.

Related parameter/feature mapping guides are incorporated into High Efficiency UMTS Swap Solution (V2.0), which is available at

http://wss.huawei.com/sites/gts/gul/Shared%20Documents/Forms/AllItems.aspx

2. Check the consistency between mapping results and actual configurations based on mapping system and current configurations.

Results 1. Provide parameter mapping review result and list parameters that cannot be accurately

mapped.

2. Provide the consistency check result between the parameter mapping review and actually configured parameters and list inconsistent parameters.

Closed Actions1. Analyze the impact of parameters that cannot be accurately mapped and adjust

parameters to observe the effect.

2. Analyze the impact of actual parameters that are not consistent with mapping parameters and adjust parameters to observe the effect.


Ltd.

11


4.2.2 Checking Reasonableness of Parameter Configurations

Descriptions

Scenario: optimization scenario. The problems are preceding type 1 and type 2 problems.

Criteria: Check reasonableness in the paging success rate optimization scenario when the paging success rate slowly deteriorates and optimization is required.



1 Configuration files FMA

Steps for Analyzing Data

For details, see UPSS V100R003C00 Guide to Full Parameter Check.

Results

Based on the rule for checking parameter reasonableness, provide the check result of paging-related parameters and list the current parameter values and values specified in the check rule.

After enabling FMA function for checking parameters, a file with a name extension of .csv is created in the target folder. The comparison results between parameter values of live networks and baseline values are listed in column I and comparison results between parameter values of live networks and parameter reasonableness results are listed in column J.


Ltd.

12


For details, see UPSS V100R003C00 Guide to Full Parameter Check.

Closed Actions

Modify parameters based on check results of improper paging-related parameters in the provided .csv file.

4.2.3 Checking Parameter Configuration Modifications (Operation Checking)

Descriptions

Scenario: Paging success rates suddenly deteriorate due to upgrade or operation. The problems are preceding type 1 and type 2 problems.

Criteria: Check operations and parameter modifications when the paging success rate suddenly deteriorates.



1 Configuration files before and after paging success rate deterioration

CME

2 Operation logs FMA

Steps for Analyzing Data1. Based on the time when problems occur, analyze the operation logs on the RNC and

NodeB before, during, and after the problems take place, altogether three days. Analyze only logs related to operations such as MOD, RMV, SET, ADD, RST, ACT, DEA, and SWP. All RNC-class operation logs must be analyzed.)

2. Compare the configuration files before and after deterioration to analyze parameter differences and provide all-parameter check differences.

Results 1. Provide a check list of operation logs and list non-query operation and their impacts.

2. Provide comparison results of all-parameter differences.


Ltd.

13


OP Time CMD Name OP CMD Name

OP Object

OpCmd&Message

Impact

2012-03-22 11:00

　　

2012-03-23 11:00

　　

2012-03-24 11:00

　　

2012-03-25 11:00

　　

The following table lists examples:

OP Time CMD Name

OP CMD

OP Object

OpCmd&Message

Impact

2012-10-20 11:05

Set the DPU configuration data.

SET UDPUCFGDATA

RNC SET UDPUCFGDATA: MaccPageRepeatTimes=2;

Paging messages over air interfaces may increase and paging congestions may occur.

Closed Actions1. Analyze the impact of operation and parameter differences and provide corresponding

solutions.

2. Perform action 3 if no obvious operation causes paging success rate deterioration.

4.3 Action 3: Checking Device and Transmission Faults (Performed by Frontline Engineers)

4.3.1 Checking Alarms

Descriptions

Scenario: Check all scenarios where paging success rates decrease, including deterioration or optimization scenarios. The problems are preceding type 1 and type 2 problems.

Criteria: According to the problem type confirmed in action 1, analyze site alarms in LAC areas (including site alarms and site-related RNC alarms) and entire RNC alarms (such as interface board alarms and SPU alarms).


Ltd.

14




1 Alarm logs (RNC and NodeB) FMA


According to the problem scope confirmed in action 1, analyze RNC alarm logs and site alarm logs within the LAC areas.

For details about alarm information and check methods, see alarm description in product documents and alarm clearing guides released by the maintenance department. Network optimization engineers need only to submit device and link alarms to device maintenance engineers.

Results

Provide analysis results of device faults and alarms, including alarm lists, cell lists, and alarm checking results.

Closed Actions

Rectify device faults and clear alarms. Observe KPI recovery status.

4.3.2 Checking Transmission Fault Alarms

Descriptions

Scenario: Check all scenarios where paging success rates decrease, including deterioration or optimization scenarios. The problems are preceding type 1 and type 2 problems.

Criteria:

1. The confirmed problem type is type 1, namely, transmission alarms are generated over the Iu interface.

2. The confirmed problem type is type 2 in action 1, namely, transmission alarms are generated on NodeBs in LAC areas.



1 Traffic statistics FMA and OMStar

2 Configuration files FMA and OMStar

3 Real-time test data None


Ltd.

15



Check transmission link configurations

Check transmission parameters of the RNC and NodeB by using the link status checking function of the maintaining SOP. Check the transmission consistency of the paging failure top cells (manual checking is required because no tools temporarily support the checking).

Some parameters are not properly configured by checking link configurations of the maintaining SOP link, as shown in the following figure:

Confirm configuration problems onsite as soon as possible.

Check the transmission link status.

Check link status of the control plane by using the link status checking function of the maintenance SOP. Link status of the user plane needs manual checking.

The following table lists transmission link status by using the maintenance SOP.

Confirm whether the link status is normal onsite as soon as possible.

Check the transmission link load.

Evaluate and analyze the link load by using the link load evaluation function of the maintenance SOP.

The following table lists the evaluation results of transmission link load.


Ltd.

16


The evaluation result shows that transmission congestion exists in many Iub interfaces.

Confirm the link congestion status onsite as soon as possible.

Check transmission link quality

The maintenance SOP supports IP transmission quality check and exports corresponding results. If transmission is in IP mode, enable the ping checking function of the IPPATH and evaluate transmission link quality using the IP transmission QoS checking function of the maintenance SOP. For the ATM transmission quality, only Iub interface has the PM function and the quality of the Iu/Iur interface cannot be checked.

The following table lists the link load evaluation result of the maintaining SOP link.


Ltd.

17


The following table shows that the IPPATH transmission quality of some NodeBs is poor.

Confirm whether the intermediate transmission network has problems onsite.

The following document provides the guide to manually checking transmission.

See RAN12 IP QoS Transmission Trouble Shooting Guide.

Results

Provide transmission checking items and results.

Closed Actions

Rectify transmission faults and observe KPIs.

4.4 Action 4: Checking Paging Channel Congestion

Descriptions

Scenario: Version upgrade results in paging KPI deterioration.

Criteria: KPIs slowly deteriorate after version upgrading or KPIs suddenly deteriorate within one week following version upgrading.



1 Traffic statistics OMStar

3 MML configuration files FMA

Steps for Analyzing Data1. Confirm the value of VS.RRC.Paging1.Loss.PCHCong.Cell based on traffic statistics

and the value of IU.Paging.Congestion.Cell.Ratio is greater than 1%.

2. The calculation shows that the modified paging channel utilization in all cells is more than 80% and physical channel utilization is greater than 25%. The formula is as follows:

Modified paging channel utilization = (VS.PCH.Utilization.Cell.Rate x 6)/7

Physical channel utilization = VS.UTRAN.AttPaging1/2160000


Ltd.

18


Results

Provide values of preceding counters in all cells.

Closed Actions

Take optimization measures by referring to UMTS Access and Paging Principles + Troubleshooting Methods + Cases + Deliverables. Some measures and actions have both positive and negative gains. Therefore, confirm the impact with R&D personnel before taking actions.

4.5 Action 5: Checking Versions

4.5.1 Analyzing RNC and NodeB Version Difference (Performed by R&D Personnel with the Assistance of Frontline Personnel)

Descriptions

Scenario: Version upgrade results in paging KPI deterioration.

Criteria: KPIs slowly deteriorate after version upgrading or KPIs suddenly deteriorate within one week following version upgrading.



1 Release Notes (RNs) None

2 MML files before and after the version upgrade

Beyond Compare and OMSTar

Steps for Analyzing Data1. Analyze RNs of RNCs/NodeBs with upgrade paging success rate before and after

deterioration. List version incorporation problems, algorithm change, new features, and the possible effects.

2. Compare RNC/NodeB configuration files to list the parameter configuration differences, and the possible impact of each difference.

Results 1. Provide impact analysis results of version incorporation problems, algorithm changes,

and new features.

2. Provide analysis results of version parameter difference impacts.


Ltd.

19


Closed Actions1. Provide a solution to version problems affecting the paging success rate and observe the

results after implementing measures.

2. If you cannot directly draw any conclusion from version differences, analyze the specific failure range and cause.

4.5.2 Checking RNC and NodeB Versions for Known Problems (Performed by Frontline Engineers)

Descriptions

Scenario: deterioration and optimization scenarios

Criteria: Confirm the problem range and cause and eliminate impacts from parameter changes based on actions 1 and 2. Eliminate the impacts of device and link defaults. Check the known version problems.




2 Case database None

3 Release Notes (RNs) None

Steps for Analyzing Data1. Obtain xx to check whether known version problems occur.

2. Analyze the latest RN to confirm whether there are version defects deteriorating the paging success rate based on problem range, initial problem causes, and network operations.

3. Check the case database to confirm whether there are known problems deteriorating paging success rate based on problem range, initial problem causes, and network operations.

Results1. Provide the version RN checking result: List the name of the checked RNs and check

result. (List the RN defect descriptions if related defects exist.)

2. Provide case database searching results: List keywords for searching and checking result. (List the link and content of cases if there are any related cases.)

Closed Actions1. Confirm related RN defects, provide and perform corresponding solutions. Record

results.

2. Confirm associated cases and perform solutions. Record results.


Ltd.

20


4.6 Action 6: Checking SPU Loads

4.6.1 DescriptionsScenario: all deterioration or optimization scenarios of paging problems of type 1. Paging problems of type 2 are caused by decreased RRC success rates of registered RRCs.

Criteria: Check SPU loads when the VS.Paging.FC.Disc.Num.CPUS counter discarded over the Iu interface has a value.

4.6.2 Data Sources and Processing Tools



2 PCHR FMA

3 DEBUG, CallFault, and CHR FMA

4.6.3 Steps for Analyzing Data1. Confirm whether values are available for VS.RANAP.CsPaging.Loss or

VS.RANAP.PsPaging.Loss or VS.Paging.FC.Disc.Num.CPUS.

2. Evaluate network loads using CMA scripts of an OMStar and generate a solution. Import traffic statistics in the OMStar to run scripts.

CMA scripts and methods for running the scripts are described in the following attachment:

4.6.4 ResultsProvide evaluation results in Excel using related tools.

SPU loads are described in the following figure:


Ltd.

21


Solutions for SPU loads are described in the following figure:

4.6.5 Closed ActionsSolutions generated from tools usually involve swapping sites, changing binding relationships, and expanding capacity. Observe the changes in paging success rates and the following three counters including VS.RANAP.CsPaging.Loss, VS.RANAP.PsPaging.Loss, and VS.Paging.FC.Disc.Num.CPUS.

4.7 Action 7: Checking RRC Setup Success Rates of Called Parties and Registered Users

4.7.1 DescriptionsScenario: Paging failures of type 1 are due to low RRC success rates of the called parties and failures of type 2 are caused by low RRC success rates of the registered users.

Criteria: Traffic statistics show that RRC success rates of the called parties or registered users are low.

4.7.2 Steps for Analyzing DataSee required actions and handling measures for RRC access success rates in Guide to Troubleshooting UPSS V100R003C00 Access Failure Problems.


Ltd.

22


4.8 Action 8: Checking Coverage (Performed by Network Optimization Engineers)

Radio link coverage is a key to network performance. Coverage problems, such as, poor or overlapped coverage would affect network access performance. In this action, try to find out poor network RF coverage, excluding the poor coverage caused by RF channels.

4.8.1 DescriptionsScenario: all deterioration and optimization scenarios where paging failures of type 2 occur.

Criteria:

According to action 1, it is a paging problem of type 2. No exceptions are found in previous checking.

In respect of poor uplink and downlink coverage, check the coverage after checking parameters, devices, and RF channels.





3 Drive test data Actixe or corresponding data processing tools

4.8.3 Steps for Analyzing Data1. See seven coverage-related analyses in MTS Network Planning and Optimization

Principles + Troubleshooting Methods + Cases + Deliverables.The detailed analysis is described in the following table.

Checking Parameters

Trigger conditions: mandatory. Check the following items:

Networking policy

Power

Handover setting

Parameters inconsistency before and after swapping or engineering adjustment

Checking coverage

Trigger conditions: mandatory. Check coverage problems caused by insufficient sites and carriers, and poor indoor coverage. Such coverage information can be used to monitor overall coverage.

Checking traffic statistics

Trigger conditions: mandatory. Check coverage problems caused by high traffic volumes. Traffic statistic can also be used to monitor the overall situation of traffics.


Ltd.

23


Analyzing basic RF information

Trigger conditions: mandatory. Check coverage problems caused by missing neighboring cell configuration and improper parameter setting (delayed handover). Identify areas with pilot pollution and high RTWP. Solve pilot pollution problems by combining parameter checking. For high RTWP problems, check RF quality, external interference, and parameters by combing channel check and capacity check.

Analyzing drive test data

Trigger conditions: This action is required after network swapping or engineering adjustments or when problems must be precisely located. Coverage problems basically can be identified. The prerequisite for identifying coverage problems is to check RF quality, external interference, and parameters by combing channel check and capacity check.

Analyzing MRs

Trigger conditions: This action is required when the MR function is enabled on live networks. Combine checking for insufficient sites and parameter problems (pilot pollution) to identify areas with high traffic volume.

Emulating the system

Trigger conditions: This action is required in engineering adjustment. Check coverage problems caused by parameter problems, for example, parameter adjustment.

4.8.4 ResultsProvide an analysis report based on the coverage-related analysis in UMTS Network Planning and Optimization Principles + Troubleshooting Methods + Cases + Deliverables.

4.8.5 Closed ActionsIf coverage changes dramatically in comparison and affects paging success rates, exclude impacts from parameter changes and RF channel changes. Perform adjustment after confirming the coverage changes. Network optimization engineers provide specific measures as required.

4.9 Action 9: Comparing Long-Term Change Trends of Traffic Volumes (Seasonal Changes and Increase in the Number of Users)

Paging success rates may come with similar fluctuation or long-term deterioration throughout the year or in two years.

If the deterioration trends of paging success rate are similar for two years, this is related to seasons, grand festivals, gatherings, and traveling seasons, which can be clarified by analysis.

If paging success rate deterioration persists for long period of time, analyze whether this is related to persistent increase in the number of users. If yes, provide corresponding measures.

4.9.1 DescriptionsScenario: Paging success rates slowly deteriorate or fluctuate in a long term.


Ltd.

24


Criteria: Paging success rates have long-term deterioration or slow deterioration instead of obvious deterioration in a long period.



1 Traffic statistics PRS

Nastar

M2000

4.9.3 Steps for Analyzing DataAnalyze the long-term change trend of the following counters.

1. Number of RRC attempts of the called party

2. VS.RRC.Paging1.Loss.PCHCong.Cell

3. VS.RANAP.CsPaging.Loss and VS.RANAP.PsPaging.Loss

4.9.4 ResultsProvide long-term change trend results and an association analysis result of the preceding counters, specifically, provide the relationship between paging success rates and the following information:

Number of RRC attempts of the called party

VS.RRC.Paging1.Loss.PCHCong.Cell

VS.RANAP.CsPaging.Loss

VS.RANAP.PsPaging.Loss

Season change trend

Network adjustment time

Increase time in the number of abnormal terminals

Networks accessing time of new users

4.9.5 Closed Actions1. If paging success rate changes are caused by seasonal changes in traffic volumes or

environment, provide analysis reports and explain this situation to customers.

2. Take the following measures to resolve the paging success rate deterioration that are caused by increase in the number of users.

a Analyze whether radio resource bottlenecks exist. If the bottlenecks exist, expand capacity and balance traffic.

b Go to Action 12: Analyzing UEs Exceptions and Trace if resource bottlenecks do not exist.


Ltd.

25


4.10 Action 10: Checking Unexpected Factors (Charging, Weather, and Gatherings)

In a short period of time, the paging success rate may deteriorate suddenly or fluctuate obviously. If the paging success rate changes are not caused by network adjustments, the changes may be caused by unexpected factors, such as charging policy adjustment, bad weathers, festivals and gathering activities. Provide measures based on whether these unexpected factors can be expected or controlled on networks.

4.10.1 DescriptionsScenario: Paging success rate deteriorates suddenly in a short period of time.

Criteria: Network adjustment impacts, device faults and channel faults are eliminated in actions 2, 3, and 8 respectively.




4.10.3 Steps for Analyzing Data1. Analyze the long-term change trend of paging success rates and traffic volumes, which is

applicable to short-term unexpected changes of paging success rates.

2. Analyze operation records and network adjustment records to check whether unexpected paging success rate deterioration is caused by network adjustment.

3. Analyze whether unexpected events, such as, charges, festivals and gathering activities, happen simultaneously or continuously when, before or after the paging success rate changes suddenly if network adjustment is not considered.

4.10.4 ResultsProvide relationships between the trend of paging success rates before and after deterioration and time of network adjustment and unexpected changes.

4.10.5 Closed Actions1. If paging success rate changes are caused by unexpected factors such as charging,

weather, and gatherings that cannot be controlled, provide analysis reports and explain this situation to customers.

2. Provide capacity expansion and hardware adjustment solutions if the paging success rate changes due to predictable and controllable factors, such as gathering activities, festivals and activities.

3. Make a summary and provide cases.


Ltd.

26


4.11 Action 11: Checking Exceptions on the CN Side

In a short period of time, paging success rates may deteriorate suddenly or fluctuate dramatically. After the RAN-side impacts are eliminated, this problem may be caused by exceptions on the CN side, including exceptions in operations and terminal servers (for example, faults of iTunes and BlackBerry servers may increase the number of paging users), and IP or port scanning in networks in the case of allocating IP addresses to UEs by the CN.

4.11.1 DescriptionsScenario: Paging success rate deteriorates suddenly in a short period of time.

Criteria: Eliminate network adjustment impacts in action 2, and device faults, low RRC success rates, and unexpected factors in actions 3, 7, and 10 respectively.



1 Traffic statistics Nastar

2 PCHR NPMaster

3 Internet information None

4.11.3 Steps for Analyzing Data1. Analyze the traffic statistics to confirm the time point when the paging success rate

deteriorates.

2. Obtain and analyze network operations on the CN side to check whether the CN adjustment and sudden deterioration of paging success rate occur at the same time point.

3. Analyze the PCHR data before and after deterioration to check whether the sudden increase paging failures are caused by a certain terminal. If such a UE exists, obtain Internet information to check whether the terminal server is abnormal.

4. Analyze user-plane data to confirm whether IP addresses of UEs are those of public networks and whether IP or port scanning occurs on public networks.

4.11.4 Results1. Provide the relationship between the paging success rate change trend before and after

deterioration and the time points of network operations on the CN side.

2. Provide distribution results of UEs contributing to sudden increased paging failures If a top UE exists, obtain terminal server information.

4.11.5 Closed Actions1. For paging counters deterioration due to network operations on the CN side, provide an

impact analysis report and roll back network operations.


Ltd.

27


2. If the paging success rate changes are caused by unexpected factors (terminal server exceptions) that cannot be controlled, provide analysis reports and explain this situation to customers.

3. Make a summary and provide cases.

4.12 Action 11: Analyzing UE Exceptions and Trace

4.12.1 DescriptionsScenario: Paging success rates do not meet related requirements and deteriorate in scenarios of networking swapping or network sharing policy adjustment.

Criteria: Top users or top UEs cause unqualified paging success rates or paging success rate deterioration.



1 Tracing data on the Iu interface FMA

2 UE logs UE log processing tools, such as Probe and Actixe

4.12.3 Steps for Analyzing Data1. Analyze tracing data at the Iu interface to identify the IMSIs in paging failures.

2. Check IMEIs of these IMSIs on the CN side to confirm whether most IMSIs belong to the same type of UEs.

3. Collect UE logs to analyze causes of paging failures.

Collection guidance for UE logs of different UEs is available at

http://3ms.huawei.com/hi/group/2299/wiki_2486839.html

a The following describes the methods for obtaining mapping between IMSIs and MSISDNs: Check the mapping in batch by running the LST SUBDATA command on the USCDB of the HLR OMU and export the results in a TXT file.

b Enter a single IMSI to check the mapping by running the DSP USERINFO command on the OM platform of the MSC and export the results in a TXT file.

c Check the mapping based on IMSIs by running the MM CONTEXT: DSP MMBYIMSI command on the SGSN LMT.

d Analyze CHRs of the CN based on the CHR database.

Turn on the IMSI reporting switch on the SGSN and collect CHR data. Convert the CHR data and import them into tools. Check the mapping based on the SQL data.

NOTE

This method is used to check the mapping in a single CHR record on the CN but cannot be associated with IMSIs on the RAN side. Therefore, this method is not recommended.


Ltd.

28

http://3ms.huawei.com/hi/group/2299/wiki_2486839.html


e Export the data of online users from the GU HLR, and save the data in a board of the DTL. The data file will be automatically uploaded to an FTP or SFPT server. The exported file is compressed into a .tar.gz packet, saved in /opt/uscdb/MemUserExp, and uploaded to an FTP or SFTP server.

Exported table contains the mapping between IMSIs and MSISDN.

W_COMMON_SER_DATA (table ID is 12), data is shown in the following table:

SID IMSI MSISDN

1 460009900088001 8613590208001

2 460009900088002 8613590208002

3 460009900088003 8613590208003

For details, see USCDB V100R002C06 Guide to Exporting Subscriber Data Online.

http://support.huawei.com/support/pages/kbcenter/view/product.do?actionFlag=detailProductSimple&web_doc_id=SC0000692445&doc_type=VersionDoc&doc_type=VersionDoc

NOTE

This operation comes with high risks. Therefore, confirm whether to perform this operation based on specific sites and number of users. For a small number of users, export data through the PGW.

This operation is performed with coordination between R&D personnel and frontline personnel.

4.12.4 Results Provide records of top user paging analysis.

4.12.5 Closed ActionsPaging failures of top UEs are usually caused by incompatibility with special networking. In this case, upgrade software through the IOT. If this upgrade experiences difficulties, adjust the networking mode to avoid paging failures.


Ltd.

29




Guide to Troubleshooting Paging Failures 4 Cases of Required Actions for Paging Problems

5 Cases of Required Actions for

Paging Problems

5.1 Cases of Parameter CheckingProblem

On August 18 in office I of country M, paging flow control alarms are frequently generated on the RNC, leading to flow control on paging messages and decrease in paging success rates.

Analysis process

After performing action 1, it is confirmed that paging success rates on the CN side decrease but those on the RAN side do not change. In this case, the problem can be roughly identified as discarded paging messages over the Iu interface.

After checking the SPU loads, it is found that loads in some subsystems are higher than those in other subsystems, as shown in the following figure:

Perform action 2 to check whether parameter configurations are proper. The results are shown in the following figure;


Ltd.

30


Closed actions and result evaluation

Enable the SPU load sharing function by running the following command. After the function is enabled, flow control on the paging messages is eliminated and paging success rates on the CN side are normal again.

SET URRCTRLSWITCH: PROCESSSWITCH=RNC_SHARE_SWITCH-1;

5.2 Cases of Paging Success Rates Caused by UE Exceptions

Problem

In office T of country T, paging success rates on both the CN and RNC side decrease dramatically from 94% to 30% after the MOCN feature is enabled.

Analysis process

According to action 1, the problem is identified to be a paging problem of type 2. No exceptions are found by performing required actions in sequence. Collect tracing logs over the Iu interface and IMISIs in paging failures by performing action 11. By checking the CN, it is found that 90% UEs are of the same type.

Frontline personnel confirm that this type of UEs is customized to support the MOCN feature. Customers require the terminal company to identify the root causes. By collecting UE logs, the terminal company finds that UEs receive paging messages but do not respond to these messages. This problem can be solved by upgrading UE software. However, it is difficult to upgrade all UEs because many UEs have been delivered to the market. This solution is not feasible.

Check the network to find preventive measures. By analyzing the network, the root causes are as follows:

Only one carrier is configured on the RNC. Frontline personnel reconstruct the cell under the RNC to an MOCN cell, and configure the ID of the primary operator to 99 and that of the secondary operator to 00. In areas covered by UMTS networks, operator 99's GSM networks also cover the areas. In these areas, operator 00 has no GSM networks but their GSM customers are allowed to roam to operator 99's GMS networks. In the preceding scenario, when the SIM card of a UE supporting MOCN is provided by operator 00, the UE does not respond to received paging messages.


This problem is solved by using the following solution:

Add a carrier, F2, on the RNC. F2 is exclusively used by operator 00. After the solution is performed, paging success rates are normal again.

5.3 Cases of Paging Success Rate Deterioration Caused by Incorrect Configurations on the CN Side

Problem


Ltd.

31


In country I, paging success rates on one of the two RNCs are low, only 50%. By checking paging counters on the CN side, it is found that this paging success rate in LAC areas is about 90%.

Analysis process

According to action 1, the problem is identified to be a paging problem of type 3. This problem may be caused by incorrect configurations on the CN side. Check the paging-related configurations on the MSC. The result shows that the number of configured LACs for the RNC is greater than the number of actual LACs. Therefore, paging objects are not within the LACs under this RNC and local paging messages fail to be received, which results in low paging success rates.


Solve this problem by using the following solution:

Delete LACs not used by the RNC on the MSC. After the invalid LACs are deleted, paging success rates on the RNC side are gradually normal and basically equal to those on the CN side.


Ltd.

32

guide to troubleshooting paging failures-20130110-v1.0

Documents

paging delay

paging procedu

paging failure problems

classifying paging failures

paging failures issue1

definitions of paging

huawei trademarks

kpis of paging success