gprs drive testing___quick_guide

Open

INSTRUCTION

1 (1) Prepared (also subject responsible if other) No.

EAB/ZG/NI Robert Blom EAB/G-04:000245 Uen Approved Checked Date Rev Reference

EAB/ZG/NI [Johannes Lindvall] 2004-02-10 A

Quick guide to GPRS drivetesting with TEMS Investigation

Abstract

This document is intended to be a quick guide on how to efficiently test and monitor performance of a GPRS network with TEMS Investigation.

Contents 1 Introduction ..........................................................................................2 2 Test Applications .................................................................................3

2.1 Attach / Detach .........................................................................3 2.2 Activate / De-activate PDP context ...........................................3 2.3 Ping ..........................................................................................3 2.4 HTTP Load ...............................................................................4 2.5 FTP Get/Put..............................................................................4

3 Measurements ......................................................................................5 3.1 Pre-requisites ...........................................................................5 3.2 GPRS Functionality Tests.........................................................5 3.2.1 Setup examples........................................................................6 3.2.2 Analysis examples....................................................................7 3.3 GPRS Radio Network Test ..................................................... 11 3.3.1 Setup example........................................................................ 11 3.3.2 Analysis examples.................................................................. 11 3.4 General Tips & Tricks ............................................................. 20

4 Overall Procedures ............................................................................ 21 5 References.......................................................................................... 21

Open

INSTRUCTION




1 Introduction

This document describes a procedure for measuring GPRS performance using TEMS Investigation 4, where two different aspects are covered.

1. How to do continuous testing of basic GPRS core functionality, such as Attach & PDP Context activation.

2. How to measure radio network quality as experienced by a GPRS user.

The main goal of the document is to describe a procedure for the second part: I.e. to present a methodology for finding problems related to the radio network, giving a few examples on possible solutions and also to give examples on how to identify problems that are not related to the radio network.

In chapter two the different applications that can be tested from in TEMS Investigation is outlined and the main usage of the test applications are described.

In chapter three examples on how to set-up and analyze measurements are given and chapter four is a short summary of the overall measurement procedures.

Open

INSTRUCTION




2 Test Applications

2.1 Attach / Detach

Attach procedure is when the MS is activating mobility management for GPRS, i.e. going from GMM Idle state. When this is done the SGSN node will continuously handle location information for the mobile, knowing the routing area that the mobile belongs and in some cases also the specific cell (when the MS is in Ready state).

The main purpose of GPRS Attach tests is to test the functionality of the GPRS core network. The procedure is so short that radio environment has limited impact on performance.

These tests are best performed with automated GPRS Attach and GPRS Detach commands in the Command sequence handler.

2.2 Activate / De-activate PDP context

PDP Context activation / de-activation is session management procedures used when the MS communicates with the GGSN node and receives a user IP address (at activation). It is only after the PDP context activation is performed that any real user data can be sent. Prior to this only GPRS Mobility Management messages and SMS over GPRS will be sent (when MS is attached).

The main purpose of Activate / De-activate PDP context is to test the functionality of the GPRS core network. The procedure is so short that radio environment has limited impact on performance.

These tests are best performed with automated dial up and hang up commands (which is the “computer” commands generating PDP context activation) in the Command sequence handler.

2.3 Ping

Ping is an Internet control message that is sent to a server and a response back is expected.

The main purpose of the ping is to check accessibility to a specific server and the roundtrip time to that server (which is a quality indicator of the network). The procedure is so short that radio environment has limited impact on performance. The time to transfer the ping message over the air interface is normally the main contributor, which makes the result BSS implementation dependant (thus different between vendors).

These tests are best performed with automated ping commands in the Command sequence handler.

Open

INSTRUCTION




2.4 HTTP Load

HTTP load corresponds to a user browsing the web and downloading a web page. This includes signaling through the GPRS core network and IP network between MS and HTTP server.

It is important to remember that the actual design of the web page will impact the measured performance. Number of objects in a webpage, what servers these objects resides on and used HTTP version are some factors contributing to how efficient the download will be.

HTTP load is thus useful in order to investigate performance of a specific web page or to do repetitive tests of downloading a standardized/default webpage. Most web pages will however not provide sufficiently stable throughput to be useful for radio network evaluation (note that a web page could also be designed to give a stable throughput suitable similar to FTP).

These tests are best performed with automated HTTP load commands in the Command sequence handler.

2.5 FTP Get/Put

FTP Get and Put are commands for downloading and uploading data to FTP servers. This includes signaling through the GPRS core network and IP network between MS and FTP server.

FTP Get and Put is normally used to initiate data transfer in downlink and uplink direction. This data transfer uses TCP and will try to maximize the throughput from the beginning to the end of the file transfer. It is thus a good choice when evaluating the radio network performance. There will be a small set-up time in the beginning of FTP Get and Put applications due to: connection to server, providing username and password, setting directory etc.

These tests are best performed with automated FTP Get and Put commands in the Command sequence handler.

Open

INSTRUCTION




3 Measurements

3.1 Pre-requisites

There are a number of pre-requisites for successfully completing a measurement and a small checklist of these is provided below. For a more complete description of connection and measurement set-up procedures, please refer to the TEMS Investigation manual.

! SIM card with a GPRS subscription

! TEMS Mobile and a GPS if geo-positioned data is wanted

! Data account that is correctly configured in the MS This includes e.g. which APN that the MS will connect to. Different data accounts will have different CID numbers.

! The mobile should be defined as a modem in windows Verify that an updated modem driver exists for the used TEMS model. Use cable connection from PC to mobile.

! Create a dial-up connection in windows Different data accounts can be selected with different phone numbers when doing dial up procedure, i.e. *99***1# refers to CID 1 (account 1), whereas X*99***2# refers to CID 2.

! Test the applications in advance It normally saves time if the applications that are to be used is tested from a LAN connection in advance, e.g. verifying FTP servers, user IDs, passwords, ping response time etc. This could otherwise be a time consuming procedure to do when drivetesting.

3.2 GPRS Functionality Tests

These tests are performed to test the functionality of a GPRS network, i.e. the actual radio environment is not really interesting.

Tests falling under this category is GPRS Attach / Detach, PDP context activation / de-activation tests for the GPRS core network and Ping for the general IP network perceived.

Typical measures are success rate, cause values of rejects and time taken to complete a procedure.

Open

INSTRUCTION




3.2.1 Setup examples

An example of a command sequence for performing multiple Attach and Activate PDP Context tests is shown in Figure 3-1

Figure 3-1 Command sequence example of Attach & PDP Activation tests

An example of a command sequence for performing repetitive Ping tests is shown in Figure 3-2.

Figure 3-2 Command sequence example of ping test

The Ping command has a number of arguments

- Size of the ping packet

- Number of consecutive pings

- Timeout time (indicates when ping is seen as unsuccessful)

- Time between consecutive pings

The setting of these parameters will affect the measured performance. E.g. a large packet will take long time to transfer over the air interface. Short time between consecutive pings will increase the probability that the MS will be in Packet Transfer mode when next ping is to be sent (which will improve performance) etc.

Open

INSTRUCTION




3.2.2 Analysis examples

The first step of the functionality analysis is normally to look at the success rate of the procedures.

For Attach and Activate PDP context procedures this can be quickly calculated by creating a report over relevant events, i.e. including GPRS Attach / Attach Failures and GPRS PDP Context Activation / Activation Failure. By comparing number of successes with failures the success rate can be calculated.

This success rate should be very high, as a correctly configured MS (with correct data for that MS in the SGSN) almost never fails. Figure 3-3 shows a TEMS report example.

Figure 3-3 Report example counting different events

The Attach failure rate should be calculated as the number of Attach Failures compared to the sum of the Attach and the Attach Failures (same procedure applies for the Activate PDP Context) – in the above example this is 0 %, corresponding to a 100% success rate.

Comparing the actual number of Attach to the number of GPRS PDP Context activation messages is difficult, as there might be session interruptions before the next step in the command sequence.

When the success rate (or failure rate) has been calculated, the time to complete the procedure can be calculated to further analyze the performance.

In order to analyze the time distribution of these procedures a text export can be done, where the information elements Attach time and PDP Context time are exported.

A selection of the messages, Attach Complete and Activate PDP Context Accept in the resulting text files, will give one time sample per event. The selection can be done in any suitable program such as: Access, any text viewer/editor capable of handling large / multiple text files, Perl etc.

Open

INSTRUCTION




Activate PDP Context Time Distribution

0

50

100

150

200

250

50 200

350

500

650

800

950

1100

1250

1400

Time [ms]

No

of s

ampl

es

Figure 3-4 Activate PDP Context Time distribution graph

An example of a time distribution graph of Activate PDP Context is shown in Figure 3-4. Most samples of the response time are in the 400 – 500 ms range. Only a few are longer then 1 second and the longest time is around 1.4 seconds. The above example shows good performance for Activate PDP context as the success rate is high and the response time is low.

GPRS Attach Time (Request - Complete)

0

50

100

150

200

250

300

1.7 1.8 1.9 2 2.1 2.2 2.3 2.4 2.7 2.9 3 3.2 3.3 3.4 4.2 8.3

Time [0.1 s bins]

No o

f sam

ples

Figure 3-5 GPRS Attach Time distribution graph

An example of a time distribution graph of GPRS Attach is shown in Figure 3-5. The time to complete the GPRS Attach procedure are fairly long, most samples are in the 1.9 to 2.1 second range. The reason for the relatively long attach time can be found if signaling during the Attach procedure is analyzed in the Layer 3 messages window in TEMS Investigation.

Open

INSTRUCTION




The Attach procedures shown in Figure 3-5 are started with the Attach Request being sent from the MS to the SGSN. The SGSN then starts authentication procedure and sends Authentication and Ciphering request message and MS answers with Authentication and Ciphering response. The Attach procedure is then completed by receiving Attach Accept from the network followed by Attach Complete from the mobile.

The time between the different messages are a few hundred milliseconds, but the total time ends up being around two seconds. A few samples of even longer Attach time exist and these are due to a delay in the reception of the Authentication Request message.

Note that the network used in the example above could be configured to use authentication less often, which would reduce the average Attach time.

Ping success rate (or failure rate, sometimes denoted the packet loss rate) can be calculated with a TEMS Investigation report by adding the events Ping Response, Ping Success and Ping Timeout, as can be seen in Figure 3-6

Figure 3-6 Ping response report

If a Ping delay threshold is also selected in the report, then a cumulative distribution graph will be generated for the time distribution, se examples in Figure 3-7 - Figure 3-9.

In these figures it is evident that different set-up of ping measurements will give different results. I.e. it is important to use similar settings for any type of comparisons of ping measurements.

With a 32 byte ping message and 100 ms delay between sequential pings the normal range of ping response is 600 – 700 ms, as can be seen in Figure 3-7. The result of increasing the time between sequential pings to 5 seconds is that ping response in many cases takes more than 1 second, see Figure 3-8. One contribution to this is that the closure of TBFs is normally delayed somewhat to have an associated signaling channel open for a short time span after data transfer (which improves set-up time of next TBF).

Larger packets also gives a longer ping response time, as can be seen in Figure 3-9.

Open

INSTRUCTION




Figure 3-7 Ping delay distribution, size 32 byte 100 ms between ping

Figure 3-8 Ping delay distribution, size 32 byte 5 s between ping

Figure 3-9 Ping delay distribution, size 1460 byte 100 ms between ping

Open

INSTRUCTION




3.3 GPRS Radio Network Test

These tests are performed in order to test the quality of the radio network and performance of radio network functionality such as cell reselection. Actual application used is not primarily of interest but should provide as stable throughput as possible. This simplifies the analysis of the radio network.

The normal test for this is with FTP download (or upload), as this will provide a stable throughput. Note that HTTP also could be used to initiate long and stable downloads similar to FTP if the HTTP object is suitable.

Typical measures are RLC throughput, cell reselection radio link outage time etc.

3.3.1 Setup example

An example of a command sequence for performing a FTP Get test is shown in Figure 3-10. In this example a very large file is downloaded where the time to download is longer than the planned testing time. Repetition of the procedure can be used when smaller files are downloaded.

Figure 3-10 Command sequence example of FTP Download

3.3.2 Analysis examples

Three different kinds of performance indicators are important when the GPRS radio network is analyzed with TEMS Investigation.

1. General performance indicators These performance indicators should ideally be affected by as many possible problems as possible in order to quickly identify performance degradation.

2. Application characteristics performance indicators These performance indicators should give guidance on how application behavior is affecting the measured radio network performance, i.e. trying to identify areas where measured radio network performance is limited by the application layer.

Open

INSTRUCTION




3. Specific performance indicators These performance indicators should be as specific as possible in order to quickly pinpoint found problems.

3.3.2.1 General Performance Indicators

The most widely used general problem indicator is the RLC throughput. It will be affected by both capacity and radio link quality problems.

It is important to know what RLC throughput should be considered good and what should be considered bad. The used FTP server should be able to provide a stable and high data stream to the mobile under stable radio conditions, thus it is important to in advance test this.

Figure 3-11 Stable FTP Download

In the above graph the used FTP server provides a stable and high RLC throughput (>50 kbit/s) over four timeslots. If no quality and capacity problems exist in a network the RLC throughput should be around this figure for a four slot mobile, see reference [1].

In the above figure it is evident that the RLC throughput is the performance indicator with the smoothest variation in time. This is a result of that throughput measurements always are time averages and the calculations will only be based on “completed” blocks / frames / packets on the various protocol layers. RLC throughput will thus have a “smoother” variation due to the fact that RLC data blocks are considerably smaller then LLC frames or IP packets.

Other general problem indicators are: Areas with no GPRS coverage and abnormal releases of TBFs.

Open

INSTRUCTION




3.3.2.2 Application characteristics Performance Indicator

The application characteristics performance indicator available in TEMS is the Application Throughput. It will “follow” RLC throughput in the sense that when capacity or radio link performance limits what RLC throughput can be achieved on, then the application throughput will also be degraded.

In some situations the application layer will limit the measured throughput on lower layers (RLC/LLC etc). This is due to that the application layer will not feed data fast enough to the BSS to keep the radio link saturated. A typical example of this is when one IP packet is lost somewhere between the FTP server and the GPRS MS as in Figure 3-12.

Figure 3-12 Lost IP packet causing lower RLC throughput

In the above figure an IP packet is lost on the downlink and the TCP layer in the computer will ask for retransmission of that packet.

The application throughput in TEMS is measured on top of the TCP layer (WinSock), i.e. data is only included in calculations when it is in the correct sequence. On measured application layer throughput, a lost packet leads to that application throughput goes down to zero until the missing packet is received. Then a lot of data is transmitted through WinSock, which gives a “spike” in the application throughput.

Due to the missing packet TCP will send packets at a lower rate and try to build up transmission speed. This will be seen as a low but slowly increasing application and RLC throughput.

This example shows low RLC throughput that is not caused by any problems in the radio network but on application layer behavior. More information on application behavior can be found in reference [2].

Open

INSTRUCTION




One way to become more familiar with the application layer characteristics is to use a packet sniffer tool such as Ethereal or WinDump and collect all IP data received. It will then be easy, although time consuming, to identify lost packets, retransmission etc. on the application layer.

3.3.2.3 Specific Performance Indicators

The potential problems limiting the performance for a GPRS mobile can be grouped into a number of problem areas from the mobile point of view.

1. Low cell capacity available

2. Sharing of resources

3. PCU capacity limitations

4. Bad radio link quality

5. Cell re-selection problems

In this chapter these problem areas are explained briefly and useful performance indicators for respective area are listed.

Low cell capacity available

This is when the available cell resources are not sufficient to give maximum bandwidth possible to a specific user. It is normally caused by CS and PS users competing over the same resources. The impact on the experienced performance depends on network configuration, prioritization between CS and PS and how high the competition is.

Medium competition for resources

CS and PS users will compete for shared resources, i.e. on-demand PDCHs. The capacity available for PS is limited but competition is not so strong that there are long outages of GPRS service. This is normally seen in cells with high CS traffic. The impact on the measured performance depends on how scarce these resources are, but indicators are:

- Low RLC Throughput

- Low number of used timeslots

- Packet PDCH releases and Packet TBF releases

Open

INSTRUCTION




Figure 3-13 Low cell capacity example

Performance indicators for low cell capacity are shown in the above figure. MS is only assigned two TS when entering a cell due to high amount of CS traffic. Additional CS traffic causes the release of used resources, seen here at 1 as a Packet PDCH Release message. The result of the low cell capacity is seen in all performance indicators: Low RLC throughput, low number of used DL timeslots and Packet PDCH Release messages.

High competition for resources

If there are no dedicated PDCHs and no idle TCHs that can be used as on-demand PDCHs, it will not be possible to assign resources to a GPRS mobile when it requests resources in the Channel request message. This will lead to GPRS service outage in that cell.

In addition to performance indicators listed for medium competition example above the following might be seen.

- Channel Request without answer or PS Immediate Assignment Reject message as a response to channel requests.

Sharing of resources

If low amount of cell resources exist, sharing between users occur and lower throughput will be the result. This is not the only reason for sharing as the amount of sharing can be one part of the GPRS network design in the same way as some congestion is planned for in GSM (GoS).

1

Open

INSTRUCTION




Thus it is impossible to say if it is too much sharing or not without knowing what quality the network is planned for. In TEMS it is very easy to identify sharing between users with the information element:


- Other data on used TS (RLC data to other users on a shared timeslot).

Figure 3-14 Sharing of resources example

Performance indicator for other data sent on timeslots used by the GPRS mobile is shown in the above figure. Percentage of the TS that is utilized by other users is shown in the information element Other data (%) as indicated by 1.

PCU capacity limitations

The reason could also be that there is a limitation in the number of available GSL devices in the PCU so that no PDCH can be created despite that free timeslots exist in the cell. This will give low number of used timeslots and low RLC throughput, although no release of PDCH or TBF should be seen.


- Low number of assigned timeslots

- NO Packet PDCH Releases or Packet TBF Releases seen.

If low number of timeslots is assigned for a number of cells throughout a drivetest then PCU capacity problems can be suspected and should be investigated.

Bad radio link quality

The impact of bad radio link quality depends on the severity of the radio link quality problems.

For minor to medium quality problems the result will be that a lower coding scheme will be used. If it is not possible to change to a lower CS, either due to the usage of fixed CS or that the MS is already using CS-1, then the Block Error Rate will increase. As well as for CS the C/I is excellent to look at to evaluate radio link quality, i.e. good performance indicators are:

1

Open

INSTRUCTION




- Low measured C/I

- CS usage for data High proportion of CS-1 indicates bad radio environment high proportion of high CS indicates good radio environment.

- High Block Error rate

- Signal strength (RxLev) Used to evaluate if quality problems are caused by low signal strength or high interference.

Figure 3-15 Bad radio link quality example

The impact of the bad radio link quality on the performance indicators is shown in the above figure. RLC throughput is low, C/I is low, BER and BLER are high. Only CS-2 is used for data as link adaptation is not used, if link adaptation was used BLER would be lower, but CS-1 Usage would be higher.

Severe radio link quality problems might lead to problems to: decode system information in the serving/neighboring cells, released TBFs etc. The following could be indicators of severe radio link problems:

- Slow update of SS, C1 & C2 parameters for neighboring cell.

- Failed cell reselection attempts to a particular cell

- Released TBF that was pre-ceded by bad radio quality

Open

INSTRUCTION




Cell re-selections

Mobility is one of the key aspects of GSM and GPRS, which makes cell re-selections an integral part of any GPRS network. The impact on performance due to cell reselections depends on a many factors.

Different applications will have different sensitivity to mobility problems, i.e. a user downloading small wap pages have very short transfers (in the range of seconds) and the probability of a cell reselection occurring during this time will be low. A user that is stationary will also seldom experience cell reselections, irrespective of the used applications.

The users that will experience mobility problems are users doing long data transfers, while moving around in the network. Drivetesting a GPRS network with TEMS can normally be seen as a worst case for mobility problems.

Cell reselections will in general be seen as a temporary outage of the link between the MS and the application server. In many cases the used applications will not suffer too much from these short interruptions in the data transfer. Protocols such as TCP will ensure that lost packets will be re-sent.

However when cell re-selections becomes to long or there are to many within a short timeframe then the user performance can be degraded more than wished for. Measures to look out for is then:

- Cell re-selection outage time

- Cell re-selection frequency

The cell re-selection outage time can be measured on different layers, e.g. on radio link level or on application layer level. The following messages are important when analyzing cell re-selection outage time on the radio link.

Packet Downlink Ack/Nack in the old cell Normally the last message sent in the old cell

System Information Type 13 in the new cell Contains system information specific for GPRS that must be read from the new cell before requesting a packet data channel. This is normally one of the main contributors to the radio link outage time as the time between sending different SI 13 messages on the BCCH can be in the range of seconds.

Channel Request in the new cell A Channel Request is normally sent quickly after SI 13 has been read in the new cell.

Immediate Assignment in the new cell An Immediate Assignment to a PDCH is normally received quickly after a channel request. This can be seen as the re-establishment of the radio link, as there now can be signaling between BSS and MS.

Open

INSTRUCTION




Packet Uplink Ack/Nack in the new cell The Packet Uplink Ack/Nack is normally received quickly after the Immediate assignment indicating that data transfer has occurred on the uplink, i.e. a verification that radio link is established and working. Thus the time from the last Packet Downlink Ack/Nack in the old cell to the first Packet Uplink Ack/Nack in the new cell can be seen as a good measure of the radio link cell reselection outage time.

Packet Downlink Assignment in the new cell When the used test application is a downlink transfer, it can not really be considered as re-established until a TBF in the downlink exists. The Packet Downlink Assignment indicates that a new downlink TBF has been established.

Packet Downlink Ack/Nack in the new cell Packet Downlink Ack/Nack message normally indicate that data has been transferred over the radio link, i.e. the time from the last Packet Downlink Ack/Nack message in the old cell to the first in the new cell could be seen as a closer measure to the user experience outage time. This measure will however be difficult to analyze, as it is the BSS that controls when TBF should be initiated on the downlink. This could be done before actual user data arrives to improve the BSS performance.

Figure 3-16 Cell re-selection example Signaling at cell re-selection is shown in the figure above. The time from the last Packet Downlink Ack/Nack in the old cell to the Immediate Assignment in the new cells is 2.2 second. Normal values for intra routing area update cell re-selections are from 2 to 4 seconds.

Open

INSTRUCTION




The different contributions to the cell re-selection outage time can be analyzed by exporting data and calculating the time difference between messages in the cell re-selection procedure.

Cell Reselection Time

02468

101214161820

1 3 5 7 9 11 13 15

Tim

e [s

]

Packet DLAssignment

ImmediateAssignment

ChannelRequest

SI_13

Figure 3-17 Cell re-selection outage time distribution example In the figure above it can be seen that the a number of cell reselections have very long outage time (> 4 seconds), the reason for this should be investigated. It can also be seen that time to read System Information 13 message is the main contributor to the total radio link outage time.

3.4 General Tips & Tricks

• Try to introduce waiting time between commands in the command sequence if unexplainable failures occur (remember to wait on the same MS as commands are sent to!).

• Set the TEMS mobile to operate in GSM only mode. This means that it will not attach at power-on but when needed – with this setting log files will always start with attach & PDP context activation (and network will have one less user doing GMM signaling when not needed).

Open

INSTRUCTION




4 Overall Procedures

The reason for doing GPRS performance measurements can vary and two important reasons are: GPRS functionality tests and GPRS radio network tests.

GPRS functionality tests should primarily be done to verify normal behavior and to do functionality comparisons (over time, parameter changes, vendors etc.)

- A correctly defined MS together with correct configuration of core nodes should have very high Attach & Activate PDP Context success rate

- Actual time measured for the procedures will depend on network configuration and general vendor performance

These tests should preferably be done under controlled radio conditions without limitations in the radio network.

The tests should be performed every now and then to verify “normal” performance levels, procedure completion time, do comparisons at system/functionality upgrades etc.

GPRS radio network tests should be performed continuously.

It is important to have established what is “normal” performance to use as a comparative measure. The test set-up of the radio network performance measurements should thus be controlled now and then with FTP transfers under ideal radio network conditions. This ensures that the application layer is not varying too much, which will make the analysis of the radio network performance much easier.

5 References

GPRS Measurements in the TEMS Products [1]

Application-level Data Service Measurements in [2] TEMS Investigation GSM

gprs drive testing___quick_guide

Documents

gprs network

gprs radio network test

gprs performance

gprs drivetesting

tems investigation

gprs user

setup examples

analysis examples