definition kpis edge doc
TRANSCRIPT
Definition of GPRS/EDGE End-to-End
KPIs
Abstract
The document provides the definition of GPRS/EDGE end-to-end Key Performance
Indicators (KPIs), which make a description of the end user’s perception of the service
performance possible.
• Definition of GPRS/EDGE End-to-End KPIs • • Introduction •
Table of Contents
1 I�TRODUCTIO� 4
1.1 SCOPE 4 1.2 STRUCTURE 4 1.3 REVISION HISTORY 5
2 REMARKS O� THE �ATURE OF THE KPIS 6
3 TERMI�OLOGY USED FOR CATEGORIZI�G THE KPIS 6
3.1 ACCESSIBILITY 6 3.2 RETAINABILITY 7 3.3 INTEGRITY 7 3.4 DISTINCTION BETWEEN KPIS AND PERFORMANCE INDICATORS (PIS) 7
4 SERVICES 9
5 DEFI�ITIO�S OF KPIS 11
5.1 CATEGORIZATION OF THE KPIS 11 5.2 ACCESSIBILITY KPIS 12
5.2.1 GPRS Unavailability 12 5.2.2 GPRS Attach Failure Ratio 13 5.2.3 GPRS PDP Context Activation Failure Ratio 15 5.2.4 Secondary PDP Context Activation Failure Ratio 17 5.2.5 GPRS Service Access Failure Ratio (for each service) 19 5.2.6 GPRS Accessibility Failure Ratio (for each service) 24
5.3 RETAINABILITY KPIS 26 5.3.1 GPRS PDP Context Cut-off Ratio 26 5.3.2 GPRS Data Transfer Cut-off Ratio 27 5.3.3 Cell Update Failure Ratio 29 5.3.4 Intra SGS, RAU Failure Ratio 31 5.3.5 Inter SGS, RAU Failure Ratio 32 5.3.6 PS Paging Failure Ratio 34
5.4 INTEGRITY KPIS 35 5.4.1 GPRS Attach Setup Time 35 5.4.2 GPRS PDP Context Activation Delay (per AP,) 36 5.4.3 GPRS Service Access Time (for each service) 37 5.4.4 GPRS Access Time (for each service) 41 5.4.5 GPRS Mean User Data Rate 42 5.4.6 Round Trip Time 44 5.4.7 Client-Server Interaction Time with Background Traffic 46 5.4.8 Packet Loss Ratio 47 5.4.9 Cell Update Time 49 5.4.10 Intra SGS, RAU Time 50 5.4.11 Inter SGS, RAU Time 52
6 OVERVIEW OF THE KPIS 53
7 SUMMARY 54
8 TERMI�OLOGY 56
8.1 ABBREVIATIONS 56 8.2 GLOSSARY 57
9 REFERE�CES 58
10 APPE�DIX 59
• Introduction • • Definition of GPRS/EDGE End-to-End KPIs •
10.1 STATISTICAL GUIDELINES 59
• Definition of GPRS/EDGE End-to-End KPIs • • Introduction •
1 Introduction
1.1 Scope
One aspect of quality when it comes to GPRS/EDGE is the performance perceived by
the end-user. Key Performance Indicators are those indicators that have big impact on
the end-user’s perception of the service performance. Therefore the KPI definitions
make a description of the end user’s perception of the performance possible.
Additionally they enable operators to compare GPRS/EDGE performance between their
own networks and with the performance of competitors’ networks.
Purpose of this document is to define GPRS/EDGE end-to-end Key Performance
Indicators (KPIs). In order to ensure comparable results when measuring KPIs the
definition of each KPI has to outline what should trigger the start of the measurement
and the stop of the measurement. This start and stop trigger can be certain signaling
messages or a specific data packet. Therefore the definitions include these trigger points
that are to be used when the KPIs are measured in a live network or in a lab
environment.
The document does not:
• specify test cases that should be run in order to measure the end-to-end performance,
• give benchmark figures for the different KPIs,
• describe parameter settings for an optimized GPRS/EDGE network.
The above-mentioned items are covered in the Generic Test Description for
GPRS/EDGE End-to-End Performance Testing (see [1]), and in the Ericsson GPRS
End-to-End Performance Guidebook (see [3]).
1.2 Structure
The document has the following structure:
• Nature of the KPIs
• Terminology used for categorizing the KPIs
• Definitions of the KPIs
• Appendix: Statistical Guidelines
• Introduction • • Definition of GPRS/EDGE End-to-End KPIs •
1.3 Revision History
• Definition of GPRS/EDGE End-to-End KPIs • • Remarks on the Nature of the KPIs •
2 Remarks on the Nature of the KPIs Main focus of the KPI definitions is that the KPIs shall reflect the end-user’s perception
of the service performance. The KPIs shall be measurable either with active or passive
tests.
Active Measurements
Measuring with active tests means that the person who performs the tests actively
generates traffic and only the performance for this generated traffic is measured. These
measurements are typically limited to a certain period of time, the conditions of the
system under test are known and detailed measurements and investigations are possible.
Test cases for performing active measurements are defined in the Generic Test
Description for GPRS/EDGE End-to-End Performance Testing [1].
Passive Measurements
Measuring with passive tests means that the performance of traffic generated by all
subscribers is continuously measured via counters in any node of the GPRS/EDGE
network (refer to Measurements of GRPS End-to-End KPIs with Counters [6])1 or with
specific tools for passive measurements. Detailed conditions are often unknown, e.g. cell
load, number of used TS etc.
3 Terminology Used for Categorizing the KPIs Terminology for grouping the KPI definitions follows the international ITU-T standards
(E.800), in order to facilitate alignment of KPI definitions between Ericsson and
vendors. Since standard ITU-T E.800 is valid for telephone network and ISDN only, the
categorization of the GPRS/EDGE e2e KPIs definitions borrows the terminology used in
the standard. This means that the KPI definitions are not standard-compliant, but that
concepts deployed in the standard are transferred onto the grouping of the KPIs.
According to ITU Recommendation E.800 [4] Quality of Service is: The collective
effect of service performances, which determine the degree of satisfaction of a user of
the service.
3.1 Accessibility
The ability of the user to obtain a service within specified tolerances and other given
conditions.
The accessibility of the service does include a chain of operations the user must be able
to execute:
• the user must be able to access GPRS/EDGE as such, meaning he must be able to do
a GPRS attach and a PDP context activation, and
• he must be able to access an IP service after he has an activated PDP context.
1 General remarks on measurement of KPIs with counters: The counters give information about the success ratios,
but not about the failure cause code.
• Terminology Used for Categorizing the KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
The accessibility KPIs are the most important set of KPIs regarding IP services, since
the user is first of all interested in the fact if he is able to access the GPRS/EDGE
network and the IP services or not. Accessibility KPIs are defined in chapter 5.2.
3.2 Retainability
Retainability indicates the ability of the user to keep a service – once it was accessed -
under given conditions for a requested period of time.
The retainability of a service or session does include that the user does not have to
perform any additional manual operations that would not be necessary under stable
network conditions, like for example manual re-activation of the PDP context.
Retainability KPIs are defined in chapter 5.3.
3.3 Integrity
Integrity indicates the degree to which a service is provided without major interferences,
once it was accessed.
Integrity KPIs show the performance of successful service attempts. Even if a service
was accessed successfully, the user’s perception of the performance may vary between
very good and unacceptably bad. Integrity KPIs are defined in chapter 5.4.
3.4 Distinction between KPIs and Performance Indicators (PIs)
Some of the performance indicators are rated to be the primary performance indicators,
because they reflect the end-user’s perception of the service performance in an
outstanding manner. Therefore they are categorized as Key Performance Indicators,
while the performance indicators that contribute to the measurement of a KPI are
categorized as Performance Indicator (PI).
This categorization into KPIs and PIs is deployed in cases in which such a distinction
can be clearly drawn, i.e.:
Accessibility
� KPI ACC 050 GPRS Accessibility Failure Ratio [%]
PI ACC 010 GPRS Unavailability [%]
PI ACC 020 GPRS Attach Failure Ratio [%]
PI ACC 030 GPRS PDP Context Activation Failure Ratio [%]
PI ACC 040 GPRS Service Access Failure Ratio [%]
Retainability
� KPI RET 020 Data Transfer Cut-off Ratio
PI RET 010 PDP Context Cut-off Ratio
• Definition of GPRS/EDGE End-to-End KPIs • • Terminology Used for Categorizing the KPIs •
Integrity
� KPI I�T 040 GPRS Access Time [s]
PI I�T 010 GPRS Attach Setup Time [ms]
PI I�T 020 GPRS PDP Context Activation Delay [ms]
PI I�T 030 GPRS Service Access Time [s]
• Services • • Definition of GPRS/EDGE End-to-End KPIs •
4 Services The following services should be taken into consideration when defining KPIs and
measuring them with specific test cases:
• GPRS access service
• WAP
• Web browsing/HTTP
• FTP
• MMS
• Streaming
When measuring KPIs related to the above mentioned services in a live network file and
e-mail sizes cannot be influenced, so that passive tests of KPIs will measure any file and
e-mail size that appears in a live network.
But when measuring the KPIs related to services with active tests the following file and
e-mail sizes should be chosen:
WAP: Fixed WAP pages to be downloaded have to be defined.
WAP browser to be used has to be defined.
E-Mail: Send e-mail: 120 Kbytes zip-file (e-Mail with attachment)
Receive e-mail: 240 Kbytes zip-file (e-Mail with attachment)
Send e-mail: 5 Kbytes plain text
Receive e-mail: 5 Kbytes plain text
No use of Java or Flash
Web/HTTP
:
Fixed web pages to be downloaded have to be defined, e.g.
www.ericsson.com
Internet browser to be used has to be defined.
Version of the internet browser has to be defined.
FTP: Uplink: 100 Kbytes
Downlink: 200 Kbytes
• Definition of GPRS/EDGE End-to-End KPIs • • Services •
Timeouts2 for Data Transfer per Service:
The timeout values summarized in Table 1 are only applicable for active measurements
of KPIs related to services.
The timeout values for e-mail download comprise only the download of the data content
(e-mail text, or e-mail text plus attachment) but not the access procedure preceding the
actual download of the data content, see chapter 5.2.4.
HTTP
File Size 40kB 120kB 40kB 240kB 120kB 5kB
5kB text +
120 kB
attachment
240kB 5kB
5kB text +
240 kB
attachment
frozen
page
Timeout 64s 192s 32s 192s 192s 28s 200s 192s 24s 196s
Data Transfer (content)
FTP UL FTP DL E-Mail UL E-Mail DL
Table 1: Timeout values for data transfer
Remark
Purely calculated timeout values for uploading 5kB and downloading 5kB would result
in 8s and 4s respectively. But the actual timeout values being applied while measuring
service-related KPIs with small files will be higher, because of the slow start behavior of
TCP or TCP timeouts which increase the delay to several seconds.
Therefore the timeouts are calculated as follows:
20s + <download size> / UL performance [5kbps]
20s + <download size> /DL performance [10kbps]
2 Timeouts are calculated assuming DL/UL via one timeslot (worst-case scenario). Example: FTP DL 240kB,
240 kB * 8 = 1920 Kbits, 1920 Kbits/10 kbps = 192 seconds
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5 Definitions of KPIs
5.1 Categorization of the KPIs
The KPI definitions are divided into three categories as shown below:
QoS KPIs
Accessibility Retainability Integrity
GPRS/UMTS unavailability
Attach failure ratio
PDP context activation failure ratio
Service access failure ratio
e2e accessibility failure ratio
PDP context cut-off ratio
Data transfer cut-off ratio
Cell Update Failure Ratio
Intra SGSN RAU Failure Ratio
Inter SGSN RAU Failure Ratio
SGSN Paging Failure Ratio
Attach setup time
PDP context activation delay
Service access time
e2e access time
Mean user data rate
Round Trip Time
Packet Loss Ratio
Cell Update Time
Intra SGSN RAU Time
Inter SGSN RAU Time
QoS KPIs
AccessibilityAccessibility Retainability Integrity
GPRS/UMTS unavailability
Attach failure ratio
PDP context activation failure ratio
Service access failure ratio
e2e accessibility failure ratio
PDP context cut-off ratio
Data transfer cut-off ratio
Cell Update Failure Ratio
Intra SGSN RAU Failure Ratio
Inter SGSN RAU Failure Ratio
SGSN Paging Failure Ratio
Attach setup time
PDP context activation delay
Service access time
e2e access time
Mean user data rate
Round Trip Time
Packet Loss Ratio
Cell Update Time
Intra SGSN RAU Time
Inter SGSN RAU Time
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.2 Accessibility KPIs
5.2.1 GPRS Unavailability
TAG: ACC 010
Definition:
The PI GPRS/EDGE Unavailability [%] denotes the probability that GPRS/EDGE is not
active in the cell used by a subscriber.
Trigger Point(s):
Check if the GPRS/EDGE-specific signaling (SI13) exists on cell selection.
Check if the signaling message can be read out and if the required signaling exists on
BCCH or on PBCCH.
%10013#
13#[%] ∗=
SIreadtoattemptstotal
SIreadtoattemptsulunsuccessflityUnavailabiGPRS
Remark:
If this PI is measured for example during drive tests information is provided about the
GPRS/EDGE coverage on network level and coverage problems in special.
Since this PI can only be measured in live networks the PI is measured for a random set
of cells in the live network, since the GPRS/EDGE availability is checked before any
test case can be executed.
Active Measurement:
PI not covered in the Generic Test Description for GPRS/EDGE End-to-End
Performance Testing.
Passive Measurement:
There is no means of passive measurement (counter or Moniq tool) for this PI.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.2.2 GPRS Attach Failure Ratio
TAG: ACC 020
Definition:
The PI GPRS Attach Failure Ratio [%] denotes the probability that a subscriber cannot
attach to the GPRS/EDGE network.
The timeout interval for the attach procedure is 30 seconds, meaning: if the attach
procedure was not completed after 30 seconds it is considered as failure. Note that the
timeout value of 30 seconds is no system internal value.
%100#
#[%] ∗=
attemptsattachGPRSoftotal
attemptsattachGPRSulunsuccessfRatioFailureAttachGPRS
Trigger Point(s):
Check if the whole GPRS attach procedure from sending the Attach Request and
receiving the Attach Accept message was executed successfully (Signaling messages 1
and 8 in Figure 1).
7d. Cancel Location Ack
7c. Cancel Location
7b. Update Location
7g. Update Location Ack
7e. Insert Subscriber Data
7f. Insert Subscriber Data Ack
6d. Insert Subscriber Data
6c. Cancel Location Ack
6b. Cancel Location
3. Identity Response
2. Identification Response
2. Identification Request
1. Attach Request
5. IMEI Check
3. Identity Request
4. Authentication
6a. Update Location
7a. Location Update Request
7h. Location Update Accept
6f. Update Location Ack
6e. Insert Subscriber Data Ack
MS BSS new SGSN old SGSN GGSN HLR EIR
old
MSC/VLR
new
MSC/VLR
9. Attach Complete
8. Attach Accept
10. TMSI Reallocation Complete
Figure 1: Signalling Messages during Attach Procedure
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
Remarks:
It might occur that the MS sends more than one Attach Request towards the SGSN,
since retries are necessary. Maximum four retries are possible and the guard time
between two requests is 15 seconds. These retries should not have impact on the Attach
Failure Ratio, since only one GPRS/EDGE Attach Request message should be counted
in the calculation.
Prerequisites:
GPRS/EDGE has to be active in the cell used by a subscriber (PI ACC 010)
Active Measurement:
This PI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
GPRS Attach, TAG: ST 010.
Passive Measurement:
This PI can be measured with a counter in the /// SGSN.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.2.3 GPRS PDP Context Activation Failure Ratio
TAG: ACC 030
Definition:
The PI GPRS PDP Context Activation Failure Ratio [%] denotes the probability that a
subscriber cannot activate a PDP context.
The timeout interval for the PDP context activation procedure is 30 seconds, meaning: if
the activation procedure was not completed after 30 seconds it is considered as failure.
Note that the timeout value of 30 seconds is no system internal value.
%100#
#[%] ∗=
attemptsactivationcontextPDPoftotal
attemptsactivationcontextPDPulunsuccessfRatioFailureActivationContextPDP
Trigger Point(s):
Check if the PDP context activation procedure from sending the Activate PDP Context
Request and receiving the Activate PDP Context Accept message was executed
successfully. (Signaling messages 1 and 7 in Figure 2.)
2G-GGSN
7. Activate PDP Context Accept
5. Create PDP Context Response
5. Create PDP Context Request
1. Activate PDP Context Request
2G-SGSN BSS
2. Security Functions
MS
6. BSS Packet Flow Context Procedures
4. Invoke Trace
Figure 2: Signaling Messages during the PDP Context Activation Procedure
Remarks:
It might occur that the MS sends more than one PDP Context Activation Request
towards the SGSN, since retries are necessary. Maximum four retries are possible and
the guard time between two requests is 30 seconds. These retries should not have impact
on the Activation Failure Ratio, since only one GPRS PDP Context Activation Request
message should be counted in the calculation.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
PDP Context Activations that failed, because they were done towards an undefined APN
should not be included in the calculation.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020)
Active Measurement:
This PI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
PDP Context Activation, TAG: ST020.
Passive Measurement:
This PI can be measured with a counter in the /// SGSN-G 3.0.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.2.4 Secondary PDP Context Activation Failure Ratio
TAG: ACC 035
Definition:
The PI Secondary PDP Context Activation Failure Ratio [%] denotes the probability that
a subscriber cannot activate a Secondary PDP context.
The timeout interval for the Secondary PDP context activation procedure is 30 seconds,
meaning: if the activation procedure was not completed after 30 seconds it is considered
as failure. Note that the timeout value of 30 seconds is no system internal value.
%100#
#[%] ∗=
attemptsactivationcontextPDPoftotal
attemptsactivationcontextPDPulunsuccessfRatioFailureActivationContextPDP
Trigger Point(s):
Check if the PDP context activation procedure from sending the Activate Secondary
PDP Context Request and receiving the Activate Secondary PDP Context Accept
message was executed successfully. (Signaling messages 1 and 7 in Figure 2 and Figure
4.)
2G-GGSN
7. Activate Secondary PDP Context Accept
3. Create PDP Context Response
3. Create PDP Context Request
1. Activate Secondary PDP Context Request
2G-SGSN BSS
2. Security Functions
MS
5. BSS Packet Flow Context Procedures
Figure 3: Secondary PDP Context Activation Procedure for GPRS
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
3G-GGSN
7. Activate PDP Context Accept
3. Create PDP Context Response
3. Create PDP Context Request
1. Activate Secondary PDP Context Request
3G-SGSN UTRAN MS
4. Radio Access Bearer Setup
6. Update PDP Context Response
6. Update PDP Context Request
Figure 4: Secondary PDP Context Activation Procedure for UMTS
Remarks:
It might occur that the MS sends more than one PDP Context Activation Request
towards the SGSN, since retries are necessary. Maximum four retries are possible and
the guard time between two requests is 30 seconds. These retries should not have impact
on the Activation Failure Ratio, since only one GPRS PDP Context Activation Request
message should be counted in the calculation.
Prerequisites:
The MS is already GPRS attached and PDP context activated. (PI ACC 020 & PI
ACC 030)
Active Measurement:
This PI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
PDP Context Activation, TAG: ST020.
Passive Measurement:
There is no means of passive measurement (counter or Moniq tool) for this PI.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.2.5 GPRS Service Access Failure Ratio (for each service)
TAG: ACC 040
Definition:
The PI GPRS Service Access Failure Ratio [%] denotes the probability that a subscriber
cannot access the service successfully, meaning that the data transfer of the content
could not be started.3
%100#
#[%] ∗=
attemptsaccessserviceoftotal
attemptsaccessserviceulunsuccessfRatioFailureAccessService
The trigger points as well as the timeout values have to be defined per service for PI
ACC 040.
Trigger Point(s):
The trigger points are defined per service:
FTP (UL)
Start trigger: MS sends its port (signaling message 2 in Figure 5).
Stop trigger: Sending of the first data packet (message 11 in Figure 5).
Figure 5: FTP UL trace
Note: The actual start trigger is the TCP SYN for port 21. If this SYN is not send the
port is not opened and the service could not be accessed. This signalling
message is not visible in the above log since the port was opened much earlier.
3 Content denotes different things depending on the service that is accessed. In case of an FTP session content is a
file, in the case of an http session a web page and content of an E-Mail session is the text of the mail and so on.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
FTP (DL)
Start trigger: MS sends its port (signaling message 1 in Figure 6).
Stop trigger: Reception of the first data packet (message 10 in Figure 6).
Figure 6: FTP DL trace
Note: The actual start trigger is the TCP SYN for port 21. If this SYN is not send the
port is not opened and the service could not be accessed. This signalling
message is not visible in the above log since the port was opened much earlier.
E-Mail POP3 (Receiving)
Start trigger: TCP SYN sent by the MS (signaling message 2 in Figure 7).
Stop trigger: Reception of the first data packet containing mail content by the MS
(message 18 in Figure 7)
Figure 7: E-Mail POP3 trace
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
E-Mail SMTP (Sending)
Start trigger: TCP SYN sent by the MS (signaling message 2 in Figure 8).
Stop trigger: Sending of the first data packet containing mail content (message 15 in
Figure 8).
Figure 8: E-Mail SMTP trace
HTTP
Start trigger: TCP SYN sent by the MS (signaling message 2 in Figure 9).
Stop trigger: Reception of the first data packet and OK for http 1.1 or 1.0 (message 7 in
Figure 9).
Figure 9: HTTP trace
Service Access Timeouts:
The timeout values summarized in Table 2 are only applicable for active measurements
of the PI Service Access Failure Ratio.
FTP E-Mail HTTP
Timeout 30s 60s10s per
attempt
Service Access
Table 2: Service Access Timeouts
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
These timeout values are no system internal values but are expected to mirror the user
behavior. If – for example - an ftp file-transfer did not start after 30 seconds the end-user
will stop this transaction. Similarly, if the end-user does not perceive that the loading of
a web page started after 10 seconds he will try to reload the page. But if these reload
attempts are not successful he will stop accessing the web page.
Remarks:
The above-mentioned trigger points for the different services are defined on a rather
technical level and not from a total end-user’s point of view. An end-user would
consider a service access as successful if the complete data transfer was successful.
Measuring the PI Service Access Failure Ratio related to the transfer of the complete
data content would mean that the measured ratio and as well the service access delay
would depend on the file size.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020) and a PDP Context is activated (PI
ACC 030)
Active Measurement:
This PI can be measured by executing the following test cases described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
D)S Lookup, TAG: SET 010,
FTP UL/DL, TAG: SET 020,
WAP Traffic, TAG: SET 030,
WWW Traffic, TAG: SET 040,
E-mail Traffic, TAG: SET 050.
Passive Measurement:
This PI can be measured with the Moniq tool:
• WAP Connection Establishment Success Ratio (5.2.5.1)
• Web connection failure reasons: aborts, protocol error, congestion
• POP3, Mail Server Connection Success Ratio (5.2.5.2)
• Domain Name Service Query Success Ratio (5.2.5.3)
5.2.5.1 WAP Connection Establishment Success Ratio
TAG: ACC WAPconnect
The number of successful connection establishments divided by the total number of
connection attempts.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.2.5.2 POP3, Mail Server Connection Success Ratio
TAG: ACC POP3connect
The number of successful connection establishments divided by the total number of
connection attempts.
5.2.5.3 Domain Name Service Query Success Ratio
TAG: ACC DNS
The number of successful domain name (DNS) resolutions divided by the total number
of DNS requests. A successful resolution means that the server has found the name and
has responded to the request with a valid name.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.2.6 GPRS Accessibility Failure Ratio (for each service)
TAG: ACC 050
Definition:
The KPI GPRS e2e Accessibility Failure Ratio [%] denotes the probability that a
subscriber cannot access the service successfully, meaning that the data transfer of the
content could not be started due to a failure that occurred within the complete chain of
service access:
GPRS Availability � Attach � PDP Context Activation � Service Access
( ) ( ) ( ) ( )[ ]04010301020101011[%] ACCPIACCPIACCPIACCPIRatioFailureAccess −∗−∗−∗−−=
Trigger Point(s):
Trigger points and timeout values that are defined for the involved PIs:
• GPRS Unavailability (PI ACC 010)
• GPRS Attach Failure Ratio (PI ACC 020)
• GPRS PDP Context Activation Failure Ratio (PI ACC 030), and
• GPRS Service Access Failure Ratio (for each service) (PI ACC 040)
are valid for measuring GPRS e2e Accessibility failure ratio. See also Figure 15:
Overview of the KPIs.
Example:
Assuming 1000 measurements and the following measured PIs ACC 010, ACC 020,
ACC 030 and ACC 040:
GPRS Unavailability (PI ACC 010): 0%
� GPRS was available all the time � 1000 samples remain for the attach procedure
GPRS Attach Failure Ratio (PI ACC 020): 30%
� results in 700 samples for PDP context activation
GPRS PDP Context Activation Failure Ratio (PI ACC 030): 15%
� results in 595 samples for the service access, and
(PI ACC 040): 10%
� 535.5 service access attempts out of 1000 were successful and in 464.5 cases the
service could not be access.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
( ) ( ) ( ) ( )[ ]
( ) ( ) ( ) ( )[ ]
( )[ ]
[ ]
%45.46
%100*5355.01
%100*9.0*85.0*7.0*11
%100*1.01̀*15.01*3.01*011
06010401020101011
[%]
=
−=
−=
−−−−−=
−∗−∗−∗−−= ACCPIACCPIACCPIACCPI
RatioFailureAccess
Active Measurement:
KPI is not covered in the Generic Test Description for GPRS/EDGE End-to-End
Performance Testing.
Passive Measurement:
This KPI can be measured with the Moniq tool:
• Web/FTP/E-mail service interrupts
Note: The Web and SMTP success ratios are measured by dividing the successful
service completions with the total number of service attempts (including those
cases where the service access was already unsuccessful). The reason for this
small difference is that the service access and data transfer are not separated in
Moniq for these services.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.3 Retainability KPIs
5.3.1 GPRS PDP Context Cut-off Ratio
TAG: RET 010
Definition:
The PI GPRS PDP Context Cut-off Ratio [%] denotes the probability that a PDP context
is deactivated without being initiated by the MS.
%100#
#[%] ∗=
contextsPDPactivatedtotal
lossescontextPDPRatioOffCutContextPDP
Trigger Point(s):
Different trigger points for a PDP Context Deactivation not initiated by the MS are
possible: SGSN failure or GGSN failure on which the PDP context will be deactivated
by the SGSN or GGSN respectively.
Expiry of the Mobile )ot Reachable Timer leads to PDP Context Deactivation and is
therefore another reason for PDP context cut-off.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020) and a PDP Context is activated (PI
ACC 030)
Active Measurement:
PI not covered in the Generic Test Description for GPRS/EDGE End-to-End
Performance Testing.
Passive Measurement:
This PI can be measured with the help of counters in the /// GGSN.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.3.2 GPRS Data Transfer Cut-off Ratio
TAG: RET 020
Definition:
The KPI GPRS Data Transfer Cut-off Ratio [%] denotes the probability that a data
transfer is incomplete after a timeout period or the transfer is aborted completely.
%100#
#[%] ∗=
transfersdatatotal
transfersdataincompleteRatioOffCutTransferData
Trigger Point(s):
1. Transfer abort � file transfer is not complete.
2. PDP context cut-off (PI RET 010) � file transfer is not complete.
3. FTP: Timeout counter of 90 seconds is triggered if no data reaches the receiver
during 90 seconds � file transfer is not complete.
Remarks:
In case of a data transfer cut-off it should be determined how much data was transmitted
before this cut-off.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
Round Trip Time, TAG: TT 020,
FTP UL/DL, TAG: SET 020,
WAP Traffic, TAG: SET 030,
WWW Traffic, TAG: SET 040,
E-mail Traffic, TAG: SET 050.
Passive Measurement:
This KPI can be measured with the Moniq tool:
• WAP Object Download Success Ratio (5.3.2.1)
• Web Object Download Success Ratio (5.3.2.2)
• FTP Download Success Ratio (5.3.2.3)
• POP3, E-mail Download Success Ratio (5.3.2.4)
• SMTP, E-mail Send Success Ratio (5.3.2.5)
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.3.2.1 WAP Object Download Success Ratio
TAG: RET WAPobject
The number of successfully finished and acknowledged object downloads divided by all
download attempts. Connection-mode WAP is measured. Precondition: established
WAP session.
5.3.2.2 Web Object Download Success Ratio
TAG: RET WEB
The number of successfully finished HTTP object downloads divided by all HTTP
download attempts.
5.3.2.3 FTP Download Success Ratio
TAG: RET FTP
The number of successfully finished file downloads divided by the number of all FTP
download attempts. Condition: connection to FTP server was successful.
5.3.2.4 POP3, E-mail Download Success Ratio
TAG: RET POP3download
The number of successfully finished mail downloads divided by all e-mail download
attempts. Condition: connection to the server was successful, and a request to download
(at least) one mail was initiated. This last condition is important since many sessions just
check the mailbox and do not attempt any mail download.
5.3.2.5 SMTP, E-mail Send Success Ratio
TAG: RET SMTPdownload
The number of successfully sent e-mails divided by the total number of attempts using
the SMTP protocol. Web-based e-mail services are not considered. (Note: all common e-
mail applications Netscape, Outlook, and mobile phone integrated clients use the SMTP
protocol.)
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.3.3 Cell Update Failure Ratio
TAG: RET 030
Definition:
The GPRS/EDGE KPI Cell Update Failure Ratio [%] denotes the probability that a Cell
Update procedure fails.
%100#
#[%] ∗=
attemptsUpdateCelloftotal
attemptsUpdateCellulunsuccessfRatioFailureUpdateCell
Note: Since a cell update can only be observed in READY state the cell update failure
ratio can be measured best during an ongoing FTP session.
Trigger Point(s):
The cell update procedure failed when the data transfer is not continued (sending or
receiving) after the MS entered the new cell.
MS BSS SGSN
FLUSH-LL-ACK
UL-UNITDATA (old cell)
DL-UNITDATA (old cell), TCP retransmission!
FLUSH-LL (old BVCI and new BVCI)
DL-UNITDATA (new cell)
DL-UNITDATA (old cell)
DL-UNITDATA (old cell)
UL-UNITDATA (new cell)
Figure 10: Signalling diagram for cell update during DL FTP
Prerequisites:
The MS is already GPRS attached. (PI ACC 020) and a PDP Context is activated (PI
ACC 030). Two different cells belonging to the same RA configured with GPRS/EDGE
capability.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
Cell Update during Downlink FTP, TAG: MT 010.
This KPI can be measured by executing the UDP based test solution: Transport Tests, IP
layer outage as described in detail in Proposed Extensions to GPRS e2e performance
Testing [2].
Passive Measurement:
There is no means of passive measurement (counter or Moniq tool) for this KPI.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.3.4 Intra SGSN RAU Failure Ratio
TAG: RET 040
Definition:
The KPI Intra SGSN RAU Failure Ratio [%] denotes the probability that an Intra SGSN
RAU procedure fails.
%100#
#[%] ∗=
attemptsRAUSGS�Intraoftotal
attemptsRAUSGS�IntraulunsuccessfRatioFailureRAUSGS�Intra
Trigger Point(s):
Check if the SGSN RAU procedure from sending the Routing Area Update Request and
receiving the Routing Area Update Accept message was executed successfully.
(Signaling messages 1 and 3 in Figure 11.)
1. Routeing Area Update Request
3. Routeing Area Update Accept
2. Security Functions
MS BSS SGSN
4. Routeing Area Update Complete C1
Figure 11: Intra SGSN RAU Procedure
Prerequisites:
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI
ACC 030). Two different cells belonging to different RAs served by the same SGSN
configured with GPRS/EDGE capability.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
Intra SGS) RA Update during Downlink FTP, TAG: MT 020
Passive Measurement:
This KPI can be measured with counters in the /// SGSN.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.3.5 Inter SGSN RAU Failure Ratio
TAG: RET 050
Definition:
The KPI Inter SGSN RAU Failure Ratio [%] denotes the probability that an Inter SGSN
RAU procedure fails.
%100#
#[%] ∗=
attemptsRAUSGS�Interoftotal
attemptsRAUSGS�InterulunsuccessfRatioFailureRAUSGS�Inter
Trigger Point(s):
Check if the SGSN RAU procedure from sending the Routing Area Update Request and
receiving the Routing Area Update Accept message was executed successfully.
(Signaling messages 1 and 11 in Figure 12.)
MS BSS new SGSN HLR GGSN old SGSN
2. SGSN Context Response
3. Security Functions
1. Routeing Area Update Request
2. SGSN Context Request
6. Update PDP Context Request
6. Update PDP Context Response
7. Update Location
10. Update Location Ack
11. Routeing Area Update Accept
8. Cancel Location
8. Cancel Location Ack
9. Insert Subscriber Data Ack
9. Insert Subscriber Data
12. Routeing Area Update Complete
5. Forward Packets
4. SGSN Context Acknowledge
Figure 12: Inter SGSN RAU Procedure
Prerequisites:
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI
ACC 030). Two different cells belonging to different RAs served by the different
SGSNs configured with GPRS/EDGE capability.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
Inter SGS) RA Update during Downlink FTP, TAG: MT 030.
Passive Measurement:
This KPI can be measured with counters in the /// SGSN.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.3.6 PS Paging Failure Ratio
TAG: RET 060
Definition:
The KPI PS Paging Failure Ratio [%] denotes the probability that an SGSN Paging
procedure fails.
%100#
#[%] ∗=
attemptsPagingSGS oftotal
attemptsPagingSGS ulunsuccessfRatioFailurePagingPSSGS
Trigger Point(s):
The network shall initiate the paging procedure for GPRS services using P-TMSI when
GMM signaling messages or user data is pending to be sent to the MS while the Mobile
Reachable timer is running. The network may page only GPRS MSs, which are GMM-
REGISTERED and identified by a local P-TMSI. The SGSN requests the RR sublayer to
start paging. Upon reception of a paging indication, the MS shall respond to the paging
with any LLC frame. [7]
Check if the Paging for GPRS Services procedure from sending the Paging Request by
the SGSN and receiving any LLC frame send by the mobile was executed successfully.
Prerequisites:
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI
ACC 030).
Active Measurement:
KPI not covered in the Generic Test Description for GPRS/EDGE End-to-End
Performance Testing.
Passive Measurement:
This KPI can be measured with counters in the /// SGSN.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.4 Integrity KPIs
5.4.1 GPRS Attach Setup Time
TAG: INT 010
Definition:
The PI GPRS Attach Setup Time [ms] denotes the length of the time period it takes to
attach to the GPRS/EDGE network.
Note: Difference between an attach of a known subscriber and an unknown subscriber
will be reflected in the length of the time period indicating the Attach Setup
Time. In case of an attach of an unknown subscriber (meaning that the SGSN
has changed since the GPRS detach, or if it is the very first attach of the mobile
to the network), the SGSN contacts the HLR in order to receive the subscriber
data.
The attach setup time of an unknown subscriber will be slightly longer than the
one of a known subscriber.
Trigger Point(s):
Measurement of the time [ms] from sending the Attach Request and receiving the Attach
Accept message, see Figure 1.
Remarks:
While determining the average Attach Setup Time only successful attach attempts are
included in the calculations. Failed attach attempts should not be part of the calculation
denoting a timeout period of 30 seconds. Note that the timeout value of 30 seconds is no
system internal value.
Prerequisites:
GPRS/EDGE has to be active in the cell used by a subscriber (PI ACC 010)
Active Measurement:
This PI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
GPRS Attach, TAG: ST 010.
Passive Measurement:
There is no means of passive measurement (counter or Moniq tool) for this PI.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.4.2 GPRS PDP Context Activation Delay (per APN)
TAG: INT 020
Definition:
The PI GPRS PDP Context Activation Delay Time [ms] denotes the length of the time
period it takes to activate a PDP context.
Trigger Point(s):
Measurement of the time [ms] from sending the Activate PDP Context Request and
receiving the Activate PDP Context Accept message, see Figure 2.
Remarks:
While determining the average PDP Context Activation Delay only successful activation
attempts are included in the calculations. Failed activation attempts should not be part of
the calculation denoting a timeout period of 30 seconds. Note that the timeout value of
30 seconds is no system internal value.
The PDP Context Activation Delay should be determined per APN.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020).
Active Measurement:
This PI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
PDP Context Activation, TAG: ST 020
Passive Measurement:
There is no means of passive measurement (counter or Moniq tool) for this PI.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.4.3 GPRS Service Access Time (for each service)
TAG: INT 030
Definition:
The PI GPRS Service Access Time [s] denotes the length of the time period it takes to
access a service successfully.
Trigger Point(s):
Measurement of the time [s] according to the trigger points defined for PI ACC 040 (see
Figure 5 - Figure 9).
Remarks:
While determining the Access Time for a specific service only successful access
attempts are included in the calculations. Failed access attempts should not be part of the
calculation.
The Access Time should be determined per service, since the actual access time depends
strongly on the nature of the service.
For e-mail and ftp the transmission rate is of higher interest than the actual service
access time.
Remarks:
The above-mentioned trigger points for the different services are defined on a rather
technical level and not from a total end-user’s point of view. An end-user would be
rather interested in the time needed for transferring the complete data content.
Measuring the PI Service Access Time related to the transfer of the complete data
content would mean that the service access delay would depend on the file size.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020) and a PDP Context is activated (PI
ACC 030)
Active Measurement:
This PI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
D)S Lookup, TAG: SET 010,
FTP UL/DL, TAG: SET 020,
WAP Traffic, TAG: SET 030,
WWW Traffic, TAG: SET 040,
E-mail Traffic, TAG: SET 050.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
Passive Measurement:
This PI Service Access Time can be measured with the Moniq tool:
• WAP Connection Setup Delay (5.4.3.1)
• Web Server Response Time (5.4.3.2)
• POP3, Mail Server Connection Time (5.4.3.3)
Measuring the time for transmitting the data content starting after the service access
itself (see Figure 13) reflects the end-user’s perspective and can be measured with the
Moniq tool:
• WAP Object Download Delay (5.4.3.4)
• Web small object download time (5.4.3.5)
• POP3, Mail Download Time (5.4.3.6)
• SMTP, Mail Sending Time (5.4.3.7)
• Domain Name Service Response Delay (5.4.3.8)
Service Access Time
Connection established/
Start of data transferMobile requests
service
Data Transfer Time
Data transfer complete
Service Access Time
Connection established/
Start of data transferMobile requests
service
Data Transfer Time
Data transfer complete
Figure 13: Service Access & Data Transfer
5.4.3.1 WAP Connection Setup Delay
TAG: INT WAPconnect
The average delay between the setup-packet from the mobile is received at the monitor
until the mobile successfully acknowledges the server response packet and the ACK
packet passes the monitor. The measured delay is the full setup round-trip time and it
includes the following delay components: Monitor � WAP Server � Mobile �
Monitor.
Precondition:
Successful connection setup
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.4.3.2 Web Server Response Time
TAG: INT WEBresponse
The average time until the server starts to transmit data measured from the first packet
from the mobile. This measure includes the time it takes to set up a connection over the
GPRS/EDGE and wired Internet, the client to deliver the request to the server, the server
to process the request, and to respond to it. The condition is that only successful
response is measured, but the content of the response may be either success or failure.
5.4.3.3 POP3, Mail Server Connection Time
TAG: INT POP3connect
The average time it takes to connect to the POP3 e-mail server. The connection time
includes the time to establish the TCP connection over the GPRS/EDGE and Internet
path, the server to send a greeting message, and the time to log into the server.
Condition: successfully established connections are measured.
PIs denoting the time for transmitting the data content starting after the service access
itself that can be measured with the Moniq tool:
5.4.3.4 WAP Object Download Delay
TAG: INT WAPdownload
The average delay between the request packet from the mobile until the server response
is successfully acked by the mobile.
Precondition:
Successfully finished download
5.4.3.5 Web Small Object Download Time (9-11kbyte)
TAG: INT WEBsmall
The average time it takes to download a small HTTP object. Only small (9-11 Kbytes)
files are measured. Due to the TCP protocol, small objects are not able to utilize the full
GPRS/EDGE throughput (timeslot capacity). The download time of such small objects
depends more on the round-trip time
Remarks:
To have comparable results between different measurements and operators, only objects
that fall into a narrow range are measured. The chosen range of 9-11 kbytes was chosen
because the median of Web downloads falls in this range. Condition: object download
was successful.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.4.3.6 POP3, Mail Download Time
TAG: INT POP3download
This KPI defines the average time to successfully download one or more e-mails from
the POP3 e-mail server. The whole time is measured including server greeting,
authentication, and time to download of all mails and quit. Condition: mail download
was successful.
5.4.3.7 SMTP, Mail Sending Time
TAG: INT SMTPsend
This KPI defines the average time for the mobile to send an e-mail using the SMTP
protocol. Condition: e-mail sending was successful.
5.4.3.8 Domain Name Service Response Delay
TAG: INT DNSdelay
This KPI defines the average delay for the DNS server to respond to a DNS request,
counting retransmissions. The time is calculated between the request packet seen in the
uplink direction and the response packet seen in the downlink direction at the monitor.
Condition: resolution was successful.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.4.4 GPRS Access Time (for each service)
TAG: INT 040
Definition:
The KPI GPRS e2e Access Time [s] denotes the length of the time period it takes to
access a service successfully, measuring the time period for the complete chain:
GPRS Availability � Attach � PDP Context Activation � Service Access
030020010][ I�TPII�TPII�TPIsTimeAccess ++=
Trigger Point(s):
Trigger points and timeout values that are defined for the involved KPIs:
• GPRS Attach Setup Time (PI I�T 010)
• GPRS PDP Context Activation Delay (per APN) (PI I�T 020), and
• GPRS Service Access Time (for each service) (PI I�T 030)
are valid for measuring GPRS e2e Access Time.
Remarks:
The lower the timeout values are for the different Access KPIs the higher the failure
ratio is. At the same time a lower timeout value goes along with a lower access time,
since the timeout value is the upper boundary for the measured time period it takes to
perform a certain procedure, i.e. attach, PDP Context activation or service access.
Active Measurement:
KPI not covered in the Generic Test Description for GPRS/EDGE End-to-End
Performance Testing.
Passive Measurement:
There is no means of passive measurement (counter or Moniq tool) for this KPI.
Attach Setup
Time
PDP Context
Activation
Time
Service Access
Time
Attach Setup
Time
PDP Context
Activation
Time
Service Access
Time
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.4.5 GPRS Mean User Data Rate
TAG: INT 050
Definition:
The KPI GPRS Mean User Data Rate [kbits/s] denotes the average data rate in kbits/s.
)()(
8)(]/[
sTriggerStartofTimesTriggerStopofTime
kByteSizeFileskbitsRateDataMean
−
∗=
The trigger points have to be defined per service for KPI I�T 050.
Trigger Point(s):
The average throughput is measured from opening the data connection to the end of the
successful transfer of content (file, e-mail or web page).
The trigger points are defined per service (FTP DL/UL, E-Mail DL/UL, HTTP) in the
same way as they are defined for GPRS Data Transfer Cut-off Ratio (PI RET 020).
Refer to chapter 5.3.2.
Prerequisites:
The MS is already GPRS attached (PI ACC 020), a PDP Context is activated (PI ACC
030) and a service was accessed successfully (PI ACC 040).
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
Round Trip, Time, TAG: TT 020,
Controlled PDCH Sharing, TAG: TT 030,
Cell Update during DL FTP, TAG: MT 010,
Intra SGS) RA Update during DL FTP, TAG: MT 020,
Inter SGS) RA Update during DL FTP, TAG:MT 030,
FTP UL/DL, TAG: SET 020,
WWW Traffic, TAG: SET 040,
E-mail Traffic, TAG: SET 050.
This KPI can be measured by executing the UDP based test solution: Transport Tests,
RAW throughput as described in detail in Proposed Extensions to GPRS e2e
performance Testing [2].
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
Passive Measurement:
This KPI can be measured for Web and FTP throughput in DL direction with the Moniq
tool:
• FTP Download Rate (5.4.5.1)
• Web Large Object Download Rate (larger than 50kbyte) (5.4.5.2)
• End-to-End Achievable Throughput (5.4.5.3)
• TCP Goodput/Throughput Ratio (5.4.5.4)
5.4.5.1 FTP Download Rate
TAG: INT FTPrate
The average rate is the size of the downloaded file divided by the time it takes to
download it. Only large files (> 50 kbyte) are measured, when the available end-to-end
path capacity dominates the measure and not the file size.
Precondition:
File download was successful
5.4.5.2 Web Large Object Download Rate (larger than 50kbyte)
TAG: INT WEBlarge
The average rate is the size of the object divided by the time it takes to successfully
download the object. This measure is also called goodput. Only large objects are
measured, when the available end-to-end path capacity dominates the measure and not
the object size. Condition: object download was successful.
5.4.5.3 End-to-End Achievable Throughput
TAG: INT THRHPUT
This KPI measures the average achievable IP layer throughput end-to-end. This KPI is
measured by tracing those mobiles that generate one or more parallel TCP downloads
saturating the downlink channel. In an optimal GPRS/EDGE network this KPI shows the
capacity of the available GPRS/EDGE data channel.
5.4.5.4 TCP Goodput/Throughput Ratio
TAG: INT THRHPUT/GPUT
The ratio of user data payload in TCP connections (Web, e-mail, FTP, etc.) divided by
the total IP layer data transmitted to the mobile terminal counted at the monitoring point.
The ratio depends on the protocol overheads (TCP/IP) and the retransmissions. 100% is
not possible; there is an upper limit even in an optimal network. The distance from this
optimal value has to be evaluated.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.4.6 Round Trip Time
TAG: INT 060
Definition:
The KPI Round Trip Time (RTT) [s] denotes the length of the time period it takes for a
data packet to be sent from the MS and being returned to the sender.
Note: The ping program is a suitable method for measuring RTT.
Either a DNS server or an external Internet server can be pinged by the MS.
Trigger Point(s):
Since the ping program uses the ICMP echo request and echo reply messages (not using
the transport layer TCP or UDP) the start trigger is the ICMP echo request message and
the stop trigger the ICMP echo reply message.
Remarks:
The ping size has influence on the RTT, therefore measuring the KPI Round Trip Time
should be done with different ping sizes.
Furthermore the delay introduced by the Internet and the response time of the server
influence the RTT.
Prerequisites:
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI
ACC 030).
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
Round Trip Time, TAG: TT 010.
Passive Measurement:
This KPI can be measured with the Moniq tool:
• End-to-End Round Trip Delay (5.4.6.1)
• Client-side Round Trip Delay (5.4.6.2)
• Server-side Delay (5.4.6.3)
Remarks:
Packet delay is measured for TCP protocol exchanges, when the mobile client initiates a
TCP setup by sending a SYN packet. The server responds with a SYN/ACK packet,
which is finally acknowledged by the client with an ACK packet. Thus the TCP setup
includes a full end-to-end round-trip between the client and the server. The monitor
calculates the times between these packets, thus it can separate the delay for client and
server-side components.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.4.6.1 End-to-End Round Trip Delay
TAG: INT DELAYe2e
The average time measured on the full client-server-client round-trip path. It is usually
dominated by the GPRS access delay (client-side delay).
Remarks:
Packet delay is measured for TCP protocol exchanges, when the mobile client initiates a
TCP setup by sending a SYN packet. The server responds with a SYN/ACK packet,
which is finally acknowledged by the client with an ACK packet.
5.4.6.2 Client-side Round Trip Delay
TAG: INT DELAYgprs
The average time measured between the SYN/ACK and the ACK packets. This measure
includes the time over the GPRS channel and the client processing time.
Remarks:
Packet delay is measured for TCP protocol exchanges, when the mobile client initiates a
TCP setup by sending a SYN packet. The server responds with a SYN/ACK packet,
which is finally acknowledged by the client with an ACK packet.
5.4.6.3 Server-side Delay
TAG: INT DELAYinternet
The average time measured between the SYN and SYN/ACK packets. This measure
includes the delay over the core and Internet to the server, and the processing time in the
server. The propagation delay and the buffer congestion in the wired path of the
connections dominate this measure. The propagation delay is usually proportional to the
geographical distance (propagation time at speed of light). The processing time in
servers is usually negligible.
Remarks:
Packet delay is measured for TCP protocol exchanges, when the mobile client initiates a
TCP setup by sending a SYN packet. The server responds with a SYN/ACK packet,
which is finally acknowledged by the client with an ACK packet.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.4.7 Client-Server Interaction Time with Background Traffic
TAG: INT 065
Definition:
The KPI Client Server Interaction Time denotes the time [s] it takes to send a small data
packet to the server and receive it back (RTT) during an ongoing DL TCP data transfer
(FTP) in the background.
Remarks:
The typical use case for this KPI is a user that downloads his emails or a big file from
the Internet in the background and starts at the same time a web session. For the web
session an HTTP GET will be sent to the web server and the server will respond with an
HTTP OK and start to send the content of the web page to the user. The user expects an
answer to his request of the web page in a reasonable time.
Note: If a big advertised TCP Window Size is used and the system buffers a lot of data,
it might happen that the data from the background traffic delays the response from the
web server to the user.
Trigger Point(s):
Since the ping program uses the ICMP echo request and echo reply messages (not using
the transport layer TCP or UDP) the start trigger is the ICMP echo request message and
the stop trigger the ICMP echo reply message.
Prerequisites:
The MS is already GPRS attached (PI ACC 020), a PDP Context is activated (PI ACC
030) and background traffic (FTP DL data transfer) is ongoing.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
Round Trip Time during Background Traffic, TAG: TT 025.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.4.8 Packet Loss Ratio
TAG: INT 070
Definition:
The KPI Packet Loss Ratio [%] denotes the rate of data packets, which are lost
regardless where they are lost in the network.
Note: The ping program is a suitable method for measuring the packet loss rate.
Trigger Point(s):
The start trigger is the ICMP echo request message and the stop trigger the ICMP echo
reply message.
The ping program should be used with different packet sizes, since sometimes only
packets of a special size are lost: The following packet sizes should be used: 32 byte,
548 byte and 1472 byte.
A ping command with a length of 548 byte would result in a total IP packet size of 576
byte (548 byte + 28 byte ICMP header). For the case of 1472 byte ICMP data the total IP
packet size would be 1500 byte. 576 and 1500 byte are typical IP packet sizes when
using TCP.
Remarks:
The Packet Loss ratio has big impact on the application performance.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020) and a PDP Context is activated (PI
ACC 030).
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
Round Trip Time, TAG: TT 010,
TCP/IP Transfer, TAG: TT 020,
WAP Traffic, TAG: SET 030.
Passive Measurement:
This KPI can be measured with the Moniq tool:
• WAP Retransmission Ratio (5.4.8.1)
• Downlink End-to-End Packet Loss (5.4.8.2)
• Server to Monitor Loss (5.4.8.3)
• Monitor to Client Loss (5.4.8.4)
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.4.8.1 WAP Retransmission Ratio
TAG: INT WAPretransmission
The average number of packet retransmissions seen during WAP object downloads.
Retransmissions greatly affect the delay of page downloads. All object downloads are
measured regardless of their eventual result (success or failure).
5.4.8.2 Downlink End-to-End Packet Loss
TAG: INT LOSSe2e
The ratio of TCP packets lost downlink, between the server and the client end-to-end.
Typically the GPRS/EDGE channel access loss dominates this measure.
5.4.8.3 Server to Monitor Loss
TAG: INT LOSSinternet
The ratio of TCP (Web, e-mail, FTP, etc.) packets lost between the Internet server and
the monitoring point in the downlink direction. This KPI indicates the level of
congestion over the wired path from the server up to the monitor.
5.4.8.4 Monitor to Client Loss
TAG: INT LOSSgprs
The ratio of TCP (Web, e-mail, FTP, etc.) packets lost between the monitoring point and
the mobile terminal in the downlink direction. This measure characterizes the GPRS
access part of the path (GSN, PCU buffers and RLC/LLC loss).
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
5.4.9 Cell Update Time
TAG: INT 080
Definition:
The GPRS KPI Cell Update Time [s] denotes the length of the time period a cell update
takes.
Note: Since a cell update can only be observed in READY state the duration of a cell
update can be measured best during an ongoing FTP session.
Trigger Point(s):
Start measuring the time interval when the last IP frame is sent/received in the old cell
and stop measuring when the first IP frame is sent/received in the new cell by the MS.
Remarks:
In live networks parameters like the moving speed, the cell configuration and the MS
have big influence on the time needed for the cell update.
Prerequisites:
The MS is already GPRS attached. (PI ACC 020) and a PDP Context is activated (PI
ACC 030). Two different cells belonging to the same RA configured with GPRS/EDGE
capability.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
Cell Update during DL FTP, TAG: MT 010
This KPI can be measured by executing the UDP based test solution: Transport Tests, IP
layer outage as described in detail in Proposed Extensions to GPRS e2e performance
Testing [2].
Passive Measurement:
There is no means of passive measurement (counter or Moniq tool) for this KPI.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.4.10 Intra SGSN RAU Time
TAG: INT 090
Definition:
The KPI Intra SGSN RAU Time [s] denotes the length of the time period an Intra SGSN
RAU takes.
Note: The duration of an Intra SGSN RAU can be measured best during an ongoing
FTP session.
In most of the case the RA belongs to a new LA, so that the MS performs a
LAU as well.
Trigger Point(s):
Start measuring the time interval when the last IP frame is sent/received in the old cell
and stop measuring when the first IP frame is sent/received in the new cell by the MS.
In case the new cell belongs to a new LA, the MS enters dedicated mode and sends a
Suspend message to the BSS. The BSS sends the Suspend message to the SGSN. The
SGSN sends a Suspend Ack to the BSS, which will discard all LLC frames stored for
the MS. After the MS has performed the LA update it will send a RA update in the new
cell to resume the transfer, see Figure 14. This will be done in case that the BSS does
not support the resume procedure. (Origin: [5]).
2. Suspend
6. Routeing Area Update Request
1. Dedicated Mode
MS BSS SGSN MSC/VLR
3. Suspend
4. Resume
5. Channel Release
3. Suspend Ack
4. Resume Ack
Figure 14: Suspend/Resume Procedure
Remarks:
The continuation of the DL transmission after the RAU depends on the number of
buffered frames in the SGSN:
• If there is still data buffered in the SGSN the transfer will continue immediately
after the RAU complete.
• Definitions of KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
• If no more data is stored in the SGSN the continuation of the transfer depends on the
application. � For an FTP DL transfer a timeout will occur on TCP layer and the
transmission continues afterwards.
In live networks parameters like the moving speed, the cell configuration and the MS
have big influence on the time needed for the Intra SGSN RAU.
Prerequisites:
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI
ACC 030). Two different cells belonging to different RAs served by the same SGSN
configured with GPRS/EDGE capability.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
Intra SGS) RA Update during DL FTP, TAG: MT 020
This KPI can be measured by executing the UDP based test solution: Transport Tests, IP
layer outage as described in detail in Proposed Extensions to GPRS e2e performance
Testing [2].
Passive Measurement:
There is no means of passive measurement (counter or Moniq tool) for this KPI.
• Definition of GPRS/EDGE End-to-End KPIs • • Definitions of KPIs •
5.4.11 Inter SGSN RAU Time
TAG: INT 100
Definition:
The KPI Inter SGSN RAU Time [s] denotes the length of the time period an Inter SGSN
RAU takes.
Note: The duration of an Inter SGSN RAU can be measured best during an ongoing
FTP session.
In most of the case the RA belongs to a new LA, so that the MS performs a
LAU as well.
Trigger Point(s):
Start measuring the time interval when the last IP frame is sent/received in the old cell
and stop measuring when the first IP frame is sent/received in the new cell by the MS.
Remarks:
In live networks parameters like the moving speed, the cell configuration and the MS
have big influence on the time needed for the Intra SGSN RAU.
Prerequisites:
The MS is already GPRS attached (PI ACC 020) and a PDP Context is activated (PI
ACC 030). Two different cells belonging to different RAs served by the different
SGSNs configured with GPRS/EDGE capability.
Active Measurement:
This KPI can be measured by executing the following test case described in detail in the
Generic Test Description for GPRS/EDGE End-to-End Performance Testing [1]:
Inter SGS) RA Update during DL FTP, TAG: MT 030
Passive Measurement:
There is no means of passive measurement (counter or Moniq tool) for this KPI.
• Overview of the KPIs • • Definition of GPRS/EDGE End-to-End KPIs •
6 Overview of the KPIs When measuring KPIs related in a live network file and e-mail sizes as well as timeout
values cannot be influenced, so that passive tests of KPIs will measure any file and e-
mail size that appears in a live network as well as system internal timeout values.
But when measuring the KPIs with active tests the following timeout values should be
chosen:
Figure 15: Overview of the KPIs
ACC 010
GRPS/UMTS Unavailability
KPIs:
Accessibility
Retainability
Integrity
max. 30s max. 30s max. 30s max. 30s
time out time outtime out time out
time out time out
service dependant service dependant
Data (content) transfer
INT 010
Attach Setup Time
INT 020
PDP Context Activation Time
INT 030
Service Access Time
ACC 020
Attach Failure Ratio
ACC 030
PDP Context Activation
Failure Ratio
ACC 040
Service Access Failure
Ratio
RET 010
PDP Context Cut-off
Ratio
RET 020
Data Transfer Cut-off
Ratio
INT 050
Mean User Data Rate
PDP Context
Deactivation
Wait for (max.) 30s, if PDP
Context in not deactivated
� reset phoneDetach
Wait for (max.) 30s, if
MS is not detached � reset phone
Start trigger: Attach request
Stop trigger: Attach Accept
Start trigger: PDP Context Activation request
Stop trigger: PDP Context Activation Accept
INT 070
Packet Loss Ratio
INT 060
Round Trip Time
ACC 010
GRPS/UMTS Unavailability
KPIs:
Accessibility
Retainability
Integrity
max. 30s max. 30s max. 30s max. 30s
time out time outtime out time out
time out time out
service dependant service dependant
Data (content) transfer
INT 010
Attach Setup Time
INT 020
PDP Context Activation Time
INT 030
Service Access Time
ACC 020
Attach Failure Ratio
ACC 030
PDP Context Activation
Failure Ratio
ACC 040
Service Access Failure
Ratio
RET 010
PDP Context Cut-off
Ratio
RET 020
Data Transfer Cut-off
Ratio
INT 050
Mean User Data Rate
PDP Context
Deactivation
Wait for (max.) 30s, if PDP
Context in not deactivated
� reset phoneDetach
Wait for (max.) 30s, if
MS is not detached � reset phone
Start trigger: Attach request
Stop trigger: Attach Accept
Start trigger: PDP Context Activation request
Stop trigger: PDP Context Activation Accept
INT 070
Packet Loss Ratio
INT 060
Round Trip Time
• Definition of GPRS/EDGE End-to-End KPIs • • Summary •
7 Summary In this chapter all KPIs are listed together with their tag and an indication whether the
can be measured actively or passively.
It is very important to use this chapter only for quick reference but not as ultimate input
when looking for a specific KPI and its definition. Detailed information on each KPI is
provided in chapters 5.2, 5.3, and 5.4.
Figure 16: Overview: KPIs and Test Measurements
KPI
Cat e
gor y
Ti tl e
Tag
Passi v
e M
easur e
ment
Act i
ve M
easur e
ment
Accessibility KPIs: GPRS/UMTS Unavailability ACC 010 - -
Attach Failure Ratio ACC 020 counter GTD: ST 010
PDP Context Activation Failure Ratio ACC 030 counter GTD: ST 020
Secondary PDP Context Activation Failure Ratio ACC 035 - GTD: ST 020
Service Acces Failure Ratio ACC 040 Moniq tool
GTD: SET 010, SET 020, SET 030,
SET 040, SET 050
GPRS Accessibility Failure Ratio ACC 050 Moniq tool -
Retainability KPIs: PDP Context Cut-off Ratio RET 010 - -
Data Transfer Cut-off Ratio RET 020 Moniq tool
GTD: TT 020, SET 020, SET 030,
SET 040, SET 050
Cell Update Failure Ratio RET 030 - GTD: MT 010
Intra SGSN RAU Failure Ratio RET 040 counter GTD: MT 020
Inter SGSN RAU Failure Ratio RET 050 counter GTD: MT 030
PS Paging Failure Ratio RET 060 counter -
Integrity KPIs: Attach Setup Time INT 010 - GTD: ST 010
PDP Context Activation Delay INT 020 - GTD: ST 020
Service Access Time INT 030 Moniq tool
GTD: SET 010, SET 020, SET 030,
SET 040, SET 050
End-to-End Access Time INT 040 - -
Mean User Data Rate INT 050 Moniq tool
GTD: TT 020, TT 030, MT 010, MT
020, MT 030, SET 020, SET 040,
SET 050, Extension GTD: RAW
throughput
Round Trip Time INT 060 Moniq tool GTD: TT 010
Packet Loss Ratio INT 070 Moniq tool GTD: TT 010, TT 020 & SET 030
Cell Update Time INT 080 -
GTD: MT 010, Extension GTD: IP
layer outage
Intra SGSN RAU Time INT 090 -
GTD: MT 020, Extension GTD: IP
layer outage
Inter SGSN RAU Time INT 100 - GTD: MT 030
• Summary • • Definition of GPRS/EDGE End-to-End KPIs •
KPI
Cat e
gor y
Ti t
l e
Tag
Passi v
e M
easur e
ment
Accessibility KPIs: Service Acces Failure Ratio ACC 040 Moniq tool
WAP Connection Establishment Success Ratio ACC WAPconnect Moniq tool
POP3, Mail Server Connection Success Ratio ACC POP3connect Moniq tool
Domain Name Service Query Success Ratio ACC DNS Moniq tool
Retainability KPIs: Data Transfer Cut-off Ratio RET 020 Moniq tool
WAP Object Download Success Ratio RET WAPobject Moniq tool
Web Object Download Success Ratio RET WEB Moniq tool
FTP Download Success Ratio RET FTP Moniq tool
POP3, E-mail Download Success Ratio RET POP3download Moniq tool
SMTP, E-mail Send Success Ratio RET SMTPdownload Moniq tool
Integrity KPIs: Service Access Time INT 030 Moniq tool
WAP Connection Set-up Delay INT WAPconnect Moniq tool
Web Server Response Time INT WEBresponse Moniq tool
POP3, Mail Server Connection Time INT POP3connect Moniq tool
WAP Object Downlaod Delay INT WAPdownload Moniq tool
Web Small Object Download Time INT Websmall Moniq tool
POP3, E-mail Download Time INT POP3download Moniq tool
SMTP, E-mail Sending Time INT SMTPsend Moniq tool
Domain Name Service Response Delay INT DNSdelay Moniq tool
Mean User Data Rate INT 050 Moniq tool
FTP Download Rate INT FTPrate Moniq tool
Web Large Object Download Rate INT WEBlarge Moniq tool
End-to-end Achievable Throughput INT THRHPUT Moniq tool
TCP Goodput/Throughput Ratio INT THRHPUT/GPUT Moniq tool
Round Trip Time INT 060 Moniq tool
End-to-end Round Trip Delay INT DELAYe2e Moniq tool
Client-side Round Trip Delay INT DELAYgprs Moniq tool
Server-side Delay INT DELAYinternet Moniq tool
Packet Loss Ratio INT 070 Moniq tool
WAP Retransmission Ratio INT WAPretransmission Moniq tool
Downlink End-to-End Packet Loss INT LOSSe2e Moniq tool
Server to Monitor Loss INT LOSSinternet Moniq tool
Monitor to Client Loss INT LOSSgprs Moniq tool
Figure 17: GPRS KPIs Based on Passive Measurements with Moniq
• Definition of GPRS/EDGE End-to-End KPIs • • Terminology •
8 Terminology
8.1 Abbreviations
BCCH Broadcast Control Channel
DL Downlink
e2e End-to-End
FTP File Transfer Protocol
GMM GPRS Mobility Management
ICMP Internet Control Message Protocol
KPI Key Performance Indicator
LA Location Area
LAU Location Area Update
LLC Logical Link Control
PBCCH Packet Broadcast Control Channel
PDCH Packet Data Channel
PDP Packet Data Protocol
PI Performance Indicator
P-TMSI Packet Temporary Mobile Subscription Identity
RA Routing Area
RAU Routing Area Update
RR Radio Resource
RTT Round Trip Time
UL Uplink
WAP Wireless Application Protocol
• Terminology • • Definition of GPRS/EDGE End-to-End KPIs •
8.2 Glossary
This chapter provides definitions of technical and other terms used in the document.
Accessibility
The ability of the user to obtain a service within specified tolerances and other given
conditions.
The accessibility of the service does include a chain of operations the user must be able
to execute:
• the user must be able to access GPRS/EDGE as such, meaning he must be able to do
a GPRS attach and a PDP context activation, and
• he must be able to access an IP service after he has an activated PDP context.
Integrity
Integrity indicates the degree to which a service is provided without major interferences,
once it was accessed.
Integrity KPIs show the performance of successful service attempts. Even if a service
was accessed successfully, the user’s perception of the performance may vary between
very good and unacceptably bad.
Retainability
Retainability indicates the ability of the user to keep a service – once it was accessed -
under given conditions for a requested period of time.
The retainability of a service or session does include that the user does not have to
perform any additional manual operations that would not be necessary under stable
network conditions, like for example manual re-activation of the PDP context.
Performance Indicator
Performance Indicators are those indicators that have impact on the end-user’s
perception of the service performance.
Key Performance Indicator
Key Performance Indicators are those performance indicators that are rate to be primary
performance Indicators reflecting the end-user’s perception of the service performance
in an outstanding manner.
Goodput & Throughput
Goodput is the user data payload in TCP connections (Web, e-mail, FTP, etc.), in
contrast to the total IP layer data transmitted to the mobile terminal. Throughput
includes the protocol overhead (TCP/IP) and the retransmissions while the goodput is
the raw data the end-user wants to receive or transmit.
• Definition of GPRS/EDGE End-to-End KPIs • • References •
9 References [1] Generic Test Description for GPRS/EDGE End-to-End Performance Testing in
a Live Network, EED/S/G-01:135 Uen, Revision C.
[2] Proposed Extensions to GPRS e2e performance Testing, (ETH).
[3] Ericsson GPRS End-to-End Performance Guidebook, EED/S/G-01:134 Uen
[4] ITU-T Recommendation E.800: Terms and Definitions Related to Quality of
Service and Network Performance Including Dependability (08/94)
[5] GPRS Service Description Stage 2 (Release 1999), 3GPP TS 23.060 V3.7.0
[6] Measurements of GPRS End-to-End KPIs with Counters, EED/S/G-02:044
[7] Mobile radio interface layer 3 specification, 3GPP TS 24.008
• Appendix • • Definition of GPRS/EDGE End-to-End KPIs •
10 Appendix
10.1 Statistical Guidelines
When analyzing loss and success ratio KPIs the proportional frequency of a specific
event is estimated from measurement data. Calculating the uncertainty of a measurement
the correct sample size can be derived.
Formula (1) represents the Gaussian approximation of the binomial distribution
considering a 95% confidence interval:
n
ff )1(**96.1
−
(1)
with:
f = measured proportional frequency of the KPI one is interested in.
n = length of the sample
p = unknown proportional frequency of the KPI one is interested in
Thus, the corresponding proportional frequency p falls in the confidence interval (CI)
with 95% probability:
−+
−−
n
fff
n
fff
)1(**96.1,
)1(**96.1 (2)
Example:
Assumption: Measured f = 20% for the GPRS Attach Ratio (KPI xx) with a sample size
of 200 measurements. Using formula (1) shows that the uncertainty of the measurement
is 5.54%. Therefore: with a 95%-probability the proportional frequency p is included in
the interval [14.46: 25.54]