e2e kpi monitoring
DESCRIPTION
End to End KPI monitoringTRANSCRIPT
Determining how to carry out accurate monitoring of E2E data performance
Mobile Network Performance
Management
London June 2014
Martin Harris Orange Corporate Services Ltd
2
Contents
Why measure data service performance and what is the benefit?
• examining strategies to benchmark and monitor end to end data performance against competitors or other countries
• ensuring performance is in line with promises
Monitoring end-to-end service performance
• understanding the benefits and trade-offs between the use of network KPIs, probes, robots, drive tests, network conters
How do we assess performance?
• looking at how and where to set performance targets, what impacts the choice, and how these evolve for LTE
• examining the customer experience by using speed tests (crowd sourced data) which can give a specific view of QoS
How can “Big Data” help us?
• or is this just another complexity?
3
Why measure service performance?
To verify performance is in line with promises given to our customers
• business offers and customer expectations
To understand how our networks (and competitor networks) are performing
• best network and meeting customer expectations
• identify weaknesses (bottlenecks, slow speed, packet loss, capacity, interconnect, outages)
• ensure we meet SLAs with business partners
• optimise expenditure, and radio spectrum use
To provide a good quality of experience
• keeping customers, preventing churning
• capturing new customers by good performance
• ensuring services perform as required
To bridge the gap between network performance and quality of experience
• ensure that good network performance means good QoE for customers
4
What is service performance?
Traditional data KPI measurements
• network KPIs: drop session rate, session setup success rate, congested cell ratio, cell loading
• E2E KPIs (robots): data rates, web page, latency
• used to ensure good network performance
• insufficient to reflect quality of experience
Services run on top of the networks
• monitor the real service performance
• what the customers are doing
• web browsing, YouTube, video, messaging,gaming
• where they are doing it
• indoors, at home, in the office, on the tube
• challenges to measure in these locations
• while respecting privacy and confidentiality
E2E performance depends on the other end and third part connectivity
• often outside the operator’s network
• expectation of network operator responsibility
5
Performance has to be measurable
Network counters, passive probes
• large volume of data from the complete network
• high level overview or focus on specific areas
• modern probes enable us to drill down to:
• specific service streams
• individuals or identified user groups
• device types
What changes for LTE?
• basic principles remain the same
• upgrade of tools, robots, probes
• extended to cover eNodeB, EPC, IMS etc
How to assess network and service performance?
• against targets or objectives
• against customer expectations?
• against local competitors?
• against other operators in the same group?
6
Traditional data KPI measurements
Variable performance
Coming under control here
Define congested cells to meet your own requirements
Further investment may be needed to reduce congested
cells
Track performance by region to identify
any poorly performing areas
Monitor KPIs or a regular basis, look at
trends, and keep them under control
Ensure good network performance = good quality of experience
7
Traditional data rate measurements
Data rates compared by operator, country, region, vendor
Data rate evolution over two years
Measure where the customer is located • business areas, transport hubs
• hotels, residential, indoors, etc
• major challenge
Ensure consistent measurement method • drive tests, robots, stationary or mobility
• understand the effect of TCP slow start
Ensure measurement tools up to date • capable of maximum data rates
• ensure all tools are LTE capable
Average data rate at 85% of instantaneous
data rate
TIP: larger file sizes are generally used for measuring LTE data rates, but these can take excessive time if the terminal connects to 3G or 2G network; to avoid this measure the
data rate over a fixed time, e.g. 10-20 seconds and stop the transfer after this time
8
Crowd sourcing – an alternative solution
What is crowd sourcing?
• a fresh way to look at network performance from the customer’s perspective
Collected from apps/agents on handsets
• different locations, users, handsets
• tens of thousands of measurements
Very consistent data month on month
• but do customers only perform these tests when they have a “network problem” ?
Currently limited to traditional KPIs
• data rate and latency
More advanced applications available
• passive (silent) or active (on-click) monitoring
• web page and video performance
• “friendly users”
• benchmarking - alternative to drive tests
• capability to test indoors
9
How can we assess service performance?
What can we measure? • data rate and latency
• video services: access success rate, speed to start playout, image quality (pauses, lost frames)
• web services: download success rate / time; DNS response
• other services: instant messaging, specific “OTT” services
• over a mix of locations
Assessment of performance • do we assess against targets?
• how do we establish targets?
• against aspirations, customer expectations, local competitors, other operators?
• should targets be different for LTE?
• can we give performance a “number”?
• e.g. 25% = “half target speed” 50% = “on target” 100% = “twice as good as target”
Example of weighted service performance
Service Major service
weighing Individual services
Overall service weighting
Web page download performance
40%
Reference web page (Kepler) 25% 10%
Local web pages 37.5% 15%
International web pages 37.5% 15%
Download data rate 20%
Indoors 40% 8%
Large cities (drive test) 30% 6%
Small cities (drive test) 20% 4%
Interconnecting routes 10% 2%
Upload data rate 10%
Indoors 40% 8%
Large cities (drive test) 30% 6%
Small cities (drive test) 20% 4%
Interconnecting routes 10% 2%
Video performance 20% 20% 20%
DNS access time 5% 5% 5%
Latency (Round Trip Time) 5% 5% 5%
10
Service performance versus customer experience?
We must not forget to measure actual customer experience
• regular surveys of randomly selected customers
• performed by third party, unbiased, covering all operators
• telling us what the customer feels about our network performance
Results of recent (4Q-2013) customer experience surveys for five European operators show a high level of correlation against weighted service performance However there is often a “lag” in customer perception, with customers “remembering” bad experiences
Can we use this to better effect to improve customer experience? Can “Big Data” help us to ?
11
How can “Big Data” help us?
“Big data” - a consolidated tool for customer experience management
• making better use of all network performance indicators and correlating these with other information, e.g. from customer services, billing
• opportunities:
• to better understand the customer experience
• to improve network performance
• to provide customers with new offers
• benefits:
• customer retention and base expansion
• new value propositions
• customer service optimization
• QoS/QoE management and network optimisation
Mobile usage
Probes CRM data Probes
IS sources (customer data, service description, network description)
Service description Network description
Mediation layer
Data processing and storage
Portal and presentation layer
Big Data
Qo
S
KPIs
Qo
E tools
probes
analy
sis
storage internet
social networks
processing
data
cap
ture
database processes
use cases
LTE
IMS
3G
consolidation
latency
data rates
services
12
“Big Data” use cases
Already proven benefits for both network operations and marketing
• multiple use cases; each country can focus on those important to the business
• examples:
• 15 complaints in one area highlighted that 600 customers had problems, corrective action was put in place to improve performance
• devices with high signalling volume, again able to take corrective action
13
“Big Data” network optimization - traffic
Traffic by application
• highest traffic volume is web, followed by streaming, file download, mail, P2P
Traffic by service provider
• highest single traffic provider is YouTube, with Facebook, Apple, Orange portal, Skype also providing high levels of traffic
14
“Big Data” network optimization – traffic by app & user
Top 10 users per day
• 7% of traffic from 10 users
• can also identify usage by application, e.g. web, download, streaming
• used to understand cause of high load and better manage the traffic
• allows marketing to target high users with upgrades
15
“Big Data”– traffic by terminal type
Traffic volume by terminal brand
• Traffic figures show almost half the traffic is on Samsung or Apple devices. Further drill down could be performed if necessary to show the precise terminal types.
Traffic by terminal type (smartphone, USB,mobile, tablet)
• Figures show 78% of mobile data usage is now on smartphone, with less than 10% on USB modems.
• Note the low volume of mobile data traffic on tablets as these probably use Wi-Fi by preference.
16
“Big Data” QoS/QoE management – IP sessions
Video performance (RTSP)
• packet desynchronisations
• lost packets
IP session QoS
• time for session establishment (between first user request and first downlink packet)
• where the time for session establishment is long, there is a need to identify any single cause
17
“Big Data” QoS/QoE management - throughput
Throughput distribution
• note that the low data rate sessions can be caused by a the number of low volume transfers
Percentage of retransmitted packets
• monitor performance, review trends, identify source of retransmissions
18
Summary
Traditional measurements of QoS obtained from network performance
• still applicable today, but not the whole story
LTE
• increased data and service usage
Services not networks
• emphasis on service measurement
• greater need to focus on customer
New tools available to the operator
• better understanding of the customer experience of our data services
Crowd sourcing
• measuring the service performance where the customer is located
Big data
• making better use of the data available
• providing a high level overview or drilling down to target specific issues
