e2e kpi monitoring

Download E2E KPI Monitoring

Post on 06-Feb-2016




3 download

Embed Size (px)


End to End KPI monitoring


  • Determining how to carry out accurate monitoring of E2E data performance

    Mobile Network Performance


    London June 2014

    Martin Harris Orange Corporate Services Ltd


  • 2


    Why measure data service performance and what is the benefit?

    examining strategies to benchmark and monitor end to end data performance against competitors or other countries

    ensuring performance is in line with promises

    Monitoring end-to-end service performance

    understanding the benefits and trade-offs between the use of network KPIs, probes, robots, drive tests, network conters

    How do we assess performance?

    looking at how and where to set performance targets, what impacts the choice, and how these evolve for LTE

    examining the customer experience by using speed tests (crowd sourced data) which can give a specific view of QoS

    How can Big Data help us?

    or is this just another complexity?

  • 3

    Why measure service performance?

    To verify performance is in line with promises given to our customers

    business offers and customer expectations

    To understand how our networks (and competitor networks) are performing

    best network and meeting customer expectations

    identify weaknesses (bottlenecks, slow speed, packet loss, capacity, interconnect, outages)

    ensure we meet SLAs with business partners

    optimise expenditure, and radio spectrum use

    To provide a good quality of experience

    keeping customers, preventing churning

    capturing new customers by good performance

    ensuring services perform as required

    To bridge the gap between network performance and quality of experience

    ensure that good network performance means good QoE for customers

  • 4

    What is service performance?

    Traditional data KPI measurements

    network KPIs: drop session rate, session setup success rate, congested cell ratio, cell loading

    E2E KPIs (robots): data rates, web page, latency

    used to ensure good network performance

    insufficient to reflect quality of experience

    Services run on top of the networks

    monitor the real service performance

    what the customers are doing

    web browsing, YouTube, video, messaging,gaming

    where they are doing it

    indoors, at home, in the office, on the tube

    challenges to measure in these locations

    while respecting privacy and confidentiality

    E2E performance depends on the other end and third part connectivity

    often outside the operators network

    expectation of network operator responsibility

  • 5

    Performance has to be measurable

    Network counters, passive probes

    large volume of data from the complete network

    high level overview or focus on specific areas

    modern probes enable us to drill down to:

    specific service streams

    individuals or identified user groups

    device types

    What changes for LTE?

    basic principles remain the same

    upgrade of tools, robots, probes

    extended to cover eNodeB, EPC, IMS etc

    How to assess network and service performance?

    against targets or objectives

    against customer expectations?

    against local competitors?

    against other operators in the same group?

  • 6

    Traditional data KPI measurements

    Variable performance

    Coming under control here

    Define congested cells to meet your own requirements

    Further investment may be needed to reduce congested


    Track performance by region to identify

    any poorly performing areas

    Monitor KPIs or a regular basis, look at

    trends, and keep them under control

    Ensure good network performance = good quality of experience

  • 7

    Traditional data rate measurements

    Data rates compared by operator, country, region, vendor

    Data rate evolution over two years

    Measure where the customer is located business areas, transport hubs

    hotels, residential, indoors, etc

    major challenge

    Ensure consistent measurement method drive tests, robots, stationary or mobility

    understand the effect of TCP slow start

    Ensure measurement tools up to date capable of maximum data rates

    ensure all tools are LTE capable

    Average data rate at 85% of instantaneous

    data rate

    TIP: larger file sizes are generally used for measuring LTE data rates, but these can take excessive time if the terminal connects to 3G or 2G network; to avoid this measure the

    data rate over a fixed time, e.g. 10-20 seconds and stop the transfer after this time

  • 8

    Crowd sourcing an alternative solution

    What is crowd sourcing?

    a fresh way to look at network performance from the customers perspective

    Collected from apps/agents on handsets

    different locations, users, handsets

    tens of thousands of measurements

    Very consistent data month on month

    but do customers only perform these tests when they have a network problem ?

    Currently limited to traditional KPIs

    data rate and latency

    More advanced applications available

    passive (silent) or active (on-click) monitoring

    web page and video performance

    friendly users

    benchmarking - alternative to drive tests

    capability to test indoors

  • 9

    How can we assess service performance?

    What can we measure? data rate and latency

    video services: access success rate, speed to start playout, image quality (pauses, lost frames)

    web services: download success rate / time; DNS response

    other services: instant messaging, specific OTT services

    over a mix of locations

    Assessment of performance do we assess against targets?

    how do we establish targets?

    against aspirations, customer expectations, local competitors, other operators?

    should targets be different for LTE?

    can we give performance a number?

    e.g. 25% = half target speed 50% = on target 100% = twice as good as target

    Example of weighted service performance

    Service Major service

    weighing Individual services

    Overall service weighting

    Web page download performance


    Reference web page (Kepler) 25% 10%

    Local web pages 37.5% 15%

    International web pages 37.5% 15%

    Download data rate 20%

    Indoors 40% 8%

    Large cities (drive test) 30% 6%

    Small cities (drive test) 20% 4%

    Interconnecting routes 10% 2%

    Upload data rate 10%

    Indoors 40% 8%

    Large cities (drive test) 30% 6%

    Small cities (drive test) 20% 4%

    Interconnecting routes 10% 2%

    Video performance 20% 20% 20%

    DNS access time 5% 5% 5%

    Latency (Round Trip Time) 5% 5% 5%

  • 10

    Service performance versus customer experience?

    We must not forget to measure actual customer experience

    regular surveys of randomly selected customers

    performed by third party, unbiased, covering all operators

    telling us what the customer feels about our network performance

    Results of recent (4Q-2013) customer experience surveys for five European operators show a high level of correlation against weighted service performance However there is often a lag in customer perception, with customers remembering bad experiences

    Can we use this to better effect to improve customer experience? Can Big Data help us to ?

  • 11

    How can Big Data help us?

    Big data - a consolidated tool for customer experience management

    making better use of all network performance indicators and correlating these with other information, e.g. from customer services, billing


    to better understand the customer experience

    to improve network performance

    to provide customers with new offers


    customer retention and base expansion

    new value propositions

    customer service optimization

    QoS/QoE management and network optimisation

    Mobile usage

    Probes CRM data Probes

    IS sources (customer data, service description, network description)

    Service description Network description

    Mediation layer

    Data processing and storage

    Portal and presentation layer

    Big Data





    E tools




    storage internet

    social networks





    database processes

    use cases






    data rates


  • 12

    Big Data use cases

    Already proven benefits for both network operations and marketing

    multiple use cases; each country can focus on those important to the business


    15 complaints in one area highlighted that 600 customers had problems, corrective action was put in place to improve performance

    devices with high signalling volume, again able to take corrective action

  • 13

    Big Data network optimization - traffic

    Traffic by application

    highest traffic volume is web, followed by streaming, file download, mail, P2P

    Traffic by service provider

    highest single traffic provider is YouTube, with Facebook, Apple, Orange portal, Skype also providing high levels of traffic

  • 14

View more >