Pawel Kuzak

VoIP Backend Development

HIGH QUALITY TELEPHONY

USING A FAIL-SAFE MEDIA

RELAY SETUP

1&1 - Member of United Internet AG

VoIP Network Overview

Quality of Service

Media relay - RTPengine

Extended VoIP Architecture

Hardware and Network configuration

Challenges

Redundancy

RTPengine setup

Redis Keyspace Notifications

Roundup

1&1 Group2

Agenda

31.05.2016

1&1 – Member of United Internet AG

31.05.2016 1&1 Group

Consumer

Business

Consumer

Business

Partners

Listed investments

25.10% 28.57% 29.02% 25.10% 30.36%

Access

Applications

3

20.11% 10.46% 8.37%* 8.31%

* Spin-off of Hi-Media S.A.

Motivated team

Around 8,200 employees, thereof approx.

2,700 in product management, development

and data centers

Sales strength

Approx. 3.2 million new customer contracts

p.a.

50,000 registrations for free services on a

daily basis

Operational excellence

49 million accounts in 11 countries

7 data centers

70,000 servers in Europe and USA

Powerful network infrastructure

41,000 km of fiber network

1&1: Internet services of United Internet AG

4 1&1 Group

Access Applications

Networks

User

equipment

Content

Standard

software

431.05.2016

Locations

5 1&1 Group31.05.2016

VoIP Backend – Some numbers

6 1&1 Group 631.05.2016

> 100 servers forming a geo-redundant active-active setup

~ 4 Mio. VoIP customers

~ 12 Mio. registered numbers

> 500 Calls/Second (Peak)

> 120.000 concurrent callls (Peak)

> 15.000 REGISTER requests per Second (Peak)

99,999% availability

SIP interconnections with

Versatel

QSC

Telefonica

Vodafone

7 1&1 Group 731.05.2016

Network Overview – until 2015

Carrier-X IP Core

Access

Network

Access

Network

IP-Peering

Interconnect

1&1 IP Core

IP-Peering

Interconnect

MSAN

DSLAM

MSAN

DSLAM

BRAS

BNG

BRAS

BNG

Datacenter Karlsruhe

Datacenter Baden-Airpark

SIP

RTP

8 1&1 Group 831.05.2016

Network Overview – until 2015

Carrier-X IP Core

Access

Network

Access

Network

IP-Peering

Interconnect

1&1 IP Core

IP-Peering

Interconnect

MSAN

DSLAM

MSAN

DSLAM

BRAS

BNG

BRAS

BNG

Datacenter Karlsruhe

Datacenter Baden-Airpark

SIP

RTP

Quality of

Service

?

Quality of Service – ADSL

9 1&1 Group

Different ATM-PVCs for Voice and Data

One PPPoE session per ATM-PVC

Different VLANs for Voice and Data in Backbone

Upstream (F!B Internet):

F!B just seperates Data traffic from Voice traffic

Based on the ATM-PVC, the DSLAM decides whether to use the prioritized VLAN

to BRAS

BRAS uses DSCP (sets TOS Bit)

Downstream (Internet F!B):

Based on the target IP, the BRAS knows which VLAN to use

Traffic may be routed best-effort to BRAS

31.05.2016

Quality of Service – VDSL

10 1&1 Group

No distinction between Voice and Data

One VLAN for Voice and Data in Backbone

31.05.2016

Quality of Service – VDSL

31.05.201611 1&1 Group

Carrier won‘t prioritize IP traffic unless it doesn‘t come from a pre-defined IP range

Prioritization of end-to-end RTP traffic not possible IPs spread

Need to bundle RTP traffic

Need media relay

Requirements for media relay:

Open source

Kamailio integration

High performance

High availability

Scalability

TOS – Bit setting

RTPengine (https://github.com/sipwise/rtpengine)

31.05.201612 1&1 Group

Network Overview – Present

Carrier-X IP Core

Access

Network

Access

Network

IP-Peering

Interconnect

IP-Peering

Interconnect

MSAN

DSLAM

MSAN

DSLAM

BRAS

BNG

BRAS

BNG

Datacenter Karlsruhe

Datacenter Baden-Airpark

Datacenter Frankfurt – Interxion (DE-CIX)

Datacenter Frankfurt – Equinix (DE-CIX)

SIP

RTP

QoS impact – VDSL: Speech delay VDSL-ISDN

13 1&1 Group31.05.2016

Activate QoS

RTP end-to-end routing

RTP over media

relay

Media relay – Some numbers

14 1&1 Group 1431.05.2016

Currently 6 server in 2 data centers

Dell R430, Intel X520 Network 10 Gbit/s, 2x 10 Core Xeon E5 w/ HT

Running RTPengine controlled via Kamailio‘s rtpengine module

Full redundant active-active setup

Each server handling 3.000 concurrent sessions (Peak)

~ 300.000 packets/s incoming and outgoing

Each server able to handle 18.000 concurrent sessions (Peak)

Worst case backup scenario

~ 18.000.000 packets/s incoming and outgoing

Successfully tested 25.000 concurrent sessions

Limiting factors:

Bandwidth Extend hardware

Available sockets Use more IPs

Interrupts Tune configuration

31.05.201615 1&1 Group

Media relay – Interrupt handling configuration

APIC

RAM

Network

ControllerCPU

Packet

DMA

MSI-X

Intel® Flow director / N-tuple filter / Hash filter:

Map RTP/RTCP to same queue

IRQ coalescing:

„Bundle“ interrupts

Configured IRQ-CPU core affinity:

Map IRQ/queues to same CPU cores Optimize utilization of CPU caches

Improve performance

Session mapped 1:1 to CPU core

31.05.201616 1&1 Group

DE-CIX Media Relay Setup

Carrier-X IP Core

Redis Master

Database

Core Router 1&1 -

Equinix

10 Gbit/s

Dell R430

Intel X520 Network 10 Gbit/s

2x 10 Core Xeon E5

40 Gbit/s

BGP Session

Core Router 1&1 -

Interxion

Redis Slave

Database

1x 10 Gbit/s

1x 1 Gbit/s

iDRAC Management-

Console

RTP

BGP

Management

Kamailio

Redis

31.05.201617 1&1 Group

RTPengine Redundancy

Carrier-X IP Core

Redis Master

Database

Core Router 1&1 -

Equinix

10 Gbit/s

Dell R430

Intel X520 Network 10 Gbit/s

2x 10 Core Xeon E5

40 Gbit/s

BGP Session

Core Router 1&1 -

Interxion

Redis Slave

Database

Challenge:

Do NOT lose sessions

when a media relay

crashes

31.05.201618 1&1 Group

RTPengine Redundancy – Simplified illustration

Kamailio

RTPengine2:

- Listening on Redis-

Notifications for

Keyspace 1 on Redis-

Slave

- Writing to Redis Master

Keyspace 2

Redis Master

Database

IP_1 IP_2 IP_1 IP_2

Core Router

RTPengine1:

- Listening on Redis-

Notifications for

Keyspace 2 on Redis-

Slave

- Writing to Redis Master

Keyspace 1

BGP

primary route

BGP secondary

routeRTP RTP

BGP

primary route

BGP secondary

route

31.05.201619 1&1 Group

RTPengine Redundancy – Simplified illustration

Kamailio

RTPengine2:

- Listening on Redis-

Notifications for

Keyspace 1 on Redis-

Slave

- Writing to Redis Master

Keyspace 2

Redis Master

Database

IP_1 IP_2 IP_1 IP_2

Core Router

1. Kamailio creates RTP session on

RTPengine1 with active IP_1

RTPengine1:

- Listening on Redis-

Notifications for

Keyspace 2 on Redis-

Slave

- Writing to Redis Master

Keyspace 1

RTP RTP

1.

BGP

primary route

BGP secondary

route

BGP

primary route

BGP secondary

route

31.05.201620 1&1 Group

RTPengine Redundancy – Simplified illustration

Kamailio

RTPengine2:

- Listening on Redis-

Notifications for

Keyspace 1 on Redis-

Slave

- Writing to Redis Master

Keyspace 2

Redis Master

Database

IP_1 IP_2 IP_1 IP_2

Core Router

1. Kamailio creates RTP session on

RTPengine1 with active IP_1

2. RTP traffic arrives. RTPengine1

persists session information to

Redis Master, Keyspace 1

RTPengine1:

- Listening on Redis-

Notifications for

Keyspace 2 on Redis-

Slave

- Writing to Redis Master

Keyspace 1

RTP RTP

2.

BGP

primary route

BGP secondary

route

BGP

primary route

BGP secondary

route

31.05.201621 1&1 Group

RTPengine Redundancy – Simplified illustration

Kamailio

RTPengine2:

- Listening on Redis-

Notifications for

Keyspace 1 on Redis-

Slave

- Writing to Redis Master

Keyspace 2

Redis Master

Database

IP_1 IP_2 IP_1 IP_2

Core Router

1. Kamailio creates RTP session on

RTPengine1 with active IP_1

2. RTP traffic arrives. RTPengine1

persists session information to

Redis Master, Keyspace 1

3. Redis Master replicates

information to slaves

RTPengine1:

- Listening on Redis-

Notifications for

Keyspace 2 on Redis-

Slave

- Writing to Redis Master

Keyspace 1

RTP RTP

3.3.

BGP

primary route

BGP secondary

route

BGP

primary route

BGP secondary

route

31.05.201622 1&1 Group

RTPengine Redundancy – Simplified illustration

Kamailio

RTPengine2:

- Listening on Redis-

Notifications for

Keyspace 1 on Redis-

Slave

- Writing to Redis Master

Keyspace 2

Redis Master

Database

IP_1 IP_2 IP_1 IP_2

Core Router

1. Kamailio creates RTP session on

RTPengine1 with active IP_1

2. RTP traffic arrives. RTPengine1

persists session information to

Redis Master, Keyspace 1

3. Redis Master replicats information to

slaves

4. RTPengine2 (listening on

Keyspace 1) handles Redis-

Notification and creates session

on IP_1

RTPengine1:

- Listening on Redis-

Notifications for

Keyspace 2 on Redis-

Slave

- Writing to Redis Master

Keyspace 1

4. Notify

RTP RTP

BGP

primary route

BGP secondary

route

BGP

primary route

BGP secondary

route

31.05.201623 1&1 Group

RTPengine Redundancy – Simplified illustration

Kamailio

RTPengine2:

- Listening on Redis-

Notifications for

Keyspace 1 on Redis-

Slave

- Writing to Redis Master

Keyspace 2

Redis Master

Database

IP_1 IP_2 IP_1 IP_2

Core Router

1. Kamailio creates RTP session on

RTPengine1 with active IP_1

2. RTP traffic arrives. RTPengine1

persists session information to

Redis Master, Keyspace 1

3. Redis Master replicats information to

slaves

4. RTPengine2 (listening on Keyspace

1) handles Redis-Notification and

creates session on IP_1

5. HW Error / Network Error / …

RTPengine1:

- Listening on Redis-

Notifications for

Keyspace 2 on Redis-

Slave

- Writing to Redis Master

Keyspace 1

RTP RTP

BGP

primary route

BGP secondary

route

BGP

primary route

BGP secondary

route

31.05.201624 1&1 Group

RTPengine Redundancy – Simplified illustration

Kamailio

RTPengine2:

- Listening on Redis-

Notifications for

Keyspace 1 on Redis-

Slave

- Writing to Redis Master

Keyspace 2

Redis Master

Database

IP_1 IP_2 IP_1 IP_2

Core Router

1. Kamailio creates RTP session on

RTPengine1 with active IP_1

2. RTP traffic arrives. RTPengine1

persists session information to

Redis Master, Keyspace 1

3. Redis Master replicats information to

slaves

4. RTPengine2 (listening on Keyspace

1) handles Redis-Notification and

creates session on IP_1

5. HW Error / Network Error / …

6. Core Router: Withdraw BGP

announcements for RTPengine1

(< 1s delay) RTPengine1:

- Listening on Redis-

Notifications for

Keyspace 2 on Redis-

Slave

- Writing to Redis Master

Keyspace 1

RTP

BGP

primary route

BGP secondary

route

31.05.201625 1&1 Group

RTPengine Redundancy – Simplified illustration

Kamailio

RTPengine2:

- Listening on Redis-

Notifications for

Keyspace 1 on Redis-

Slave

- Writing to Redis Master

Keyspace 2

Redis Master

Database

IP_1 IP_2 IP_1 IP_2

Core Router

1. Kamailio creates RTP session on

RTPengine1 with active IP_1

2. RTP traffic arrives. RTPengine1

persists session information to

Redis Master, Keyspace 1

3. Redis Master replicats information to

slaves

4. RTPengine2 (listening on Keyspace

1) handles Redis-Notification and

creates session on IP_1

5. HW Error / Network Error / …

6. Core Router: Withdraw BGP

announcements for RTPengine1

(< 1s delay)

7. Traffic routed via alternative routeRTPengine1:

- Listening on Redis-

Notifications for

Keyspace 2 on Redis-

Slave

- Writing to Redis Master

Keyspace 1

RTP

BGP

primary route

BGP secondary

route

Roundup

26 1&1 Group 2631.05.2016

Additional VoIP server in 2 new data centers

Redundant active-active setup

RTPengine software running on media relay servers

Invoking RTPengine via Kamailio‘s rtpengine module

Tuning IRQ handling

Synchronizing RTPengine session information via Redis

and keyspace notifications

Using BGP mechanisms for redundancy

31.05.201627 1&1 Group

Interested?

jobs.1und1.de