Nokia Blueprint E2E Testing in Real Cloud Customer Environment Conditions

Nikos Vrakas

Test Lead, Nokia Core Engineering Solution Testing

1. The Solution2. Testing

Areas

3. Automation

The Telco Could: Network Function Virtualization

• Dedicated HW replaced by VMs

• Automated deployments and LCM

• Virtualized Network Functions (VNFs)

• Flexibility and better resource utilization

• Dynamic Scaling

• Faster time to market

The chassis-based implementation has been virtualized by turning each line card in the

system into a virtual machine

Simplified ETSI NFV MANO reference architecture

ComputingHardware

StorageHardware

NetworkHardware

Hardware resources

Virtualization LayerVirtualized

InfrastructureManager(s)

VNFManager(s)

NFV Orchestrator

NFVI

EMS 1

Service, VNF and Infrastructure Description

Virtual Computing

Virtual Storage Virtual Network

VNF 1

EMS 2

VNF 2

EMS 3

VNF 3

E2E vs E2E

RealElement

RealElement

RealElement

RealElement

RealElement

RealElement

SUT: Gateway

Phone Phone

E2E vs E2EHTTPS/JSON

A-BGWATGW

E-CSCFEATF

I-/S-CSCFBGCF

TASSCC-AS

SM-IP-GWMFRC

LIMS

Charging system

AAA

PCRF

WCDMA

GSM

LTE

DirectWAP

BSC

RNC

SMSC

S2b (GTP)

S6a

SCPSWx

P-CSCFATCF

MME

MSS

ePDG

SGW

PGW

UntrustedWAP

HSS-FE

HLR-FE

SDM

MRFP

MRF

DNSENUM

IM-MGW

MGCF

PSTN

PGW

S1-MME

S1-U

Gm

S11

S5

SWm s6a

Cx/Dx

Sh

MAPMAP

Hd

BSF/NAF

Ut/XCAP

Ut/XCAP

Sv

Rx

Gx

Mw

ISC, Ma

Iq

Mr’

MpMb

SGi

Mb

Mb

Nb

Nc

Swu

AuC

Mw

MNP DB

MAP

Mg

IxIMS ntw

I-BCF

TrGW

NetAct/COM/SAM

Traffica

Op

era

bility

iOS ES

Mx

Registers

Voice Core

EPC

SPR

Sp

I2

Mm

RCS, UMC

MSAN

AGCF

AGW

POTS/ISDN

IP PBXFixed Access

E2E vs E2E

The Cloud Core Solution

Scalability

SBC

1. The solution2. Testing

Areas

3. Automation

E2E - Voice over X Testing

VNF Functional validation

• Calls CS-VoLTE-VoWiFi

• Emergency calls

• Lawful Interception

• Handovers

Cloud Operability / Life Cycle Management

• VNFM based LCM (20+ E2E core VNFs)

• Scaling, Evacuation/Migration

4G/5G

2G/3GWi-Fi

Test LabE2E - Voice over X Testing

3G

4G

3G

4G

Performance Testing

StabilityStudy Solution’s stability with at least such traffic intensity that it is normally designed to be handled and that all the VNFs are stable when loaded for defined duration (24h/48h).

OverloadMass traffic is applied outside the required capacity until an overload situation is achieved.

ProvocativeThe solution is tested under special conditions, performing provocative actions with mass traffic ongoing.

Stability ProvocativeOverload

Tools: Simulators/Call GeneratorsPerformance Testing

500k concurrent calls

CloudCore Solution

48h stability

1M mobile phones!

Tools: Simulators/Call GeneratorsPerformance Testing

CallSimulator

CallSimulator

CallSimulator

CallSimulator

CallSimulator

500k concurrent calls

CloudCore Solution

48h stability

Simulate: UEs, Radio, WiFi Access Points

Security/Robustness Testing

Tools: DDoS OrchestratorSecurity Testing

• Internally designed and developed

• SDN datacenters can handle enormous amount of traffic

• Initiating attacks from simple VMs cannot have impact on such deployments

• Orchestrates many servers for various DDoS attacks

• Supports SDN by using the Nuage VRS-B plug-in

ReflectionSIP

TCP SYN

Flooding

Controller

Engine

DD

oS

Orc

he

stra

tor

Clo

ud

Co

re

Tools: Simulating Extreme ConditionsRobustness Testing

Deviations

CloudCore Solution

CallSimulator

80 64 e1 40

00 40 06 44X

X

X X

1

overwrite

bursts

GBps

Sec

fragmentation

Line breaks

X

1. The Solution2. Testing

Areas

3. Automation

Robot Framework

• Developed at Nokia Networks, now sponsored by Robot Framework Foundation

• Keyword driven, keywords created by python experts

• No programming skills are needed from testers

• Testers can create new higher-level keywords by composing old ones

• Clear and detailed reports in html/xml format

Test StructureRobot Framework

• Setup, Test Steps, Teardown

• Restores system configuration

• Supports tags

• Easy, clear and human readable test steps

Suite Setup

Test Setup

Test Steps

Test Teardown

Test Setup

Test Steps

Test Teardown

Suite Teardown

Test 02

Test 01

Robot Framework

‘Start ipsec tunnels’ :

• Connect to a certain network element

• Execute cli command for tunnel bring-up

• Check cli response

In case of error:

• Execution breaks

• Teardown runs for recovery

Security_IPSec_Tunnel_3DES

[Documentation]

... Establishes IPsec tunnel - 3DES-CBC - MD5

[Setup] Configure_IPsec_Tunnels_3DES

[Tags] security ipsec regression

start ipsec tunnels

show ipsec ikev1 tunnels

start traffic on simulator simulator_A5 basic_ip_trafic.conf

robot wait for seconds 10

stop traffic on simulator simulator_A5

[Teardown] Delete_Configure_IPsec_Tunnels_3DES Collect_Logs

Test Automation Framework

• For Business and stakeholders

• Pass/Fail indication

• Measurements categorized by Test Case and Feature Name

• Compares historical data

• Difficult to read by Business and Product owners

• Beautiful and very detailed graphs

• Helpful for extensive analysis of KPIs and measurement results

robot

• python 2.7• robot

framework Call KPIs & resource Graphs

Generic PipelineTest Automation Framework

CallSim.

The WAF (Wireless Automation Framework)E2E - Voice over X Testing Automation

• It is a Python Library by Apple

• Used to control R&D software equipped iPhone mobiles from a Mac

• Control functions are triggered from Robot Framework

Radio

WAF setupE2E - Voice over X Testing Automation

Attenuators

RF Boxes

WAF Macs

EthernetUSB/lightning

Coaxial

WAF PipelineE2E - Voice over X Testing Automation

Robot Framework:

Robot Framework is a generic test automation framework for acceptance testing. It is a keyword-driven testing framework that uses tabular test data syntax. All our E2E Regression DevOps tests are coded in Robot.

Jenkins:

Jenkins is a server-based system that helps to automate the non-human part of software development process, with continuous integration and facilitating technical aspects of continuous delivery.

WAF Tool:

WAF tool is developed by Apple, and it can be used only with Apple devices which run RnD software.

It is a Python library that runs only on Mac after installing from a simple installer..

WAF tool can be used to control R&D SW equipped iPhone mobiles, so all iPhones should be connected to a Mac and an RPC service should be installed and started on it to call these function from Robot Framework.

E2E Solution Environment:

An overview of the environment we use in E2E Solution Testing.

Jenkins connects to GITLAB server, where all cases are stored. They are written in Robot.

python 2.7

DDoS PipelineSecurity Testing Automation

Security Testers

commit

pullExecute Robot

Robot Dev

commit

Run Tools

CallSim.

DDoSOrch.

PublishKPI Measurements

Analyze the impact

“Run, then automate” vs “Automate before even run”

Thank you!