deliverable d4.3 deployment and interoperability … final.pdf5g communication with a heterogeneous,...

5G Communication with a Heterogeneous, Agile Mobile network in the PyeongChang Winter Olympic competitioN

Grant agreement n. 723247

Deliverable D4.3 Deployment and interoperability report

on Distributed Mobile Core testbed between Korea and Oulu

Date of Delivery: 31 May 2018 (Contractual) 31 May 2018 (Actual)

Editor: Muhammad Arif

Associate Editors: Wouter Tavernier

Authors: Muhammad Arif

Dissemination Level: PU

Security: Public

Status: Final

Version: V2.0

File Name: 5GCHAMPION_D4.3_Final.pdf

Work Package: WP4

Title: D4.3 - Deployment and interoperability report on distributed

mobile core testbed between Korea and Oulu Date: 31 Mai 2018 Status: Final

Security: Public Version: V2.0

The information contained in this document is the property of the contractors. It cannot be reproduced or transmitted to thirds without the authorization of the contractors.

2 / 94

Abstract

This deliverable documents the outcomes of T4.1 in WP4. The focus of this task has been on the design, implementation, deployment and integration of the two mobile core network testbeds and EU and KR side. As reported in earlier deliverables, both testbeds have been interconnected via a both the public internet, as well as via a dedicated interconnection. In this deliverable, we report on the interoperability testing and related PoCs that have been performed focusing on SDN/NFV-based mobile core networks for 5G. Two PoCs have been demonstrated during the Olympics: i) the 5G mobile core networks for the Olympics, and ii) advanced 5G mobile core networks supporting multi-RAT and distributed mobility management. For both PoCs, the key components have been detailed, as well as their testbed infrastructure, and last but not least functional and performance tests have been performed validating the implementations. In parallel, the SDN/NFV-based 5G mobile testbed on the EU has further evolved into a multi-slice cloud core 5GTN testbed, consisting of multiple vEPC implementations, each with their own features. OpenBaton was used as MANO platform for deploying OpenEPC, and has been evaluated with respect to package management, service instantiation and multi-site deployment. The closing part of the deliverable focuses on the interoperability of SDN/NFV-based 5G mobile core networks, as made available by EU and KR partners. Two types of interoperability have been designed and successfully evaluated: service interoperability and management interoperability. The first has been demonstrated during the Olympics venue in Gangneung focusing on the interoperability between EU and KR PoPs focusing on real-time streaming and interactive augmented reality services. Management interoperability between two SDN/NFV-based vEPCs has been successfully validated in two manners: i) ETSI SOL003-interface based interoperability between the NFVO and VNFMs of the vEPCs, and ii) NFVO-VIM-based interoperability. The first was successfully demonstrated during the Olympics between KR and EU EPCs, while the second was demonstrated on the OpenBaton MANO framework using OpenEPC and the OpenAirInterface EPC.

Index terms

Distributed Mobile Core Network





3 / 94

Contents

1 Introduction ........................................................................................ 4

2 KR - 5G Mobile Core Deployment ...................................................... 5

2.1 5G mobile core networks for Olympic-related PoC 5

2.2 PoC for advanced 5G mobile core networks 33

3 EU - 5G Mobile Core Deployment .................................................... 59

3.1 5G Test Network at a glance 59

3.2 Cloud core resources within 5GTN in a multi slice environment 60

3.3 OpenBaton in a nutshell 61

3.4 NFV based MANO deployment in 5GTN infrastructure 63

3.5 Multi-Site NFVI and VIM deployment 69

3.6 Evaluation 72

4 5G mobile core Interoperability test ................................................ 85

4.1 Overview (KR+EU) 85

4.2 Service Interoperability (KR+EU) 85

4.3 Management Interoperability 86

5 Conclusion ......................................................................................... 93

6 References ......................................................................................... 94





4 / 94

1 Introduction WP4 is responsible for the design and implementation of necessary components in the mobile core network targeting minimal end-to-end latency by adequately virtualizing functionality and applying SDN principles. This involves research and implementation of required Management and Orchestration (MANO) functionality supporting the control of virtualized EPC components, as well as the implementation as the implementation and deployment of EPC-focused virtualized network functions (VNFs). The mobile core network design and testbed builds further on the architecture as described in WP2, and are demonstrated in the Proof of Concepts of WP6. The work conducted in this WP is structured into three tasks. Below we report on the work performed in each of these tasks. The outcomes of tasks T4.2 and T4.3 have been documented in D4.2. In this deliverable we bring these pieces together as needed for the PoCs during the Olympics, and we provide in-depth evaluations of all of the developed components. Functional evaluations have been done on both KR and EU MANO platforms. This includes package management, on-demand instantiation, scaling and termination of vEPC services. The instantiated vEPCs themselves have been evaluated as well with respect to their functionality and their performance. In support of the advanced 5G PoC, several testbeds have been deployed to deploy and evaluate multi-Radio Access Technologies, Software-Defined Infrastructure, and SDN/NFV-enabled distributed mobility management on the KR side. This has been demonstrated in the corresponding PoCs. On the EU side, the SDN/NFV-based 5G mobile testbed on the EU has been extended to multi-slice cloud core 5GTN testbed, consisting of multiple vEPC implementations including the Nokia Airframe vEPC, OpenAirInterface vEPC, and multiple OpenEPC deployments. OpenBaton was used as MANO platform for deploying OpenEPC, and has been evaluated with respect to package management, service instantiation and multi-site deployment. Interoperability has been one of the focuses of this WP, and here deliverable we report on service- as well as MANO interoperability designs and experiments that have been performed, resulting into successful Olympics PoCs. Interoperability on the service level involved real-time streaming and interactive augmented reality services, while MANO-based interoperability was validated for two interfaces: i) the NFVO-VNFM interface as defined by ETSI SOL003, and ii) the NFVO-VIM interface as implemented by OpenBaton. Both variants have been evaluated and generated positive results. The document is structured as follows. Section 1 gives an overview on the two major KR PoCs: the 5G Mobile Core Deployment and the PoC for advanced 5G mobile core networks, and provides an in-depth analysis of the performed evaluations. Section 2 documents the EU testbed, the OpenBaton framework, its SDN/NFV MANO processes supporting OpenEPC and OpenAirInterface vEPC deployment, and the performed evaluations. Interoperability between SDN/NFV-based mobile core networks is reported in Section 3. Finally, Section 4 provides some concluding remarks.





5 / 94

2 KR - 5G Mobile Core Deployment

2.1 5G mobile core networks for Olympic-related PoC

2.1.1 Key components

Core networks refer to telecommunication network’s core part, which offers numerous services to the customers who are interconnected by the access networks. In terms of mobile network connectivity, the mobile packet core is what makes a wireless access network a truly mobile network by performing functions fundamental to the delivery of mobile data services such as internet and carrier’s own services. Korea’s 5G mobile packet core consists of two main components, mobile packet core functions (HSvEPC) and their management system (DCNM).

Figure 1. Key components of 5G mobile core networks

2.1.1.1 Highly Scalable vEPC

In Korea’s 5G mobile packet core dubbed HSvEPC (Highly Scalable vEPC), to provide highly scalable 5G mobile core networks, we employed two types of scalability: functional scalability and service scalability. Functional scalability means the capability of expansion of the vEPC by separating conventional consolidated functions into user plane and control plane functions by dynamic scaling operations over virtualized network functions, while service scalability is about diversification of core networks for end users classified by applications, policy and other context information using network slicing technology. The SDP (Software Define Protocol) is used for communications between control plane functions and user plane functions. Packet forwarding policies generated by signalling operations can be delivered from control plane functions via SDP and vice versa for the feedback.

The HSvEPC consists of several virtual network functions including vMME, vSGW-CU, vSGW-DU, vPGW-CU, and vPGW-DU.





6 / 94

Functions Description

vMME Manage mobility of UE and process authentication.

vSGW-CU vSGW-CU creates a session with vMME and GTP-C, then send it to vSGW-DU with using SDP.

vSGW-DU vSGW-DU connects with vSGW-CU to get session information, then communicate UE traffic between eNB and PGW-DU with GTP-U.

vPGW-CU vPGW-CU creates a session with vSGW-CU and GTP-C, send related session information to vPGW-DU with using SDP, and apply QoS from PCRF.

vPGW-DU vPGW-DU gets session information from vPGW-CU, communicates UE traffic between SGW-DU by GTP-U, and communicates traffic between PDN by SGi the interface.

2.1.1.2 Distributed Cloud NFV Management System

Characterized by its agility and flexibility, 5G networks are required to adapt to the service to be deployed on demand. The DCNM (Distributed Cloud NFV Management System) mediates service and network infrastructure connected with business and service support systems. DCNM plays a role of integrated orchestration for distributed Micro DC (Data Center) by conducting multi-domain, multi-technology resource optimization. It manages not only network services deployable over the multiple micro DCs but also resources and functions in the individual micro DCs.

The Orchestrator is responsible for managing the functions such as network service life-cycle management and the overall resource management. Service management or orchestration deals with the creation and end-to-end management of the services — made possible by composing different virtual network functions (VNFs). The Function Manager oversees the lifecycle (typically involves provisioning, scaling, terminating) management of instances of virtual network function (VNF). It is typically assumed that each VNF will be associated with a Function Manager that will manage that particular VNF’s lifecycle. The Infra(structure) Manager controls and manages the NFVI compute, storage, and network resources.

2.1.2 Deployments for Olympics

In conjunction with the 2018 PyeongChang Winter Olympics, the HSvEPC service was distributed in Daejeon and Gangneung using the DCNM system described above. There may be many ways to disperse. There is a distributed-DCNM method that installs an orchestrator, a function manager, and an Infra Manager in one package in Daejeon and Gangneung, respectively.

Another way to deploy is to place an orchestrator in one place and distribute the Function Manager and Infra Manager by region which is called single-orchestrator with distributed-manager distribution method. Since the separate protocol is needed for a single package in the distributed-DCNM, and the interworking between these orchestrators is still under





7 / 94

standardization, we adopt the second method, single-orchestrator with distributed-manager, for the Olympics PoC.

There are also two ways of deploying HSvEPC. There is an integrated package method that integrates the entire functions of HSvEPC into one system and installs full set respectively. In another way, there is a distributed package method that distributes functions of HSvEPC and installs specific functions in the area interacting with each other. At this Olympics PoC, we installed integrated packages in each region and concentrated on mobile core functions in different regions to keep traffic localized.

Figure 2. Deployment of 5G mobile core networks

2.1.3 Evaluation

2.1.3.1 HSvEPC Package management

Test

Description

Test VNF Package management for HSvEPC(vMME, vSGW-CU, vPGW-CU,

vSGW-DU, vPGW-DU)

Precondition

s

1. VNF Package Description:

The VNF of the Eulon consists of a VNF package that can install the VNF and

a vEPC software. The VNF package includes VNFD, SW package (software

package) and metadata (a configuration file to match the environment) that

can be used when the OS image, VNF package, and software packages are





8 / 94

installed. The VNFD includes information that the VNF can do onboarding. The

VNF agent for interlocking with VNFM is projected in the SW package and will

work when installing the VNF. The VNF package includes a software that

functions as a VNF which will be downloaded from the VNF repository when

the VNF package is installed.

VNF Package

VNF

Componen

ts

VNFD MME, SGW-CU, PGW-CU, SGW-DU, PGW-DU

os image Centos 7.1

VNF agent Agent for operation over VNF-VNFM interface

SW Package VNF(HSvEPC) Package

Metadata Configuration

file

Configuration file for environment for up and

running of HSvEPC process

2. For performance in Data plan, SGW-DU and PGW-DU use path-through

function. SRIOV technology is used for performance, DPDK driver is used for

data transmission Using DPDK and SRIOV, virtual port (virtual port) in

physical ports to prevent the transmission of packets from slowing down

even when SGW or PGW is installed at virtual machine.





9 / 94

< VNF architecture including Agents>

<VNFs interworking with SR-IOV>

3. VNF agents for

- interworking with Function Manager

- Configuration management and initialization of SW PKG

- Periodic report of VNF resource status

4. Be in place of rpm files to initialize VNF from the repository for

the VNF application software





10 / 94

5. In the VNF package, agent should be compiled including OS

image(Centos 7.1) for deployment VNF package according to the

VNF descriptor which is acknowledged in advance.

Test

Procedure

1. Access to Orchestrator

2. Examine onboarding information of the VNF package which is

registered in Orchestrator including MME, PGW-CU, PGW-DU, SGW-

CU and SGW-DU.

Test

Results

Screenshot of the VNF package information (VNFD, manifest, metadata,

image) for MME, PGW-CU, PGW-DU, SGW-CU and SGW-DU.

VNFs registered to Orchestrator

VNF Descriptors for HSvEPC





11 / 94





12 / 94





13 / 94





14 / 94





15 / 94





16 / 94

2.1.3.2 HSvEPC Operation

Test

Descriptio

n

Test to confirm the behavior of EPC service using Femtocell.

Preconditi

ons

This test connects Femtocell to HSvEPC and attaches terminal (galaxys3lte)

directly to Femtocell via HSvEPC to the outside (PDN Proving that each VNFC

is configured according to the function and specification of HSvEPC

Functionalities of HSvEPC의 기능 :

HSvEPC composed of vMME, vSGW-CU, vSGW-DU, vPGW-CU and vPGW-

DU

vME performs the mobile management and authentication of UE

vSGW-CU generates sessions through vME, vPGW and GTP-C, and

distributes related session information to vSGW-DU using SDP





17 / 94

vSGW-DU obtains session information in conjunction with vSGW-CU, and

transmits UE Trafic through eNB, PGW-DU and GTP-U

vPGW-CU generates sessions through vSGW-CU and GTP-C, distributes

related session information to vPGW-DU using SDP, and QoS provided by

PCRF

vPGW-DU obtains session information in conjunction with vPGW-CU, and

transmits UE Traffic through SGW-DU and GTP-U

Test

Procedure

Run EPC services using Femtocell.

Turn on the power on the terminal to femtocell attach.

Provides signaling messages and status information that are transmitted

between EPC components through the pcap file while the EPC service is

operating.

Turn off the terminal to check the detach in the femtocell

Test

Results

EPC architecture interworking with UE through Femtocell

UE attach / detach procedure





18 / 94

IP allocation

1. Initial attach request from UE





19 / 94

2. Delete Session

3. Authentication Information Request/Answer

4. Session ID from Authentication Information Request(AIR)

5. IMSI value from Authentication Information Request(AIR)

6. Authentication vector information from Authentication Information

Answer(AIA)

7. Authentication Information Request and Answer from MME to HSS(AIR,

AIA)

8. Network information from UE

9. Authentication Information Response / NAS security setup





20 / 94

10. Update Location Request / Answer(ULR, ULA)

11. Credit Control Request / Answer(CCR, CCA)

12. Attach accept / EPS bearer establishment

13. Create Session Request from MME -> SGW-CU -> PGW-CU

14. Create Session Response from PGW-CU -> SGW-CU -> MME

15. UE IP(192.168.10.115) assigned from PGW-CU(10.0.0.103)





21 / 94

16. Connection between SGW-CU and SGW-DU with port 10005

17. Connection between PGW-CU and PGW-DU with port 10005





22 / 94

18. Using internet with UE after attached to femto-vEPC

2.1.3.3 HSvEPC Performance

Test

Descriptio

n

Test to confirm the performance of EPC service using eNB emulator.

Preconditi

ons

1. Goal : Interworking and Verification of Radio Emulator for 5G Virtual Mobile

Core Function and Performance Verification

2. To measure the performance of vEPC with the enb emulator between UE

and HSvEPC.

3. Enb Emulator to receive packets from EPC client to create a GTP tunnel and





23 / 94

transmit it to vSGW.

Test

Procedure

1. Execute EPC service using eNB emulator.

2. Measure band performance using iPerf (iPerfserver, installed client)

3. Measure performance indicators (e.g., call processing performance) other

than bands (measureing by increasing the number of terminals to five)

4. Enb Emulator used to measure performance (enbsim uses 1000mbps max,

10 terminals, 100mbps per terminal,

Test

Results

Performance Test List

- Measure the max value with 1 terminal (to be processed up to 1000

mbps in the enb emulator)

- 5 terminals uplink, downlink measurement (100mbps per terminal, total

500mbps processing target)

Performance test with 1 terminal





24 / 94

uplink for 5 terminals

downlink for 5 terminals

Packet Transmission form vSGW-DU

Mbits/sec

Mean(1st) Value Mean(2nd) Value Mean(3rd) Value Mean(4th) Value Mean(5th) Value Mean(6th) Value Mean(7th) Value Mean(8th) Value Mean(9th) Value Mean(10th) Value

866.7 788 884.4 763 890 797 886.7 792 890.7 791 879.4 798 880.4 797 886.4 783 882.4 798 888.3 796

922 869 899 932 931 904 884 910 896 912

675 896 897 902 932 886 878 893 902 899

898 909 881 901 905 891 881 890 873 918

894 908 886 887 881 883 902 900 894 885

901 895 907 884 891 881 899 906 882 897

896 908 899 894 921 871 890 896 896 903

912 909 919 898 883 885 883 892 911 893

892 909 901 887 886 906 896 899 884 890

889 878 914 890 886 889 894 895 888 890

Total Average 883.5

1 terminal





25 / 94

Packet Transmission form vPGW-DU





26 / 94

2.1.3.4 HSvEPC Management (Instantiation)

Test

Description HSvEPC instantiation using MANO (DCNM)

Preconditions

This test is related to IOP based on ETSI NFV SOL3 in NFV MANO

environment

Test Procedure

1. Register VNFD, a component of NSD. Enter VNF name, Image File,

YAML script, etc., and click on the "Save" button to save it

2. Enter the information needed to generate VNFR on the VNFD Detailed

Information View screen. You can enter that information at Extension

Information of VDU Tab





27 / 94

3. Register Network Service Descriptor

4. On the View screen for NSD Detailed Information, enter the

information needed to generate NSD. VLD, CP and VNFD at the Basic

Information, Extension Information, and VNFD Information Tab





28 / 94

5. Click the "NS Instantiate" button in the NSR list and select NSR name,

NSD and MicroDataCenter to generate VNFR at the pop-up screen

6. Check the output of the operation





29 / 94

7. Check the topology of the NSR generated

Test Results - Confirm the VNFs generated from the instantiation procedures

- Confirm the configuration results for the VNFs generated

2.1.3.5 HSvEPC Management (Scaling)

Test

Description HSvEPC Scaling through MANO(DCNM)

Preconditions - The test is related to IOP based on ETSI NFV SOL3 in NFV MANO

environment





30 / 94

- Scale-out if CPU usage rate 95% or more

- Scale-out of VNFR (eNB, sGW) is performed only once and then again

scale-in is a condition that can be done again

- For pGW VNFR, it is not scale-out even if CPU load occurs

- port chain is produced only when two VNFCs, eNB and sGW, are

scale-out

Test Procedure

1. If the CPU load is 95% or more, the NFVO internal monitoring

function will cause a fault event, which will be notified to the VNFM

so that the VNFR will be scale-out.

2. Check whether VNFR is created automatically

3. Check the VNF instances scaled out





31 / 94

4. Check the port chaining resulted in scale-out operation

Test Results

- Confirm the VNFC configuration

- Confirm the VNFC scaled out

- Confirm the port chaining after scale-out operation

2.1.3.6 HSvEPC Management (Termination)

Test

Description HSvEPC termination through MANO (DCNM)

Preconditions

- The test is related to IOP based on ETSI NFV SOL3 in NFV MANO

environment

- VNF instances for HSvEPC is running

Test Procedure 1. When click the "NS Terminate" button on the NSR list and click "OK"

on the confirm window, asked to delete vEPC by VNFM,





32 / 94

2. Check the VNF instances to be deleted at the VNFR information

3. Check the port chaining information deleted

4. Check the NSR list to be deleted

Test Results

- Confirm stop of the VNF instances

- Confirm the deleted VNF instances

- Confirm the deletion of the Port chain for the NSR from the list

- Confirm the NSR deletion from the list





33 / 94

2.2 PoC for advanced 5G mobile core networks

2.2.1 Key technologies

With emerging of Edge-Cloud paradigm, 5G SDI (Software-Defined Infrastructure) anticipates consisting of three layers: Centralized Core-Clouds, Multi-Site Edge-Clouds, and Heterogeneous End-Things. Edge-Cloud becomes a key of 5G SDI since it reduces latency and bandwidth waste due to end-to-core communications. Currently, researches on 5G SDI focus on Edge-Cloud due to its importance and possibility.

To cope with the paradigm shift, we have researched on building 5G Multi-RAT Edge-Clouds Testbed with a goal to support researches on Multi-Site Edge-Clouds as well as Software-Defined Multi-Access. We have designed and constructed the testbed in two different approaches, K-ONE Playground and Openwincon Multi-RAT testbed. However, those efforts are totally separated and lack to support multi-access networking.

Based on this motivation, we introduce Multi-Access Box that inter-connects End-Things with Edge-Cloud via various software-defined access networking. With Multi-Access Box, we combine two testbed efforts into one which provides Multi-Access as well as Multi-Site infrastructure.

In this document, we show the experimental prototype of Multi-Access Box as the first step. For the experimental prototype, we choose and prototype four access networking: SDN-enabled Wired Interface, OVS-integrated Interface. BATMAN-based WiFi Interface and OpenAir-based 4G Interface. In addition, we show the experimental prototype of Multi-Access Box and its Multi-Access Control Software with those prototyped access networking. At last, we verify multi-access networking using the integrated Multi-Access Box.

Also, we show a prototype of Distributed Mobility Management (DMM) testbed for supporting mobility management between Edge-Clouds. Firstly, we design DMM architecture suited in K-ONE playground multi-site edge clouds. Then we describe in detail of our DMM prototype composed SDN-based control plane and data plane functions. These DMM functions can be running as VMs in the K-Cluster boxes. We also introduce our implementation and handover scenario between edge clouds in detail. Finally, our experimental results will be given.

2.2.2 Testbeds

2.2.2.1 Multi- RAT access networking

In this section, we describe design Multi-Access Box and four access networking interfaces that we select for prototyping of Multi-Access Box. In addition, we show implementation of preliminary version of the interfaces.

Multi-Access Box Prototype Design





34 / 94

Figure 3. Multi-Access Box: Multiple Interface Options

Multi-Access Box is a mini Server-Switch Box for testing and verifying different access networking at Proof-of-Concept (PoC) level, so it is not designed to support production-scale workloads. That is, it provides a testing box to develop automating access networking configuration, improving visibility, improving networking reliability, and ensuring performance.

According to the definition, we design a prototype of Multi-Access Box as show in Figure 4. Even though Multi-Access Box originally targets for supporting lots of different access networking, but a prototype of Multi-Access box has four access networking technologies as the first step: SDN-enabled Wired Interface, OVS-integrated WiFi Interface, BATMAN-based mesh WiFi Interface, and OpenAir-based 4G Interface. Left part in the Figure 4 shows multiple interfaces we selected. Each access networking interface requires not only interface hardware device, but also control software of it. In this document, we define “Interface” as a pair of interface device and control software. Even though Multi-Access Box originally targets for supporting lots of different access networking, but a prototype of Multi-Access box has four access networking technologies as the first step: SDN-enabled Wired Interface, OVS-integrated WiFi Interface, BATMAN-based mesh WiFi Interface, and OpenAir-based 4G Interface.

Firstly, SDN-enabled Wired Interface means a wired access network providing reliable networking controlled by SDN controller. In this network, there are multiple paths with SDN-enabled hardware switches between End-Things and Edge-Clouds. SDN controller steers dynamically networking flows to ensure reliability and performance of networking. OVS-integrated WiFi Interface provides WiFi Access Point (AP) function controlled by SDN controller.





35 / 94

We design the WiFi AP function to work on commodity Linux box, and use SDN controller to control flows and WiFi features. Next, BATMAN-based mesh WiFi Interface assumes a network infrastructure where mesh network is required since the Internet is not available. We design a WiFi interface that uses BATMAN, a routing protocol for mesh networks that only memorizes the best link information between nodes. The last but not least, OpenAir-based 4G Wireless Interface is open-source based 4G wireless interface. The interface adopts a hardware / software platform called OpenAir Interface.

And Multi-Access Box integrates the four interfaces to provide multi-access networking. The integration means hardware integration to equip all interface devices on a single box, and software integration to develop Multi-Access Networking Coordinator controlling the devices by using separated control software.

Multi-Access Box: SDN-enabled Wired Interface Prototype Implementation

Figure 4. SDN-enabled Wired Interface Implementation

When data collected from End-Things needs to be guaranteed to be transmitted to Edge-Clouds or Core-Clouds, 5G Access Networking technology is required to ensure stable networking. Based on the requirements of the above use case, we design SDN-enabled Wired Interface which is access networking that guarantees networking stability and performance between IoT Things and Edge-Clouds using SDN controller. In SDN-enabled Wired Interface Prototype, multi-path between End-Things and Edge-Cloud is secured and SDN controller is





36 / 94

used to ensure networking stability and performance through dynamic flow steering in case of networking failure. This prototype takes into account line disruptions, switch failures, and excessive traffic on the path due to networking failure. The goal is to implement a function that automatically identifies the fault by collecting / analyzing information of all switches with the SDN controller, and calculates the optimal bypass path to automatically set the flow.

In order to physically acquire the multi-paths required in the above scenario, a multi-path topology that interconnects multiple SDN-enabled switches between end-things and multi-access boxes is designed. Each switch is connected to two different switches, and both end switches are connected to Multi-Access Box, End-Things. This makes it possible to construct a multipath topology in which one or more paths are acquired between two end switches. (More detailed description) For the dynamic control of the multi-path thus configured, a configuration for the SDN controller is required. For this purpose, SDN controller is operated on the box corresponding to Multi-Access Box. At this time, open source software ONOS SDN controller is used as SDN controller. In addition, for the flow setup through the SDN controller, all switches on the multi-path must support the OpenFlow protocol and must be connected to the SDN controller on the Multi-Access Box basically. For this reasons we use MikroTik’s products, which can support OpenFlow protocol.

A software function is needed to steer the flow so that when a networking failure occurs in the configuration environment, it detects it in real time and bypasses the path. In the SDN-enabled Wired Interface, a steering function providing the corresponding function is designed in the form of an application of the SDN controller. The Steering Function was created to run the IoT service on the Multi-Access Edge Cloud testbed. For the IoT service, the steering function reads the configuration file of the service developer and parses it to check the network resource. Steering function provides the function to collect network information (Switch, Host, Link, Flow information) in real time using REST API of SDN controller to determine fault and to set switch by calculating new path in case of trouble. Therefore, SDN-enabled Wired Interface can quickly and automatically recover networking faults that can occur on Access Networking. The experimental results using the steering function are shown in the lower part of the above figure.





37 / 94

Multi-Access Box: OVS-integrated WiFi Interface Prototype Implementation

Figure 5. OVS-integrated WiFi implementation

WiFi network is one of the most common wireless access networking technologies, with the networking technology mentioned with 4G LTE in Multi-RAT. Therefore, to demonstrate 5G Multi-Access Networking, a Multi-Access Box supporting WiFi access networking is required. Therefore, we extend the Multi-Access Box to design the OVS-integrated WiFi Interface, which can be controlled by a single SDN controller while providing WiFi Access Point (AP) function. The OVS-integrated WiFi Interface should be able to build on the Linux-based Commodity Hardware rather than using special purpose hardware, according to the basic design of Multi-Access Box. In addition, all interfaces on the Multi-Access Box must be controllable via a single networking control software, so hardware / software must be designed to be controllable via the SDN controller.

The OVS integrated WiFi interface must be able to operate as an AP so that IoT Things can be accessed via WiFi. Also, IoT Things connected to WiFi should be provided with IP address dynamically via DHCP and SDN-enabled in order to get help from SDN Controller. In order to satisfy these requirements, we configured the internal network as shown in the figure above. In order to operate as an AP, we used hostapd daemon which can set up AP according to user's specification. We used dnsmasq as the DHCP server, connected the WiFi interface to OVS (Open vSwitch) and connected the bridge to the SDN controller. Through this configuration, IoT Things connected to WiFi can be controlled by the SDN Controller respectively, and by using Multi-Access Box Control Software, it is possible to select data reception function for WiFi Access IoT Things and to select wired and WiFi interface Verification is possible.

Verification of the interface was done by connecting two laptops to the Multi-Access Box. we connected two laptop computers to the WiFi in the Multi-Access Box and then did ping test. On ONOS Web GUI, the two host is looked with an IP address, and we confirmed that it can





38 / 94

communication each other. Also, to check the control by SDN, we connected a wired host to the Multi-Access Box and performed a ping test after installing a point-to-point intent for between wired host and a WiFi connected laptop. In addition, the steering function will be reinforced using the OVS-integrated interface. For IoT devices that use both wire and WiFi interfaces, use WiFi to transmit data when the wired connection is lost. When the steering function detects the change of the topology, it installs a new Intent on the WiFi interface. This allows IoT End-Things to continuously send data to the Multi-Access Box.

.

2.2.2.2 SDI based Distributed Cloud Cluster

In this section, we show a preliminary prototype of Multi-Access Box that integrates access networking interfaces described in the previous section. And we verify Multi-Access Networking support by Multi-Access Box, in terms of hardware as well as software. For the integration and verification, we select three of those four interfaces that are widely used in practical area and are relatively easy to be integrated: SDN-enabled Wired interface, OVS-integrated WiFi interface, and OpenAir-based 4G wireless interface. Lastly, we extend Multi-Access Box into multi-site testbed, and verify its support of Multi-Access/Multi-Site experiments.

Multi-Access Interfaces Integration in Multi-Access Box

Figure 6. Multi-Access Box interface Preliminary Integration

SDN-enabled Wired, OVS-integrated, and OpenAir-based 4G interfaces must be integrated on the Multi-Access Box. To accomplish this, we will integrate the network configuration and software for running the interface into a single box as shown in Fig. 6.

For the SDN-enabled Wired Interface, configure the end of the SDN-enabled switch topology to connect to the wired interface of the Multi-Access Box. All switches are controlled by a unique





39 / 94

SDN controller, and the wired interface of the Multi-Access Box is also controlled by the SDN controller, so the wired interface of the Multi-Access Box is connected to the OVS (Open vSwitch) bridge which is connected to the SDN controller. SDN-enabled topology is configuredb with MikroTik's switch that support Openflow.The models CRS109-8G-1S-2HND-IN and CRS125-24G-1S-2HnD-IN are used, and the configuration of the Multi-Path and connection to the Multi-Access Box are shown at the top of the figure. These models can be configured with OpenFlow Enable using GUI Software called Winbox. These two switches support both wired and WiFi, but utilize only the wired interface for wired connection of IoT things. The WiFi interface to which IoT is connected is covered by OVS-integrated WiFi in Multi-Access Box.

To configure OVS-integrated WiFi interface, connect WiFi adapter to Server. WiFi adapter hardware has various types such as Mini PCIe (PCI express) type and USB dongle type. However, there are many compatibility problems with Mini PCIe type according to Server Hardware. Thus, the current prototype extends the Multi-Access Box using a USB dongle type WiFi adapter, which is relatively free for compatibility issues. OVS Integrated WiFi is implemented as an internal network configuration as shown in the figure above where the WiFi adapter is connected. we used the hostapd daemon to operate the AP according to the user's configuration and used the dnsmasq DHCP server to provide the IP address to IoT Things connected to WiFi. Br-IoT is an OVS (Open vSwitch) bridge, which is connected to SDN Controller. However, there is a bug that does not recognize the WiFi password when the WiFi interface is directly connected to the OVS bridge. That's why I put another Linux bridge named Br-AP between the OVS Bridge and the WiFi interface. With this configuration, IoT Things can be connected via WiFi and these devices can be controlled by SDN Controller.

The integration of the OpenAir-based 4G wireless interface into the Multi-Access Box is largely divided into two parts. The first part is the part of the eNB that carries the direct radio communication with the device which contacts the openAir-based 4G wireless interface and transfers the received data to the core network. The second part is the core network part that authenticates and controls devices contacting the OpenAir-based 4G wireless interface in Multi-Access Box and manages eNBs. The eNB must use RF equipment that provides a hardware platform from OpenAirInterface for radio communication with the device. The eNB of the OpenAir-based 4G wireless interface integrated in the Multi-Access Box adopts the ExMIMO2 board.

In order to contact the OpenAir-based 4G wireless interface of the integrated Multi-Access Box, the device's sim card programming is required. In the case of sim card programming, it is necessary to match the settings of the sim card with the preset values in the core network part of the OpenAir-based 4G wireless interface. The values to set are MCC, MNC, ID, and OP values. To use the device in the OpenAir-based 4G wireless network, finish the sim card programming and mount the sim card on the device to be used. After subscribing to the device on the core network, the device can be used in contact with the OpenAir-based 4G wireless interface of the integrated Multi-Access Box.





40 / 94

Multi-Access Networking Control Software

Figure 7. Multi-Access Box Control Software preliminary integration

Multi-access networking control software is software that controls multiple different access networking interfaces integrated into a multi-access box as a single interface. We have developed software that controls each interface, and then developed and verified software that integrates the developed software. The current software configuration includes a controller that controls the SDN-enabled wired and OVS-integrated WiFi, and a controller that controls the OpenAir based 4G interface. The Multi-Access Control Software operates as a Coordinator that coordinate SDN Controller and OpenAir 4G Controller. SDN Controller controls SDN-enabled Wired and OVS-integrated WiFi interface, and OpenAir 4G Controller controls OpenAir 4G Interface. Coordinator is located in Post of K-Cluster with SDN Controller and OpenAir 4G Controller, and controls two controllers at a high level to select the best interface for IoT End-Things to transmit data.

The operation and configuration of the Coordinator are shown in Fig. 7. To control each interface, IoT devices are equipped with an Agent that identifies what interfaces they have, and what the state of each interface is, and switches the interface to which the IoT devices transmit data. This agent is controlled by communicating with the coordinator and reports the interface status of IoT End-Things.

The overall operation of the coordinator is as follows. Agent reports to the coordinator the status of the interface of IoT End-things. The coordinator checks the report and stores it in the Device Status Database. Based on the contents of the DB, the coordinator selects the appropriate interface to transmit the IoT data of IoT End-Things. When the Coordinator interface selection is completed, the result of the selection is transmitted to the agent of IoT End-Things, and the data is transmitted to the Edge Cloud using the selected interfaces.

In the following subsections, we will discuss two scenarios and their testing results to verify the multi-access integrated control. The first scenario is a verification using a smartphone supporting LTE and WiFi, and the second scenario is a verification using Raspberry PI





41 / 94

supporting wired and WiFi Interfaces. For the first scenario, the Coordinator control signal must be able to be transmitted to the Agent through OVS-Integrated WiFi and OpenAir based 4G Interface. And to verify the second scenario, we connected an additional Ethernet port to Raspberry PI. Through this additional port, Agent and Coordinator communicate. The Coordinator can also check the status of the Agent and IoT End-Things and issue control commands to the Agent. This separation of management and control signals makes it easier and more efficient to control IoT end-things.

2.2.2.3 SDN//NFV based DMM

Before designing of the DMM, we briefly review our overall K-ONE playground architecture again as described figure 8. In that figure, various access technologies provide in each edge cloud and user devices can connect to edge cloud through these multi-RAN networks. Of course, most of the connecting devices have mobility behavior, so they can move not only inside one edge cloud coverage but also between two edge clouds. Even though our Multi-Access/Multi-RAT environment mentioned in previous chapter can support intra edge-cloud mobility, inter edge-cloud mobility is out of scope. Our Distributed Mobility Management(DMM) testbed aims to support inter edge-cloud mobility which can provide session continuity with minimized delay when user device moves and connects to other edge cloud. From this perspective, main principles for designing our DMM prototype are described as follows;

- Anchor Function separation: In DMM approach, a single anchor function which is defined in centralized mobility schemes such as Mobile IP(MIP) or Proxy Mobile IP(PMIP) should be distributed to multiple edge clouds. Anchor function may have IP anchoring, tracking of user's location or packet interception for forwarding management. Additionally, we also consider that separation of control plane and data plane of anchor function can give us efficient management aspects. Tunnel Elimination/Traffic Optimization: In current mobility management schemes (e.g. MIP and PMIP), IP-in-IP tunneling is used for traffic forwarding between two network entities after handover. However, our DMM approach provides a cost-effective and optimized path through SDN-based dynamic path configuration.

- SDN/NFV Compatibility: All DMM functionalities can be deployed as Virtual Network Functions (VNFs) via core cloud, and control the connectivity between distributed GWs through SDN controller.

The overall prototype design for DMM architecture that reflects the above principles is shown in Fig.8. The DMM Controller located in core cloud manage and configure all DMM data plane entities including the DMM-GW deployed in edge clouds and SDN-enabled switches between DMM-GWs. The DMM-GW implemented in each edge cloud is an endpoint of traffic in that cloud. The DMM-GW forwards packets from user device to the external network as well as it forwards packets of user’s traffic to another edge cloud that is newly anchor of moving user. All packet forwarding rules are configured from the DMM Controller and delivered via OpenFlow interface. Our DMM design does not affect operation and configuration of Multi-Access/Multi-RAT environment so there is no interference between them.





42 / 94

Figure 8. Overall Design of Distributed Mobility Management Support in K-ONE Playground

2.2.3 Evaluation

2.2.3.1 Multi--RAT access networking

Multi-Access Box: BATMAN-based mesh-WiFi interface Prototype

Figure 9. Multi-Access Box: BATMAN-based WiFi Interface





43 / 94

If only the representative WiFi AP is wired, the mesh network is a network in which existing wireless communication routers become mesh nodes and wirelessly connect all the segments. Mesh network supports Multi-RAT in wireless form and supports various devices, not all existing WiFi APs need to be connected by wire. To provide the Internet in an environment where the Internet is not available, the mesh network interface of the Multi-Access Box must be designed.

The topology of this interface is also constantly changed due to the dynamic connection and termination of the connected devices. Thus, it is not necessary to have all the routing information of the entire node. Therefore, this interface is designed to use BATMAN, a routing protocol for mesh networks that only memorizes the best link information between nodes.

The BATMAN-based mesh-WiFi interface requires a WiFi adapter to work as an AP. BATMAN-based mesh-WiFi node devices must be WiFi enabled devices. It must be a device capable of installing BATMAN and AP essential libraries for BATMAN operation and AP operation. For the operation to the AP and the DHCP server, this interface adopts hostapd and isc-dhcp-server libraries.

The structure of BATMAN-based mesh-WiFi interface is shown in Figure 9. When a node device is newly turned on, if a BATMAN-based mesh-WiFi network exists around the node device, the node device can join. If it does not exist, you can create a BATMAN-based mesh-WiFi network with any SSID and Channel. The node devices use Bluetooth beacon to inform the surrounding devices of the information of the BATMAN-based mesh-WiFi network to which they are connected. Also, a DHCP server is running on the first device that is turned on, and the connected device can then receive IP from the master node that is turned on first.

The BATMAN-based mesh-WiFi interface is connected to a controller that provides two functions. With this interface, node devices connected to the mesh-WiFi network can be provided with topology visualization and flow control functions using Multi-Access Box Control software and controllers.





44 / 94

Figure 10. BATMAN-based mesh-WiFi Implementation

To construct a Multi-Access testbed, a BATMAN-based mesh-WiFi testbed was constructed.

The testbed consists of a mesh network using WiFi, all of which consist of a Linux-based

operating system and an open hardware-based AP. The structure of the testbed was

constructed by building a mesh network by grouping 8 APs as shown in Figure 10.

The testbed used BATMAN to build the mesh network, a routing protocol for mesh network. All

BATMAN based mesh-WiFi APs use RaspberryPi2 B + and Ubuntu Mate 16.04 as OS. In order

to be controlled and managed in a Multi-Access Box, OVS, a Linux-based virtual software

switch, is set in the testbed. For basic WiFi operation of BATMAN-based mesh-WiFi network

AP, This testbed installed hostapd and isc-dhcp-server as AP related libraries. Testing was

conducted to provide flow control and topology visualization. Thus, basic flow control test using

OVS was performed on the corresponding testbed. In addition, we conducted web-based

visualization test for the topology of the testbed.





45 / 94

Multi-Access Box: OpenAir-based 4G wireless interface Prototype

Figure 11. Multi-Access Box: OpenAir-based 4G wireless Interface

4G network is a technology that provides various services such as ultra-wideband Internet access and multimedia. To provide Internet access and various services of mobile devices through LTE, 4G wireless interface of the Multi-Access Box must be designed.

Hardware / Software 4G network platform is required to build 4G wireless interface. Therefore, we adopted a hardware / software platform called OpenAirInterface. OpenAirInterface has emerged for lab-level 4G network construction and experimentation, and provides a variety of software and hardware drivers and firmware.

The eNB of the 4G network serves as a base station, and it exchanges traffic with devices connected to the 4G wireless interface through radio communication. To build the eNB, this interface adopted the ExMIMO2 board with four RF chipsets. In addition, it needs a device that can install OpenAirInterface software to operate 4G wireless interface. The software of OpenAirInterface is divided into two parts of eNB and Core Network, and each part needs independence.

The structure of the OpenAir-based 4G wireless interface is shown in Figure 11. The OpenAir-based 4G wireless interface consists of two parts: Core Network and eNB. The Core Network part manages profiles for devices that contact the OpenAir-based 4G wireless interface and provides subscription and authentication capabilities. The eNB part can communicate with the contact devices via OpenAir-based 4G wireless interface and radio communication.

The OpenAir-based 4G wireless interface is connected to a controller that provides various functions. Through this interface, devices connected to the 4G wireless network are subscribed and terminated through the Multi-Access Box Control software and controller. Through this controller, the information of the eNBs of the 4G wireless network can be continuously updated and set.





46 / 94

Figure 12. OpenAir-based 4G Wireless Network Implementation

In order to construct Multi-Access testbed, OpenAir-based 4G wireless network testbed was

constructed. The testbed constitutes 4G network using LTE communication and consists of

Core Network, eNB, and UE. The OpenAir-based 4G wireless network testbed was built using

Core Network, eNB and mobile devices as shown in Figure 12.

The testbed was constructed using OpenAirInterface, a hardware and software platform for

building and testing lab-level 4G networks. OpenAir-based 4G wireless network testbed was

built using x86 based PC and Ubuntu 14.04 LTS as operating system. To build the Core

Network and the eNB corresponding two parts respectively, two separate hosts were

constructed as core network (MME, HSS, and SPGW) and eNB part respectively. The eNB

part uses ExMIMO2 board equipped with 4 RF chipsets and is built using OpenAirInterface

hardware driver. Monitoring information such as signal intensity, channel impulse response,

etc. of eNB can be monitored using the monitoring tool provided by OpenAirInterface.

Based on the 4G Network testbed, we tested connectivity between UE and corresponding testbed. Also, the modules for controlling and managing the constructed OpenAir-based 4G network were tested. The experiment consisted of three parts. The first is the subscription and termination experiment of mobile devices to the established OpenAir-based 4G wireless network. Next, an experiment was conducted to configure the eNB using the remote controller. Finally, we experimented to monitor the configuration status and profile of mobile devices and eNB using remote controller.





47 / 94

2.2.3.2 SDI based Distributed Cloud Cluster

Multi-Access integrated Control Experiment: Scenario #1 & Results

The first scenario involves using smartphones that support LTE and WiFi. No additional Ethernet for the Management and Control can be connected to the smartphone. Therefore, the agent and the coordinator of the smartphone communicate using the channel through which the IoT data is transmitted. In addition, when smartphone is connected to WiFi, smartphone does not transmit data through LTE, but transmits data using only WiFi. For this reason, smartphone’s interface selection implemented based on the strength of the WiFi signal.

Figure 13. Sequence Chart for Scenario 1

Coordinator and Agent communicate based on Socket communication. The coordinator's TCP / IP server creates a socket, and the Agent's TCP / IP client connects to the server socket. Once connected, the Coordinator will populate the Console with the IP address of the Agent. When a smartphone uses Wifi to transmit data, Smartphone IoT End-Things continuously sends the WiFi AP's RSSI (WiFi signal strength) value to the Coordinator. The Coordinator checks the RSSI value in units of 3 seconds and disables the WiFi when the RSSI value falls below -80 dBm. When WiFi is disabled, IoT End-things uses LTE to transmit data. During this control operation, the Coordinator can receive keyboard input. By using the keyboard input, user can arbitrarily control ON / OFF of WiFi of smart phone IoT End-things and control the interface by utilizing it. If 'ON' is input to the keyboard input, WiFi of the smartphone is enabled, and when 'OFF' is input to the keyboard input, the WiFi of the smartphone is disabled. Figure 13 is a sequence chart showing the sequence of software operation for verification of scenario 1.





48 / 94

Figure 14. Verification Results of Scenario 1

Verification of scenario 1 proceeded as follows. I used a smartphone with WiFi enabled status and connected it to Multi-Access Box. Then check that the Agent is correctly measuring the RSSI value. The RSSI value appears as a pop-up window when you run the Agent. The left figure in Figure 14 shows the result of RSSI measurement in Agent. After confirming RSSI value measurement, input 'OFF' through keyboard input to check if WiFi of smartphone is disabled. Figure 14 shows the result of controlling WiFi through keyboard input. Finally, force the RSSI to drop below -80 dBm and check if WiFi is transferred to disabled state. To forcefully lower the RSSI value, I went out of the room with the AP and proceeded with the experiment. The right figure in Figure 14 shows the results for the experiment.

Multi-Access integrated Control Experiment: Scenario #2 and Results

In order to verify the multi-access integrated control between SDN-enabled Wired and OVS-integrated Wifi interfaces, in the first scenario, we utilize Raspberry PI with additional Ethernet port. Another Ethernet port is used to exchange management and control signal between coordinator and IoT End-Things. And on this scenario, the interface to transmit IoT data is selected according to the priority of the interface. SDN-enabled Wired is the highest priority, followed by OVS-integrated WiFi and OpenAir based 4G. If the SDN-enabled Wired is connected, the data is transmitted through the wired network and the OVS-Integrated WiFi If it is connected, data will be sent using WiFi.

Figure 15 is a sequence chart of the software created to verify the scenario 2. Scenario 2 Verification software is largely divided into three phases.

The first phase is the Initial Status Report Phase to check the initial state of the test bed environment. The top part of Figure 15 corresponds to this, and SDN Controller, OpenAir 4G Controller and IoT End-Things report their status to Coordinator. The controllers send a list of





49 / 94

IoT End-Things under their control to the Coordinator, and IoT End-Things sends the status of their interfaces and interfaces to the Coordinator. Coordinator compares the information received from IoT End-Things and controllers and stores them in the Device Status Database. The database stores the interface information of all IoT End-Things connected to the test bed. If there is IoT End-Thing 1, check if this device has Wired or WiFi Interface. If you have an interface, also store the IP address of the interface. Also, the operation is performed once every 30 seconds.

The second phase is the Interface Selection Phase. In the phase, IoT End-Things selects the interface to use for transmitting data. Select the interface of IoT End-Things based on the Device Status Database contents saved in Initial Status Phase. The Sequence Chart for that action is in the middle of Figure 15. If IoT End-Thing 1 has Wired Interface, install Intent using SDN's Intent Framework to transfer data using Wired according to priority. If you have both Wired and WiFi interfaces, you will use Wired to transmit data, and WiFi will be stored in a structure called Candidate Interface.

The third phase is the Interface Transfer Phase, which handles the transfer of data using the Candidate Interface due to a change in the interface of IoT End-Things. If IoT End-Thing 1 can not send data using wired interface due to physical disconnection during data transmission using Wired Interface, install Intent with Interface saved in Candidate Interface structure The operation is performed. The Sequence Chart for that action is shown at the bottom of Figure 15.





50 / 94

Figure 15. Sequence chart for Scenario 2

Figure 16. Verification Results of Scenario 2

The verification of the above software proceeded as follows. Phase 1 utilizes an additional ethernet port to allow IoT End-Things to report its own status. However, there were difficulties in implementing it. Because of this, I have stored the information of IoT End-Things in the topology in advance in the JSON file. The format of the JSON Format is shown on the left of





51 / 94

Figure 16. Each IoT End-Things stores the IP address of its interface. If it does not have a specific interface, the IP of the interface is stored as "NONE".

In order to verify the software, the topology shown in middle of Fig. 16 is configured. And the result of Phase 2 Interface Selection is shown on the right of Fig. 16. Compare the data in the SDN controller and the JSON file to produce the results. The intent is set up so that data can be transferred using the point-to-point Intent using the location value resulting from the interface selection. After this selection is completed, the interface transfer will proceed. Currently, the implementation of this part is insufficient, so the verification of the contents is left as a future work.

Multi-Access integrated Control: Future Extension

We verified feasibility of Multi-Access integrated Control with Multi-Access Box and Multi-Access Control Software in previous sections. In addition, it is possible to extend the previous experiments into Multi-Access/Multi-Site-integrated scenario with Multi-Access/Multi-RAT Edge-Clouds Testbed. In order to show the possibility, we describe an example scenario that covers both Multi-Access and Multi-Site aspects.

Figure 17. Future Extension Scenario: Multi-Access/Multi-Site-integrated Experiment

Fig. 17 represents an example scenario of Multi-Access/Multi-Site-integrated experiment. In this scenario, we use two K-Clusters (i.e., GIST and Soongsil University) in K-ONE Playground. K-Clusters are geographically separated each other and connected via L3 WAN (Wide Area Network). We construct OpenStack-based edge-clouds on each K-Cluster. In addition to edge-clouds, we apply Multi-Access-integrated control to GIST K-Cluster. That is, we add Multi-





52 / 94

Access Box, its networking control software to GIST K-Cluster. Then we connects Raspberry PIs and mobile devices that have wired Ethernet, WiFi wireless and 4G wireless interfaces.

On this testbed, we assume that a user at Soongsil University want to receive data from a subset of End-things in GIST K-Cluster. We configure a VM in each K-Cluster, and end-things starts to send its hardware status such as CPU and RAM to GIST VM through its access interface. Multi-Access Box controls the multi-access networking, and relays data received from End-Things to the VM. Then SSU (Soongsil University) VM requests data of specific End-Things, then GIST K-Cluster passes only requested data.

Consequently, this scenario shows that a simple Multi-Access/Multi-Site-integrated use case. Likewise, other use cases of multi-access/multi-site SDI are able to experiment easily with our Multi-Access integrated Control feature and Edge-Clouds testbed. In this reason, we insist that combining Multi-Access integrated Control feature and Multi-Site Edge-Clouds testbed is able to support multi-access/multi-site-integrated SDI experiments.

2.2.3.3 SDN/ NFV based DMM

OpenFlow-based Multi-site DMM Control Plane & Data Plane

Fig. 18 shows the details of OpenFlow-based DMM components. SDN technology enables to separate control and data plan function. Network forwarding devices such as switch or router are simplifies and abstracted as data plane entity and the control plane functions can be programmable application running on the SDN controller. In our testbed, OpenFlow-based DMM functions are implemented based on open source [10] and it is modified to fit our design. We use Ryu Controller v.3.17 as SDN controller and all DMM control plane functions are written in python format and running as Ryu SDN application. In the original source, they implemented accesspoint data plane function which is responsible for making user connection and forwarding management between user device and the DMM-GW. Since that we assume that access control is already done in our testbed by Multi-Access controller mentioned previous chapter, we removed access control function in DMM function and modified related dependencies.

The DMM Control Plane functions running as SDN controller application is describes as follows;

- Node Management(Nmm): It monitors data plane nodes including DMM-GWs and transport switches. When a data plane node connects to the DMM controller, this function determines whether connecting data plane node is DMM-GW or transport switch and it calls according events for data plane node configuration. This function also manages link between data plane nodes.

- Gateway: It manages configuration and status of the DMM-GW. This function configures OpenFlow rules according to events related to operation of the DMM-GW. Events includes connecting of user, initializing of gateway and writing OpenFlow rule for user traffic.

- Anchor Management(Amm): This function includes operation of anchor function. Using this function, the DMM-GW can assign IP address to the user, detect handover between anchors. OpenFlow rules between anchor is also managed by Amm function.

- Mobility Management Entity(MME): This function is responsible for detail status of user device. It detects user attachment in network and make association with user. It also manages subscriber database.





53 / 94

The DMM data plane is classified to the DMM-GW and transport switch. The DMM-GW is implemented as a virtual machine inside of edge cloud. It can be dynamically deployed on the edge cloud by Management & Orchestration(MANO) function in the control tower of K-ONE Playground. Inside of the DMM-GW, OpenvSwitch(OVS) is running as a SDN-enabled switch and it makes connection with the DMM controller using OpenFlow v.1.3 protocol. Transport switch is deployed to forward traffic between DMM-GWs. It is also SDN-enabled which is managed by the DMM controller and configured forwarding rules by using OpenFlow protocol.

Figure 18. OpenFlow-based DMM Control Plane and Data Plane

In the following sections, we implement DMM functions in out testbed and test its operation following two scenarios. First scenario is included initial setting of the DMM-GW, user attachment and anchor assignment. In this scenario, we verify user connection to the DMM-GW of edge-cloud, IP address assignment and configuration of forwarding rules at the DMM-GW. Then we move forward to verify handover scenario including user mobility detection, re-configuration of forwarding rule and new anchor assignment. After verifying these scenario, we finally discuss about flexible deployment of the DMM-GW function as VNF which is managed by MANO component in K-tower of core cloud.





54 / 94

OpenFlow-based Multi-site DMM Experiment: Scenario #1 & Result

Figure 19. Procedure of connection establishment between the DMM-GW and the DMM controller

The first scenario includes procedures from connection between the DMM-GW and the DMM controller to configuration of OpenFlow forwarding rules for delivering traffic of user device. Figure 19 describes procedure of connection establishment between the DMM-GW and the DMM controller, and initial configuration of the DMM-GW. When the DMM-GW in edge cloud is activated, since OVS has already set controller IP as SDN controller in core cloud, OpenFlow handshaking procedure is performed between the DMM-GW and SDN controller. During handshaking, SDN controller triggers EventSwitchEnter event and the Nmm function in the DMM controller determine whether switch information of triggered event represent the DMM-GW. The DMM-GW is determined from port name specified only for the DMM-GW. If the Nmm determines it as the DMM-GW, it calls Gateway function to make OpenFlow rules and configure these rules to the corresponding DMM-GW via the SDN controller. As general SDN operation, all data plane node exchange Link Layer Discover Protocol(LLDP) message between them to detect neighbor. When the DMM-GW or transport switch detects its neighbor node, this information is sent to the SDN controller and triggered EventLinkAdd event in the SDN controller. This event calls Amm function of the DMM controller to update network topology and routing information, finally it makes to configure OpenFlow forwarding rule between two data plane nodes and deliver rules to corresponding data plane nodes.





55 / 94

Figure 20. Procedure of UE initial attachment

Figure 20 describes procedure of UE initial attachment to the DMM-GW after completing previous procedure. Note that procedure and details how UE can connect to edge cloud and how packet of UE can be delivered to the DMM-GW is out of scope. When the (Router Solicitation(RS) packet of UE comes to the DMM-GW in edge cloud, following OpenFlow rules which are inserted at the DMM-GW initialization phase, this packet is captured and sent to the SDN controller via OpenFlow PKT_IN message. Following general operation of the SDN controller, EventOFPPacketIn event is triggered and it calls the MME function of the DMM controller. The MME functions firstly determine whether this packet comes from known UE stored in the database, then it triggers EventUEConnected event to deliver UE information to other functions. This event calls Gateway and Amm functions to configure OpenFlow forwarding rules for UE traffic and assign DMM-GW as anchor to the UE. After pushing OpenFlow rules to the DMM-GW via SDN controller and the DMM-GW sends Router Advertisement(RS) message including network prefix to UE, the DMM-GW can handle uplink/downlink traffic of UE and finally UE can send packet to the corresponding node. Optionally, the SDN controller can configure rules to transport switches for forward UE’s packet to the corresponding node.





56 / 94

Figure 21. Multi-site DMM experimental testbed for scenario #1

To verify these scenario, as shown in Figure 21, we implemented three VMs on single physical server using OpenStack; the DMM-GW, DMM controller and User Equipment(UE). All linked between VMs (e.g. between DMM-GW and DMM controller, between UE and DMM-GW) are deployed via OpenStack Neutron, and connection establishment is triggered automatically when VM turns on. Our experiment result is show in figure 22. When the DMM-GW turns on and connects to the DMM controller, we verified that it received initial OpenFlow rules from the DMM controller. Following these rules, the DMM-GW function can handle several ICMP packets (e.g. Router Solicitation, Neighbor Advertisement, Neighbor Solicitation) from connecting devices and send RS packet to the DMM controller for triggering UE connection procedure. After completing UE connecting procedure, you can check that the global IPv6 address of UE is assigned following network prefix of connecting DMM-GW.

Figure 22. Verification Result of OpenFlow-based DMM scenario #1

UE(Ubuntu Trusty)

DMM-GW DMM controller

OVS

SDN-Controller(Ryu)

Nmm Amm

GatewayMME

Net1

Net2

Net3

NeutronNetwork

Multi-Site DMM testbedVM1 VM2 VM3





57 / 94

OpenFlow-based Multi-site DMM Experiment: Scenario #2 & Result

Figure 23. Procedure of UE handover

In this chapter, we verified handover scenario between two DMM-GWs. Figure 23 describes detail procedures of all DMM functions in this scenario. When UE detaches from edge cloud 1, attaches to edge cloud 2 and sends RS message, the DMM-GW 2 in edge cloud 2 send RS message to the SDN controller via PKT_IN message. Due to the EventOFPPacketIn event, the Amm function in the DMM controller checks previous attachment point of UE based on the mapping table by using MAC address of UE and it triggers EventAnchorUpdate event to maintain IP address assigned by pervious anchor. The Gateway function receives this event, configures OpenFlow rules for forwarding uplink/downlink traffic of UE, and push these rules to two DMM-GWs by triggering EventWriteOFRule event. For handover completion phase, when RA message triggered by Amm is delivered to UE, UE can use not only previous IP address assigned from previous DMM-GW but also new IP address assigned by new DMM-GW. New IP address is used for new sessions created after handover. After completing all operation, on-going session traffic is forwarded between previous anchor DMM-GW and new anchor DMM-GW using OpenFlow rules. When UE establishes new session, traffic of this session is directly forwarded to the corresponding node at new DMM-GW.





58 / 94

Figure 24. Multi-site DMM experimental testbed for scenario #2

To verify this scenario, as shown in Figure 25, we implemented 4 VMs in single physical server using OpenStack; two DMM-GWs, UE and DMM Controller. As similar as scenario #1, all links between VMs are deployed via OpenStack Neutron. For simulating handover, we manually set same MAC address at two NICs of UE and changed attach point by sequentially turning on/off these network interfaces. Our verification result is shown in Figure 25. When UE changes its attachment point, we verified that DMM controller knows UE handover, pervious anchor of UE, and finally controller re-configures OpenFlow forwarding rules between two DMM-GWs. We verified that all operations were correctly followed our designed procedure. At the result of handover operation of the DMM controller, we verified that UE finally has two global IPv6 addresses; pervious IP address was maintained and new IP address was assigned from new DMM-GW. We also checked OpenFlow tables in two DMM-GWs and verified that all rules are correctly configured as we design. Total downtime during handover was calculated about 0.2 second by UE from detaching NIC to receiving RA message.

Figure 25.Verification Result of Open Flow-based DMM scenario #1





59 / 94

3 EU - 5G Mobile Core Deployment

3.1 5G Test Network at a glance

Fostering research and innovation within industry and academia to come up with promising and adaptive solutions for next generation mobile networks, 5G Test Network Finland (5GTNF) is a platform for the collective coordination and integration of testbeds all around Finland. To realize its aim and objectives, currently 5GTNF is focusing on all 5G related research areas from physical end devices to network and applications. Figure 26 depicts a generic overview of the effort being carried out having combined testbeds of different nature all around Finland and also with cross continental experimental testbeds. [1]

Figure 26. 5G Test Network Finland [1]

5GTN+ project within 5GTNF is jointly hosted by University of Oulu and VTT Oulu along with 20 industrial partners [1].

The University of Oulu has an open, innovative and 5G proof of concept testbed deployed which is developed and maintained by industry experts working in the 5GTN Integration team. The testbed is open for research and its frequently being used at the Center for wireless communications by researchers in different projects. The testbed is also used by different consortium partners under certain terms and conditions to test their solutions within a 5G capable environment. Figure 27 provides with an overview on the generic infrastructure and the key functional blocks within 5G test network. keeping the scope of this document in view, we will only consider the details of the evolved packet core (EPC) resources within 5GTN in connection with the 5G champion project [2].





60 / 94

Figure 27. 5GTN infrastructural layout [2]

3.2 Cloud core resources within 5GTN in a multi slice environment

5GTN has a multi slice implementation of its core resources both locally and in different cloud environments. It primarily comprises of an end to end Nokia based proprietary virtual EPC, OpenEPC powered by core network dynamics and OpenAirInterface (OAI) open source core network. Figure 28 represents all the core network resources located in 5GTN in different virtual infrastructure environments. Some of them comprise of local instances on COTS hardware and some are deployed on IaaS cloud run time environments.

Figure 28. Multi Slice cloud core resources in 5GTN





61 / 94

3.3 OpenBaton in a nutshell

OpenBaton is an open source MANO platform in compliance with ETSI MANO specifications. It comprises of a modular software based NFVO, a generic VNFM and Element Management System (EMS), a driver mechanism supporting different types of VIMs, an event engine for dispatching life cycle execution events based on a pub/sub mechanism and a set of libraries in java, Go and python. The generic VNFM is an intermediate node between the NFVO and VNFs. It interoperates with the EMS, an agent inside the VMs, to execute scripts in the VNF package inside the virtual network function descriptor (VNFD). The communication between the NFVO and EMS is managed by the generic VNFM using the AMPQ protocol over RabbitMQ. Figure 29 illustrates the way in which the lifecycle of a VNF is managed by the VNFM on the instructions of the NFVO. In practice, the EMS executes scripts in the VNFC on the instructions send by the VNFM [3].

Figure 29. AMPQ protocol based communication over RabbitMQ [3]

3.3.1 NFVO-VNFM Rest Interface

The VNF Manager communicates with the NFVO using the exposed REST interface. According to ETSI MANO specifications, it is the or-vnfm interface between the NFVO and the VNFM. [3] Figure 30 depicts the ordering of the REST methods in a sequential order. In principal, the NFVO calls REST methods on the VNFM and expects a response in return. It starts with registering the VNFM to the NFVO by a POST request on “OrEndpoint/admin/v1/vnfm-register”. Then an instantiate call tries to manage the lifecycle of a VNF by instantiating and modifying it along with the start/stop process. The instantiate call as shown in Figure 30 provides the VNFM with the VNFDs used to create the records of a VNF i.e. VNFR. It also sends the VNFM the associated scripts to be executed during the life cycle of a VNF. It is actually a POST request on “vnfmendpoint” [3].





62 / 94

Figure 30. Sequence Diagram [3]

3.3.2 VMs deployment

The instantiation of VNFs can be performed either with resource allocation done by NFVO or with resource allocation done by VNF manager. The generic VNF manager in our case realizes VMs deployment using the resource allocation done by the NFVO [3]. Two types of messages are sent to the NFVO in this approach:

1. GRANT_OPERATION message





63 / 94

Initially it verifies the availability of resources on a particular PoP. If enough resources are available “GRANT_OPERATION message” is returned otherwise it will throw and error message. Upon returning the verification message, the resource allocation message can be sent to the VNF manager i.e. ALLOCATE_RESOURCE message. [3]

2. ALLOCATE_RESOURCE message

The NFVO creates all resources upon receiving the ALLOCATE_RESOURCE message and it will be returned to the VNF manager if there are no errors. Having got the message, VM are created and the VNF record (VNFR) is filled with all associated values. [3]

3.3.3 Script Execution Constraints and VNF lifecycle management

The VNF manager sends scripts contained in a VNF package to the EMS which then executes them in individual VMs. The NSD contains ordering of the scripts which then creates NSR as part of the CNF Lifecycle Management. Instantiate, Configure, Start, Stop, Terminate and Scale_IN are the scripts associated with VNF lifecycle. While packaging each VNFs, the parameters are defined in the VNFDs utilizing specific environment variables like configurations, hostname, floating IP etc. [3]

3.3.4 VMs Termination

In order to stop a VNF service, the STOP lifecycle event is used which can then be start again later on. The Terminate lifecycle event triggers removal of resources from the PoP. The NFVO performs the VMs termination by putting lifecycle termination script under the Terminate lifecycle event. [3]

3.4 NFV based MANO deployment in 5GTN infrastructure

NFV makes the abstraction of VNFs from the underlying hardware possible by introducing a hypervisor layer in between. The compute, storage and network resources on a physical hardware can then be efficiently distributed among all the network functions which are placed in different virtual infrastructure environments. The MANO framework enables dynamic orchestration, provisioning and scaling of the virtual instances. Keeping the impacts of virtualization in view, 5GTN has realized an orchestration mechanism for some of its cloud core resources. OpenBaton is the MANO framework being used and Openstack serves as the virtual infrastructure manager (VIM). Figure 31 demonstrates the multi vim deployment utilizing OpenEPC and OAICN in different OpenStack environments both being orchestrated from Open Baton. Initially both default and test VIMs are registered within OpenBaton as a point of presence (PoP) which in principle is realized by the OpenStack VIM driver. The images of both EPCs are created using Disk Image builder (DIB) after compiling them form their respective repositories. Making the image readily available on Open Stack, VNF packages are created for them individually. OpenEPC comprises of five tar archives i.e. mme, spgw, epc-enablers, epc-client, eNodeB whereas OAICN consists of mme, hss and spgw tar archive VNF packages.





64 / 94

Figure 31. multi-vim vEPC deployment in 5GTN

The network and topology in each vEPC case is realized by Heat orchestration which in practice is the creation of virtual routers, gateways and all the required interfaces for each virtual machine. The instantiation, configuration and termination is then realized by OpenBaton NFVO and VNFM by executing and managing scripts associated with the life cycle management of VNFs.

3.4.1 Open Baton Service catalogue

3.4.1.1 Open stack VIM and PoP registration

Virtualized infrastructure Manager (VIM) is the managing and controlling entity for the Network Function Virtualization infrastructure (NFVI). It primarily manages the compute. storage and network resources in a point of presence (PoP) or multiple PoPs. OpenBaton has a driver mechanism to support a range of VIMs, open stack is one of them. The Openstack driver provides with a smooth integration of OpenBaton with Openstack VIM. We can instantiate resources on a PoP once the PoP is registered within the OpenBaton Platform. In order to realize a multi vim deployment using two different EPC solutions, two different Openstack run time environment are registered as different PoPs. As depicted in Figure 32, default and Test are the two VIMs registered with OpenBaton. [3]





65 / 94

Figure 32. multi VIM PoP registration

3.4.1.2 VNF Package Management

A VNF package is a tar archive and it contains all the information regarding the life cycle of a VNF [3]. A VNF should be properly packaged to be om boarded to the NFVO. A VNF package comprises of following requisite components:

A VNF Descriptor having all information needed for a VNF deployment.

An image link to a QCOW type file.

A Metadata which provides the NFVO information regarding the content of the VNFD.

A folder containing all the scripts needed for VNF lifecycle management.

Each EPC nodes are separately packaged to be uploaded to the service catalogue as VNF packages. Figure 33 demonstrates all the VNF packages of OAICN and OpenEPC being on boarded on Open Baton.





66 / 94

Figure 33. OAI and OpenEPC VNF packages on boarded

Figure 34 depicts all the VNF descriptors as part of each EPC packages along with their respective VNF IDs.

Figure 34.OAI and OpenEPC VNF descriptors





67 / 94

3.4.1.3 Network Service Descriptors

The Network service descriptor (NSD) is a template file in compliance with ETSI MANO specifications [3]. Having onboard the VNF packages, the NFVO use NSD to deploy network services. NFVO supports both JSON and TOSCA file template for NSD. A single descriptor is created for OpenEPC and OAICN VMs as shown in Figure 35.

Figure 35. multi-vim Network Service Descriptor

The multi-vim NSD comprises of VNFDs of all the VNF packages along with the associated VNF dependencies. Figure 36 provides with the list of all the VNFDs upon which the NSD is composed for both OpenEPC and OAICN.

Figure 36. Network Service Descriptor





68 / 94

3.4.1.4 Launching Network Service

Utilizing the created NSDs for both OpenEPC and OAICN, each one is launched in different VIMs. Figure 37 depicts the launching of network service while specifying the VIM on which individual VM instances is to be launched.

Figure 37. Launching Network Service Descriptor

3.4.1.5 VNF Records

Initially, the network service record (NSR) created upon launching the network service descriptor is in NULL state. Once the instantiation is finished in the VNFM, the virtual network function records (VNFR) will be set to ‘INSTANTIATED’. The NSR will also change to the INSTANTIATED state when all the VNFR are in INSTANTIATED state. Now, when the START message is sent to the VNFM and returned, the NFVO sets the VNFR state to ACTIVE. When all the VNFR are in ACTIVE state, the NSR will also change its state to ACTIVE. The similar change of state will be followed in case of MODIFY and TERMINATE states. Figure 38 represents the states and the VNF records of each VM.





69 / 94

Figure 38. VNF Records

3.5 Multi-Site NFVI and VIM deployment

Having a centralized MANO system with a distributed NFVI ensures the reduction of operational expenditure. It also makes the NFV platform scalable and highly available for VNFs in multi-site clouds. To realize a multi-site deployment two virtual EPC are chosen to be deployed in different NFVI from a centralized Open Baton NFVO/VNFM. [4]

3.5.1 OpenEPC

OpenEPC is a mobile core network deployment in software keeping in view 3GPP standards up till release 12. It is developed and maintained by core network dynamics (CND). [5] Figure 39 shows the network graph of OpenEPC VM instances on ‘default’ VIM and Figure 40 depicts the detailed network topology.

3.5.1.1 OpenEPC Network Topology

OpenEPC uses different LAN segments having different subnets on each network segment. OpenEPC VMs are interconnected in a meshed topology. Following are the LAN segments: [6]

net_a: It is the PDN and it connects the PGW with the application functions. Usually it is the NAT network which gives access to the internet.

net_b: It is the operators backhaul, connecting the access network gateways and the PGW.

net_d: It is the 3GPP radio access backhaul and it connects RAN with the EPC core network.

mgmt: It is the management and signaling network. All the signaling i.e. Diameter goes through this network and it serves as a management network for accessing each VM.





70 / 94

Figure 39. OpenEPC network topology (copyrights core network dynamics)

3.5.1.2 OpenEPC VM Instances

The functional nodes of OpenEPC core network along with the client and eNodeB emulation is instantiated on ‘default’ VIM based on the on boarded VNF packages and the NSD in Open Baton provides with an overview on the way the OpenEPC VMs are interconnected. All the LAN segments can be seen along with their interconnections. Figure 40 provides with an overview on the way the OpenEPC VMs are interconnected. All the LAN segments can be seen along with their interconnections.

Figure 40. OpenEPC VNFs Instantiated by OpenBaton on ‘default’ VIM (copyright core network dynamics)





71 / 94

3.5.2 OpenAirInterface Core Network

OpenAirInterface is an open source software implementation of the LTE system realizing the complete protocol stack of 3GPP standards. Though it has a customized build for an LTE base station, User equipment (UE) and core network (OAICN) but here we will narrow down, OAI to core network only. The OAI Evolved packet core (EPC) implements mme, spgw and hss in software instances. The protocol stack implementation as shown in Figure 41 comprises of GTPv1u, GTPv2c, NAS integrity, encryption using AES and snow3G algorithms. [4]

Figure 41. OAICN evolved packet core [4]

3.5.2.1 OAICN Network Topology

The OAI CN is deployed in an Open Stack VIM names as ‘Test’. The VMs are connected to public internet through OAI-router and OAI-InternalNet provides with an interconnection of the VMs. Figure 42 depicts the network graph of OAI CN on ‘Test’ VIM.

Figure 42. OAICN network topology





72 / 94

3.5.2.2 OAI VM Instances

The OAI core network has three functional nodes which are hss, mme and spgw. The OAI CN virtual machines are interconnected via the OAI-InternalNet tenant network. The detailed network topology on Test Vim is shown in Figure 43.

Figure 43. OAI VNFs instantiated by OpenBaton on ‘Test’ VIM

3.6 Evaluation

3.6.1 On boarded vEPC interworking with LTE RAN

Test Title Testing LTE connectivity using Pico cell on band 7 with OpenEPC

Test Objectives EPC service testing using Pico cell

Attach an LTE band 7 terminal

Verify the attachment using Wireshark logs

Test Setup

IP Addresses eNodeB mme SPGW EPC-Enablers

192.168.4.9 net_d 192.168.4.80 net_d 192.168.4.10 mgmt 192.168.254.33

mgmt 192.168.254.80 mgmt 192.168.254.10 net_a 192.168.1.40

net_b 192.168.2.20 net_a 192.168.1.10





73 / 94

UE attach

Request

Delete Session

Authentication

Information

Request and

Answer

Session ID

IMSI value

Authentication

Information

Answer

NAS Security

Setup

Update Location

Request/Answer

Attach Accept and

bearer

establishment

Create Session

Request from

MME and SPGW





74 / 94

IP address

allocation to the

terminal





75 / 94

Status of the

Terminal UE





76 / 94

Internet Browsing

GTP <TCP> from

SPGW

verifying 5gnt.fi (193.166.161.188) from spgw while browsing it from the attached UE i.e. 192.168.3.50 by following the TCP stream

Test Title Open Air Interface CN Connectivity using USRP B200

Test objectives Attaching COTS and emulated UE with OAI CN





77 / 94

Test Setup

COTS UE attach

One active COTS UE as can be seen in the statistics and the scope depicts the software define radio

based eNodeB.

OAI UE





78 / 94

The UE scope showing the successful attachment

IP Allocation

The Allocated IP address to the UE i.e. 192.168.3.50

3.6.2 OpenBaton VNF Package Management

Test Title OAI and OpenEPC VNF package Management using OpenBaton

Prerequisites vnfd.json

metadata.yaml

scripts

Test Objectives Examining the on boarded VNF packages and VNF Descriptors of OpenEPC and OAI CN on Open

Baton MANO

VNF packages

List





79 / 94

OpenEPC VNF

Descriptors on

PoP ‘default’





80 / 94

OAI CN VNF

Descriptors on

PoP ‘Test’





81 / 94

3.6.3 VNF instantiation using Open Baton MANO

Test Title OAI CN and OpenEPC VNFs Instantiation using Open Baton

Test Objectives To showcase instantiation of OAI EPC and Open EPC in different VIM leveraging NFVO-VIM interface

NSD catalogue





82 / 94

Launching

NSD

VNF Records

Network

topology of

OpenEPC and

OAI CN on

open stack





83 / 94





84 / 94

confirmation

of VNF

instantiation

from logs

3.6.4 VNF Termination using Open Baton MANO

Test Title OAI CN and Open EPC VNFs termination using Open baton

Test Objectives To showcase the deletion of VNFs of OpenEPC and OAI CN on terminating from Open Baton

NSD Delete

Verification from

Logs





85 / 94

4 5G mobile core Interoperability test

4.1 Overview (KR+EU)

4.2 Service Interoperability (KR+EU)

Interoperability is a characteristic of a product or system, whose interfaces are completely understood, to work with other products or systems, at present or future, in either implementation or access, without any restrictions.

For Service interoperability, we have conducted inter-continental application service demonstration over KR-EU dedicated network geared to show proof-of-concept of availability of 5G mobile core networks for disparate types of services.

We have established 5G interconnection between GangNeung Olympic venue in Korea and 5GTN in Finland to connect two mobile core networks at the end of interconnection created by Korea and Europe respectively.

For service interoperability, we are aiming to prove that the NFV/SDN based 5G mobile core interconnection network is suitable for various types of applications such as broadband services, low-latency services and immersive real-time service with AR and VR experiences to the end users.

For 5G mobile core service interoperability, we have demonstrated application services at the GangNeung ICT center during the 2018 PyeongChang Olympic Games.

Real-time streaming 360-degree video from EU to KR

Interactive Augment Reality with Maps

Figure 44. The bandwidth requirements for streaming various visual content

4K video streaming was demonstrated via our L2VPN dedicated connection and public internet access to compare the quality. Video servers were located in KR. Video streaming used UDP transfer. The downlink bandwidth could be verified by the computer's performance tools available from the “Task Manager.” With dedicated access, 4K video streaming showed very good performance, and the downlink bandwidth used was about 60 Mb/s to 65 Mb/s. The bandwidth was less than 10% of the total available bandwidth. However, 4K video streaming





86 / 94

via the public internet exhibited very bad quality. The downlink bandwidth via the public internet

was about 8 Mb/s, which was not sufficient to obtain a good end‐user experience.

Figure 45. Intercontinental Service Interoperability Testbed

4.3 Management Interoperability

4.3.1 SOL003-based management Interoperability (KR)

From an operational management perspective, interoperability should be determined first by deciding where to take the standard interface.

There may be several ways to demonstrate interoperability from the viewpoint of control & management for 5G mobile core network service, which is currently developed separately in Korea and Europe. The first method may have interoperability between MANOs of Korea and Europe separately. However, this method requires interworking between orchestrators, and the standard for this is not yet discussed by the International Organization for Standardization.

Within the MANO, interoperability management points are divided into three main categories. The first is interoperability between VIMs. It is a method to control heterogeneous VIM through one interface by unifying interfaces between different VIMs. However, it is not easy to unify the current VIM because it has slightly different characteristics depending on the type and version. The second method is to check interoperability through an interface between VNFM and vEPC





87 / 94

(VNF) and the last method is to use an interface between the orchestrator and VNFM. Both methods are in the process of standardization of the stage-3 protocol in conjunction with the 2018 PyeongChang Winter Olympics, the HSvEPC service was distributed in Daejeon and Gangneung using the DCNM system described above. There may be many ways to disperse. There is a distributed-DCNM method that installs an orchestrator, a function manager, and an Infra Manager in one package in Daejeon and Gangneung, respectively.

Another way to deploy is to place an orchestrator in one place and distribute the Function Manager and Infra Manager by region which is called single-orchestrator with distributed-manager distribution method. Since the separate protocol is needed for a single package in the distributed-DCNM, and the interworking between these orchestrators is still under standardization, we adopt the second method, single-orchestrator with distributed-manager, for the Olympics PoC.

There are also two ways of deploying HSvEPC. There is an integrated package method that integrates the entire functions of HSvEPC into one system and installs full set respectively. In another way, there is a distributed package method that distributes functions of HSvEPC and installs specific functions in the area interacting with each other. At this Olympics PoC, we installed integrated packages in each region and concentrated on mobile core functions in different regions to keep traffic localized. SOL WG of the ETFI NFV ISG, and standards have been established in the SOL-2 and SOL-3 WG, respectively.

In a distributed environment where the system is remotely separated, a method is required to minimize the interface between the central and distributed systems, thereby reducing the remote communication load and reducing the complexity. For this purpose, the interface between the orchestrator and VNFM is relatively smaller than that of the VNF and VNFM, so this PoC determines the SOL-3 as the interoperability test interface, which is the interface between the orchestrator and VNFM,

5G Mobile Core Management Interoperability

Distributed vEPC deployment compatible to ETSI NFV SOL003 interface

Orchestrated by Global 5GCHAMPION Orchestrator in Daejeon ETRI

End-to-End service conformance between KR & EU vEPCs





88 / 94

Figure 46. Management Interoperability interface definition

In this PoC, we showed one orchestrator located in Daejeon installed Korea (HSvEPC) in Gangneung ICT center and installed European vEPC in Daejeon to prove interoperability between them. As shown in the figure below, we conducted lifecycle management over vEPCs in Gangneung and Daejeon respectively through the SOL-3 interface.





89 / 94

<SOL-3 interfaces for interoperability>

Figure 47. Deployment of vEPCs for management interoperability test

4.3.2 NFVO-VIM-based management interoperability (EU)

To ensure efficient resource management and to showcase VNF LCM utilizing the NFVO-VIM, the interoperability of two different EPC VNFs is demonstrated. A centralized MANO controls the lifecycle of VMs in different VIM and NFVI regardless of wherever the VIM and its underlying hardware exists. The VNF instantiation and termination is showcased in the evaluation section of this document. It also ensures the interoperability of different VIM vendors with the ETSI based MANO platforms. Apart from showcasing dynamic orchestration of EPC VNFs in different VIM, the scope this test is to showcase the importance of high level interoperability between VIM and NFVO/VNFM observing the vi-vnfm/or-vnfm and ve-vnfm-vnf reference points.





90 / 94

Figure 48. NFVO-VIM based multi vEPC deployment

To showcase the interoperability, an Open Source Orchestrator and VNF manager is utilized at a centralized location and VNFs of different EPC solutions are instantiated and terminated. We have managed to test the lifecycle management of a multi-site deployment using the NFVO-VNFM interface. The lifecycle management primarily consists of functions set enabling create and delete identifiers [8]. Following are the VNF identifiers and the observed interfaces:





91 / 94

Observed reference points for interoperability

The multi-site deployment from a centralized orchestrator for two vEPC was carried out while executing VNF lifecycle events mentioned below:

4.3.3 Multi-site deployment and interoperability: state of the art and Lessons Learned

The multi-site deployments as targeted on the EU side was successfully achieved during the 2nd NFV Plugtests as well [9]. During the Plugtests, nine tests were executed with different combinations and the outcomes of the interoperability was approximated to be 100% [9]. The NFVO-VIM based interoperability in our case also turned out to be successful in multi- site deployment scenario. The Open source MANO is interoperable with both EPCs used and a single descriptor could be used to launch several instances of VNFs in different VIM. The testing scenario was limited to basic lifecycle events, but it would be interesting to check further interoperability with advanced test cases using the target interfaces. The tests could be further extended to study performance metrics of virtualized resources required for an NS instance. Moreover, utilizing the fault management system there could be some tests to verify that a fault alarm triggers upon failure of network connectivity due to non-availability of virtualized resources. It was observed in such interoperability during the tests that the version of the VIM and the MANO has potential impacts. The tests when executed with older versions of VIM and MANO resulted in non-compliance issues. Moreover, the choice of VIM was very limited on the MANO used but there are other potential solutions which supports variety of VIMs. As an example, OSM MANO supports VMWare, Amazon AWS, Cloudsim, Open VIM. There have been advancements also with the tacker project





92 / 94

of Openstack where kubernetes is supported as a VIM for containerized network functions in addition to Openstack

With single orchestrator we have shown through the PoC that two different vEPC packages from both KR and EU could be deployed at different POPs. That interprets the standardized APIs between the orchestrator and the VNFM enables installation and lifecycle management of different types of 5G mobile cores. These APIs are the result of a wide industry consensus. Compliance to them permits a wide range of multi-vendor deployment scenarios. For example, a VNF can be managed by a generic VNF Manager function, an NFVO can consume the service of a VNF-specific VNFM, and the services exposed by an NFVO can be consumed by higher-level service orchestration functions.

In other words, even if you use any kind of VIM if you use only the API between the orchestrator and VNFM in each POPs, it shows possibility to perform the following management functions.

• vEPC Lifecycle Management interface (as produced by the VNFM towards the NFVO).

• vEPC Performance Management interface (as produced by the VNFM towards the NFVO).

• vEPC Fault Management interface (as produced by the VNFM towards the NFVO).

• vEPC Indicator interface (as produced by the VNFM towards the NFVO).

• vEPC Lifecycle Operation Granting interface (as produced by the NFVO towards the VNFM).

• vEPC Package Management interface (as produced by the NFVO towards the VNFM).

• Virtualized Resources Quota Available Notification interface (as produced by the NFVO towards the VNFM).





93 / 94

5 Conclusion

This deliverable provided the final outcomes of WP4. Through the demoed PoCs, and the wide range of functional and performance evaluations, SDN/NFV-driven mobile core networks were validated to provide the necessary flexibility and on-demand performance necessary for 5G networks. Both real-time video and interactive augmented reality services proved that the targeted interoperability can even fulfil the most demanding services. Interoperability on the Management and Orchestration (MANO) level proves to be an ongoing challenge on the standardization level. For that reason, we evaluated two different options: i) the re-use of the interface between the orchestrator (NFVO) and VNFM as prescribed by the stage-3 protocol in the SOL WG of the ETFI NFV ISG, and ii) the use of the interface between the NFVO (i.e., OpenBaton) and the VIM (OpenStack). Both have their advantages and disadvantages, and need further consolidation on the standardization and (potentially open-source) software level. Particularly in the second option, the dependency on the version of the VIM (i.e., OpenStack) might become a bottleneck for the sustainability of this interoperability option.





94 / 94

6 References

[1]. 5G Test Network Finland. http://5gtnf.fi [2]. 5G Test Network. https://5gtn.fi [3]. OpenBaton. http://openbaton.github.io/documentation/ [4]. k. B. Long, T.-X. Do and Y. Kim, "Policy-based clustering service for network function

virtualization over multi-site clouds," in Information and Communication Technology Convergence (ICTC), 2017 International Conference, Jeju, South Korea, 2017

[5]. OpenEPC. http://www.openepc.com/ [6]. OpenEPC. http://www.openepc.com/decentralized-core-networks/ [7]. OpenAirInterface. http://www.openairinterface.org/?page_id=864 [8]. ETSI GS NFV-TST 002 V1 [9]. ETSI Plugtests reports

deliverable d4.3 deployment and interoperability … final.pdf5g communication with a heterogeneous,...

Documents