5g-miedge · 2019. 9. 26. · 2 5g-miedge page 2 the document is proprietary of the 5g-miedge...

Deliverable Horizon2020 EUJ-01-2016 723171 5G-MiEdge D4.3

Date : July 2019 Public Deliverable

5G-MiEdge

5G-MiEdge Millimeter-wave Edge Cloud as an Enabler for 5G Ecosystem

EU Contract No. EUJ-01-2016-723171

Contractual date: M36

Actual date: M36

Authors: See list

Work package: D4.3 MiEdge field trials integrated in 5G-Berlin Testbed toward Tokyo

Olympic 2020

Security: Public

Nature: Report

Version: 1.0

Number of pages: 51

Abstract

This deliverable reports on feasibility and availability of all key 5G MiEdge network components and

algorithms and their measured performance after implementation.

Keywords

millimeter-wave, millimeter-wave access, millimeter-wave backhaul, edge cloud, edge computing, liquid

ran c-plane, user/application centric orchestration, sdn, software-defined network, multi-rat

All rights reserved.

2

5G-MiEdge

The document is proprietary of the 5G-MiEdge consortium members. No copy or distribution, in

any form or by any means, is allowed without the prior written agreement of the owner of the

property rights.

This document reflects only the authors’ view. The European Community is not liable for any use

hat may be made of the information contained herein.

Authors

Fraunhofer Heinrich Hertz

Institute

Konstantin Koslowski

Thomas Haustein

Julian Daube

[email protected]

[email protected]

[email protected]

Tokyo Institute of Technology Kei Sakaguchi

Gia Khanh Tran

Hiroaki Nishiuchi

Makoto Nakamura

Yu Tao

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

Intel Valerio Frascolla [email protected]

mailto:[email protected]





Date : July 2019 Public Deliverable

5G-MiEdge

Table of contents

Abbreviations .................................................................................................................. 5

Executive Summary ........................................................................................................ 7

1 Introduction ............................................................................................................. 8

2 Testbed Architecture and Selected Scenarios ....................................................... 9

2.1 Testbed Architecture ......................................................................................... 9

2.2 Selected scenarios for evaluation ..................................................................... 9

2.2.1 Dynamic Backhaul Reconfiguration .................................................... 9

2.2.2 Dynamic Crowd.................................................................................. 12

2.2.3 Automated Driving ............................................................................. 12

2.3 Targeted Features and Details of Scenario-specific implementation ............. 13

2.3.1 Dynamic Backhaul Reconfiguration .................................................. 13

2.3.2 Dynamic Crowd.................................................................................. 14

2.3.3 Autonomous Driving .......................................................................... 15

3 Deployment of Testbed and Operational Testing ............................................... 17

3.1 Dynamic Backhaul Reconfiguration .............................................................. 17

3.2 Dynamic Crowd ............................................................................................. 18

3.2.1 Outdoor testbed architecture............................................................... 20

3.2.2 Definition of KPIs .............................................................................. 22

3.2.3 Results ................................................................................................ 23

3.3 Autonomous Driving ...................................................................................... 24

4 Measurements and Evaluation Results ............................................................... 27

4.1 Dynamic Backhaul Reconfiguration .............................................................. 27

4.1.1 Indoor testbed architecture ................................................................. 27

4.1.2 Definition of KPIs .............................................................................. 28

4.2 Dynamic Crowd ............................................................................................. 37

4.2.1 Results on Prefetching ........................................................................ 37

4.2.2 Results on Route Multiplexing ........................................................... 39

4.2.3 Results on Integration with Internet and Wi-Fi .................................. 40

4.2.4 Results on Multiple UEs ..................................................................... 41

4.2.5 Results on Blocking mmWave access ................................................ 42

4

5G-MiEdge

4.3 Autonomous Driving ...................................................................................... 44

4.4 Integration of 5G!Pagoda and 5G-MiEdge testbeds ...................................... 45

5 Final Demo at EuCNC .......................................................................................... 47

6 Summary ................................................................................................................ 50

7 References .............................................................................................................. 51

5

5G-MiEdge

Abbreviations

Acronym Description

AP Access Point

BS Base Station

C plane Control Plane

EC Edge Computing

EU European Union

HD High Definition

IT Information Technologies

JP Japan

KPI Key Performance Indicator

LiDAR Light Detection and Ranging

LOS Line of sight

MBS Macro base station

MEC Multi-access Edge Computing

mmWave Millimeter-wave

OBU On-Board Unit

ODC Outdoor Dynamic Crowd

OVS Open vSwitch

RAN Radio Access Network

RAT Radio Access Technology

RF Radio frequency

RRM Radio Resource Management

RSSI Received Signal Strength Indicator

RSU Road-Side Unit

SC-BS Small Cell Base Station

SC Small Cell

SDN Software-Defined Network

SOCRA Software-Defined Small Cell Radio Access Network

UE User Equipment

U-plane User Plane

V2V Vehicle to Vehicle

6

5G-MiEdge

V2X Vehicle to Everything

WP Work Package


Date: July 2019

Public Deliverable

5G-MiEdge

Executive Summary

The EU-Japan funded research project 5G-MiEdge (Millimeter-wave Edge cloud as

an enabler for 5G ecosystem) envisions to design, develop and demonstrate an

innovative 5G architecture, which manages to smartly and smoothly combine

millimeter-wave (mmWave) access/backhauling with edge computing (EC). This

document is the final deliverable in WP4, which runs from month 7 to 36 of the 36-

month lifetime of the project.

WP4 is dedicated to the evaluation of the 5G system performance enhanced by the

MiEdge concepts using system level simulation tools and real world field-tests in the

5G Berlin and Tokyo testbeds for indoor and outdoor. The work includes numerical

system simulations on high performance computing clusters to capture system relevant

KPIs and effects in repeated and controlled modelling environments representing

typical use cases under investigation. After all necessary steps of development,

integration and testing of the key MiEdge network components are completed; we

conducted field trials for specific use cases and scenarios, which were previously

evaluated by system level simulations. The Over-the-Air tests were conducted in the

5G-MiEdge testbeds in Berlin and Tokyo under real world environment as feasibility

proof of concept and evaluation of the key achievements of the 5G MiEdge project.

Joint demonstrations between EU and Japan are the central scope of WP4.

This deliverable begins with an introduction of the used architecture, the selected

scenarios and the features, which are to be evaluated. Then the deployment and

operation testing of the testbeds is explained, before presenting the measurement

results.

Another section describes our final presentation on the EuCNC 2019 in Valencia,

Spain and virtual joint measurements with the 5G-Pagoda! Project.


Date: July 2019

Public Deliverable

5G-MiEdge

1 Introduction

This deliverable is the final release of WP4 and describes the outcome of the work

done in Task 4.2 ‘Development of common/joint 5G MiEdge Testbed’ and Task 4.3 ‘:

Field trials toward Tokyo Olympic 2020’. While Task 4.2 was already detailed in the

previous deliverable D4.2 [D4.2], there are some notable changes related to the main

subject of this deliverable, mainly worked on by Task 4.3.

When the project was planned in 2016, we wanted to put our focus on the Tokyo

Olympics 2020, but this focus shifted during the project runtime. We still cover related

scenarios like the stadium and demonstrate that the key enabling technologies

developed in this project are ready for large events like the Olympics, but also general

5G deployments like the outdoor dynamic crowd. Lately the autonomous driving

scenario gained a lot of interest, from researchers as well as the automotive industry

and especially when combining mmWave and EC elements to achieve low latency and

high data rate Vehicle-to-Vehicle (V2V) and Vehicle-to-Everything (V2X)

communication.

In Section 2 “Testbed Architecture and Selected Scenarios” starts describing the

architecture proposed by the project, the selected scenarios and the features targeted

for evaluation, in order to provide a detailed evaluation of the outdoor capabilities of

the 5G-MiEdge testbeds in Berlin and Tokyo.

The following “Section 3 Deployment of Testbed and Operational Testing”, the exact

setup for evaluation of each of the selected scenarios is detailed.

Section 4 “Measurements and Evaluation Results” presents the measured results and

explains the difference between our baseline setup and the evaluated scenarios.

Moreover, an analysis is provided about when it is advantageous using a dedicated

network slice over the public network for intercontinental communication.

Section 5 “Final Demo at EuCNC” displays details of our final presentation at the

EuCNC 2019 in Valencia, Spain.

Finally, Section 6 “Summary“ concludes this deliverable.


Date: July 2019

Public Deliverable

5G-MiEdge

2 Testbed Architecture and Selected Scenarios

In deliverable D4.2 [D4.2], we introduced the 5G-MiEdge testbeds, which consist

mostly of off-the-shelf hardware that we enhanced with specifically developed custom

components and runs open source software, which we adapted to fit the project

requirements. Based on the architecture proposed by the 5G-MiEdge project, we

created a viable testing environment for the novel developed ideas and concepts.

The testbed architecture can be adapted to support our introduced use-cases and

scenarios, verify the simulation results and highlight the improvements over current

state of the art deployments.

2.1 Testbed Architecture

Figure 2-1 shows the testbed architecture used for outdoor deployment and to evaluate

the combination of mmWave backhaul, mmWave access and EC. The following

sections will explain selected scenarios that we evaluated on the testbeds, with the

corresponding architecture and achieved results.

2.2 Selected scenarios for evaluation

2.2.1 Dynamic Backhaul Reconfiguration

The first evaluated scenario is the dynamic reconfiguration of the mmWave backhaul

links. For this, we prepared the testbed with the architecture shown in Figure 2-2.

Figure 2-1 Application scenario of the testbed


Date: July 2019

Public Deliverable

5G-MiEdge

We already presented a part of the content of this in deliverable D4.2 [D4.2] and in the

paper presented at the BMSB2018 conference [Koslowski2018], but at that point in

time the evaluation was still ongoing and this Deliverable finally contains results that

are more detailed, as published in the Hindawi “Wireless Communications and Mobile

Computing” Journal [Santos2019].

The dynamic backhaul reconfiguration builds a foundation for almost all defined use-

cases and scenarios of the 5G-MiEdge project. With a constantly moving center of

attention in the stadium and a high number of HD cameras generating enormous

amounts of traffic, for the outdoor dynamic crowd that moves throughout a city

requiring high-speed connectivity and for temporarily deployed small cells for events.

With the central orchestration, a dynamic backhaul reconfiguration enables new levels

of flexibility and network optimization. It can quickly detect network failures, switch

to other routes and even repair the topology by re-aligning the links towards other

nodes.

For this, we use the SOCRA (Software-Defined Small Cell Radio Access Network)

architecture, shown in Figure 2-3. It consists of a northbound API, the orchestrator

interface, receiving reconfiguration commands from external applications, the internal

Figure 2-2 Architecture for evaluation of Dynamic Backhaul Reconfiguration


Date: July 2019

Public Deliverable

5G-MiEdge

SDN controller modules, that handle the SDN functionality and the southbound API

to relay commands to the connected nodes.

SOCRA uses a split-plane network design with an out-of-band control channel, the

liquid RAN C-plane.

For evaluation, we used four nodes and placed them throughout the Fraunhofer HHI

lab in Berlin. The nodes were commercially available minicomputers to support the

high throughput rates. We used different brands and generations:

Class 1: Intel NUC Kit D54250WYK (Intel i5 4250U, 16GB RAM, SSD)

Class 2: Intel NUC Kit NUC7i7BNH (Intel Core i7 7567U, 16GB RAM, SSD)

Class 3: Gigabyte BRIX GB-BKi7HA-7500 (Intel Core i7 7500U, 16GB RAM,

SSD)

Nodes N1, N3, N4 are class 2 devices, while node N2 is a Class 1 device. The SDN

controller operates on a dedicated Class 1 device and is connected to the control

network. Both receivers are Class 3 devices and the sender is a Class 1 device.

For orchestration of the tests, we use an additional laptop that is connected to the

control network and uses a REST client application to communicate with the SDN

controller. All devices run on Ubuntu 16.04 with the 4.15.0-34 kernel.

Figure 2-3 SOCRA architecture


Date: July 2019

Public Deliverable

5G-MiEdge

2.2.2 Dynamic Crowd

This section describes demonstration experiments assuming an outdoor dynamic cloud

scenario. Figure 2-4 shows the architecture for evaluation. This testbed has a

combination of the millimeter-wave mesh backhaul in the previous section and the

edge computing function. The orchestrator obtains the movement information of each

user via the microcell base station, and based on that, the SDN controller instructs

migrating the virtual application server to a node (edge) near the user. This allows each

user to receive service with high speed and low latency anywhere in the network. In

this demonstration experiment, they were verified whether the system of migrating

virtual server dynamically operates normally, how much the data rate and delay

improve by comparing the cloud with the edge, and how long it takes to migrate the

virtual server.

2.2.3 Automated Driving

In this section, the testbed structure of the selected EC scenario, i.e., a context-based

dynamic map reception/distribution system with EC, mmWave and SDN technologies

is introduced. Figure 2-5 shows the overall system architecture.

The real-time HD dynamic map of driving environment is a key enabler for safe

automated driving. With dynamic maps measured by 2D or 3D LiDARs, vehicles can

detect and avoid obstacles, navigate and localize moving obstacles in unknown

environments. Moreover, via enabling vehicles to share their local map data during the

cooperative perception [Fukatsu2019] through bi-directional communications, the

driving safety can be improved in a tangible manner.

Figure 2-4 Architecture for evaluation of Dynamic Crowd


Date: July 2019

Public Deliverable

5G-MiEdge

The proposed system aims to provide a safer and more reliable driving environment

for automated vehicles based on EC. In the testbeds, mmWave V2X is utilized for

LiDAR data transmission among EC host and vehicle OBUs, and the SDN technology

is introduced to help manage the mmWave V2X links with a global network view, so

that a better coordination of communication among EC host and vehicle OBUs can be

performed in this network.

2.3 Targeted Features and Details of Scenario-specific

implementation

2.3.1 Dynamic Backhaul Reconfiguration

With the evaluation of dynamic backhaul reconfiguration, we aim to verify the

functionality of the chosen hardware, selected software, implemented algorithms and

the interworking of all those components. We plan to repeat the functional tests to

ensure consistent results. As this feature is one of the main components of both testbeds,

we need to make sure there are no hidden problems.

The features targeted in this evaluation are

Baseline backhaul link reconfiguration

Optimal Steerable mmWave Mesh Backhaul Reconfiguration

Adaptive On/Off Mesh Backhaul Operation

In the Baseline backhaul link reconfiguration, we make sure that the interworking of

all components is accurate, reliable, with a low constant load on the network, and rather

simple configurations.

Afterwards, in the Optimal Steerable mmWave Mesh Backhaul Reconfiguration, we

carefully stress the setup, increase the load on the network, create less ideal conditions

on the channels and evaluate what effects those conditions have on the setup.

Figure 2-5 A SDN based mmWave V2X architecture for safe automated driving


Date: July 2019

Public Deliverable

5G-MiEdge

Thanks to the final feature, Adaptive On/Off Mesh Backhaul Operation, we verify the

power on/off functionality of the nodes and the resulting reconfiguration of the

topology and routes.

All targeted features can be centrally orchestrated by the SDN controller.

2.3.2 Dynamic Crowd

Densification of Small Cell Base Stations (SC-BS) produces massive backhaul traffic

in the core network and employing optical fibers for dense SC-BS backhauling would

result in prohibitively high cost and practical difficulty in implementation. Wireless

backhaul may offer a scalable and cost-effective solution. With GHz bandwidth, the

gigabit data rate is practically achievable, a feature which solves the capacity problem

that exists in lower frequency backhaul systems. In mmWave networks, directional

links are commonly established to compensate for the high path loss and high

directivity paves the way for spatial reuse. Another advantage of mmWave networks

with meshed topology is their capability of flexible traffic routing, which is extremely

suitable for outdoor dynamic crowd, a typical scenario considered where traffic

distribution dynamically varies in both space and time.

On the other hand, as UE nowadays wants to experience services everywhere without

disruption and low latency even when moving, it is desirable that UE-specific

multimedia contents are located as close as possible to the UE via EC technology. The

goal of edge computing is to bring cloud-computing capabilities, including computing

and caching, at the edge of the mobile network, within the Radio Access Network

(RAN), in close proximity to mobile subscribers. Mobile edge applications run as

virtual machines on top of a virtualization infrastructure provided by the mobile edge

host (MEH). Merging mmWave and EC are also essential for other scenarios and

applications, i.e. Olympic Games, multimedia broadcasting services, moving hotspots,

outdoor dynamic crowd and automated driving, as discussed. This paper demonstrates

a content delivery system at the edge over an mmWave meshed backhaul.

The overall testbed systematic architecture is shown in Figure 2-6. It consists of four

pairs of mmWave backhaul links, four pairs of mmWave SC-BSs and UEs, an edge

server (or gateway) which is connected to a global application server, a SDN controller

with an out-of-band control channel, and an optimizer (orchestrator) who orchestrates

the overall network based on UE’s context information e.g. location, application

demands etc. Some tasks of the orchestrator include optimal backhaul route selection,

backhaul link channel allocation and beam selection, prefetching control etc.


Date: July 2019

Public Deliverable

5G-MiEdge

The control protocol for adopting the out-of-band control plane in our work is depicted

in Figure 2-7. Here, the macro base station’s carrier of 2GHz band is adopted as the

frequency band used for the control plane to guarantee a large coverage in order to

centrally manage context information necessary for the C-plane. In addition, in general,

the load of the control plane is sufficiently smaller than that of the U-plane, so narrow

band communications at 2GHz band is sufficient.

2.3.3 Autonomous Driving

The selected scenario incorporates three components:

SDN controller: The central unit, which monitors the network condition and achieves the management logic by creating mmWave links and configuring bi-communication flows of HD map on the components of data plane

OBU: Mobile data plane elements that are controllable by the SDN controller. They are equipped on the automated vehicles, which upload context information and receive control message from the controller to perform cooperative perception

Edge computing host in RSU: Stationary EC data plane elements that are controllable by the SDN controller. They are the infrastructures that are deployed at the intersections or corners (typically on street lamp or buildings). They have a bird view of the traffic condition, and can perform cooperative perception with the SDN OBU

In this selected edge computing scenario, an OBU connects to the backhaul network

of EC hosts in RSUs through mmWave V2X links established by SDN controller

according to its context information. For example, when an incoming OBU arrives at

the edge of a complicated intersection environment, a feasible method to ensure the

driving safety is to get local dynamic maps of those EC hosts in RSUs, which are

Figure 2-6 Systematic architecture of the testbed

Figure 2-7 Out-of-band C-plane for SDN control


Date: July 2019

Public Deliverable

5G-MiEdge

deployed near the intersection. To implement it, the SDN controller will first send

commands to the OBU for an mmWave V2X link with the closest EC host, and then

routes the data flows of maps from different EC hosts to the target OBU. The data

traffic travels across mmWave backhaul and V2X access links. In general, the received

local maps will be merged at the last hop of the transmission to generate a global

dynamic map rather than at each hop for the sake of latency reduction. Therefore, in

this case, the map merging is conducted at the nearest EC hosts. The merged map will

be sent to the OBU via mmWave V2X. Through this process, the mentioned OBU will

obtain a global dynamic map, with which it becomes capable of potential danger

awareness in the out-of-sight area, so that an accident-free driving decision can be

made in advance. This scenario will definitely be an illumination to promote the safety

of today’s automated driving from the perspective of both internal and external system

design.

As for the consideration of the control plane, LTE cellular networks, Wi-Fi hotspots

and WiMAX would be the best choice due to the adequate throughput, wide

communication range and high network densification.

Figure 2-8 shows the composition of functional components that enables our

architecture. Five of them are recognized as key enablers and described below in detail.

Context Manager

A module that enables uploading context information of OBUs to the SDN controller. It helps the controller to maintain a global image of the dynamic network topology. Besides, some safety strategies are made by the controller based on the context of OBUs, e.g. the controller will allow cooperative perception via mmWave V2X when the position of an OBU is close to an EC host.

Link Manager

Figure 2-8 Functional components of proposed architecture


Date: July 2019

Public Deliverable

5G-MiEdge

A module that is triggered by the SDN controller when HD map transmission needs the participation of mmWave V2X links. It creates communication between specific OBUs and EC hosts. It also includes some mechanism that monitor the channel status and perform link repair to ensure the link reliability.

Flow Manager

A module that is triggered by the SDN controller as soon as the link management is done. It guides the map data flow from the source to the destination by sending dynamic OpenFlow tables to specific OBUs and EC hosts in the data plane.

Cooperative perception

A module implemented by the 2D/3D LiDAR with specialized applications for map merging in EC host. It helps an OBU integrate the received local dynamic maps collected from multiple EC hosts to generate a global dynamic map.

Radio Resource Management (RRM)

A module that defines the resource allocation and management logic of our system. It allows the deployment of various RRM algorithms based on the demand for safe automated driving.

3 Deployment of Testbed and Operational Testing

3.1 Dynamic Backhaul Reconfiguration

For the evaluation of this feature, the nodes were deployed in the Fraunhofer HHI lab

in Berlin, as shown in Figure 3-1.

Figure 3-1 Node positions in testbed


Date: July 2019

Public Deliverable

5G-MiEdge

The link distance ranges from 2.4 to 3.8 meters, with unobstructed, direct line of sight

(LOS) links.

As a first step, we performed functional testing of the components, aligned the

interfaces and established links, verified the connectivity and made sure the hardware

was working reliably, also after a reboot or the next day.

The first actual evaluation was the conduction of power measurements. Those are

meant to evaluate the possible power savings when shutting down individual nodes

and reconfiguring the network to keep up with the bandwidth requirements. Power

measurement results are displayed in Table 3-1. This shows an idle power consumption

of approx. 8 Watt, a maximum power consumption of approx. 42 Watt when under full

load and adding another 2-3 Watt when transmitting on both mmWave interfaces for

class 2 nodes.

Power State Class 1 Class 2

Idle without mmWave 4.03 (± 0.22)W 8.01 (± 0.07)W

Idle with mmWave 7.42 (± 0.62)W 11.02 (± 0.13)W

Load without mmWave 17.74 (± 0.60)W 41.70 (± 0.79)W

Load with mmWave 20.12 (± 0.35)W 43.99 (± 1.08)W

Based on this, we believe that significant power saving can be achieved with central

orchestrated power control options for the nodes.

3.2 Dynamic Crowd

Using our developed testbed, an experiment campaign was conducted with preliminary

results to demonstrate the capabilities of the proposed architecture. First, small-scale

indoor test was conducted to confirm functionalities of each component of the testbed.

In the indoor test, since coverage of both U-plane and C-plane is not an issue, reflective

antenna was not used for the backhaul links and Wi-Fi is used as the out-of-band

channel of C-plane. After confirming all components work, the testbed was moved

outside to an about 5000m2 campus area for full-scale outdoor test. In the outdoor test,

due to coverage requirement, reflective antenna was attached to the WiGig USB

dongles for backhaul links and WiMAX was employed as the out-of-band C-plane

channel.

Table 3-1 Power consumption of mmWave nodes


Date: July 2019

Public Deliverable

5G-MiEdge

The full overview of the testbed in indoor environment is shown in Figure 3-2. Again,

for the backhaul links, we use the IEEE 802.11ad WiGig modules, which are produced

by Panasonic Inc., Japan. The modules are accompanied with the driver for Linux

kernel version 4.4.0-36-generic and WiGig supplicant. We configured the module with

MCS 9. The module can operate on the channel 2 and channel 3 among the four

channels of 802.11ad. The maximum value of supported MTU is 7912 bytes. In our

testbed, these modules are connected to a Gigabyte GBBKi7HA- 7500 mini-PC. The

mini-PCs run the Ubuntu 16.04 LTS distribution with Linux kernel version 4.4.0-36-

generic. Each mini-PC has 16GB RAM and 240GB SSD space with a 3.5GHz Intel

processor. For each link, the IEEE 802.11ad module in one side is configured to

operate in the access point (AP) mode, while the other is in the client mode.

We configured a 4 mmWave small cell node testbed topology in a laboratory

environment, where each node is equipped with 2 IEEE 802.11ad dongles. As Fig. 8

depicts, Node 1 (N1) is connected to Node 2 (N2) in channel 2 and to Node 3 (N3) in

channel 3. Node 4 (N4) is connected to N2 in channel 3 and to N3 in channel 2. It is

noted that orthogonal channels were used for adjacent backhaul links in indoor test to

avoid interference while the same channel is used for all backhaul links in outdoor test

due to the difference of link coverage and environment. According to the configured

rules, network traffic between N1 and N4 is forwarded using N2 (path 1, N1-N2-N4)

or N3 (path 2, N1-N3-N4) as next hop. The small cell nodes use Open vSwitch (OVS)

with version 2.7.0 as software switch. We added the IEEE 802.11ad interfaces as OVS

switch ports, and configured two internal interfaces in N1 and N4 to handle network

traffic. In addition, a node running a SDN controller was connected to the mmWave

nodes through a Wi-Fi interface (OF control channel). This controller is based on

OpenDaylight and is responsible for adding and configuring the flow tables of the

managed nodes. We used iperf to generate traffic from N1 to N4, sending 7882 byte

UDP packets during 10 s. Every conducted experiment was repeated 10 times. We used

the ping network utility to measure the round-trip time (RTT) from N4 to N1, as well

as the packet loss/success rates between these two nodes, by sending ICMP echo

packets every 10 ms. Link throughput was measured with tshark in N4, by capturing

network traffic from its mmWave interfaces. Given our testing topology, we confirmed

that the maximum achievable throughput for both 1 hop and 2 hops are about 1.5 Gbps

while the RTTs are 0.835 ms and 1.088 ms for 1 hop and 2 hops respectively.

Figure 3-2 Testbed in an indoor environment


Date: July 2019

Public Deliverable

5G-MiEdge

3.2.1 Outdoor testbed architecture

We deploy 4 nodes in Line-of-Sight environment with a rectangular topology, as

shown in Figure 3-3Error! Reference source not found.. The place is at a university

campus in Tokyo. The nodes are deployed so to have almost the same altitudes for

facilitating beam alignment of backhaul links. In addition, inter-distance between

nodes are about 70m.

The architecture of devices is shown in Figure 3-4. Recapping Sect. II, “Nighthawk

X10 R9000” of NETGEAR were used as APs. “TravelMate P648” of ACER was used

for UE. “GB-BKi7HA-7500” of GIGABYTE for packet transfer and edge computing

were used as mesh node PCs. SFP+ was used as extended M.2-PCIe adapter for

connecting PCs and access points via 10Gbps Ethernet (10GbE) cable. SFX power

supplies were used to activate 10GbE NIC. “WX03” mobile routers of NEC were used

as WiMAX control plane. The GPS receiver for positioning employed “BU-353S4” of

GLOBALSAT.

Figure 3-3 Node deployment in outdoor test


Date: July 2019

Public Deliverable

5G-MiEdge

For orchestrating the experiment, a graphical user interface was designed and

implemented inside the orchestrator PC. It is based on the open source software node-

red, which offers an access to a wide variety of functions, like interacting with the

SDN controller and displaying results in clearly designed and well-structured graphs

functionality without reinventing the wheel. It was configured to issue commands to

the SDN controller, like migrating an EC container from one node to another, and to

the UE, like connecting to a specific AP. Figure 3-5 shows the map view, containing the GPS positions of the available nodes

and the UE. Additionally, the location of the EC container is shown by two colons, at

this time it was running on node 2. The buttons above the map send commands to the

SDN controller to migrate the container to the corresponding node.

Figure 3-4 Hardware architecture in outdoor test


Date: July 2019

Public Deliverable

5G-MiEdge

The GUI also shows the current link performance of the mmWave link of the UE.

Figure 3-6 (a) shows the throughput, which is at roughly 250 Mbps at this time, just

streaming a single HD video. In Figure 3-6 (b) the current latency is displayed, for

the capture duration it showed some slight variations between 0.3 and 0.9 ms and in

average it was around 0.5 ms.

(a) Throughput (b) Latency

3.2.2 Definition of KPIs

The outdoor testbed is evaluated for the KPIs throughput & latency.

For the interface with SDN controller, the orchestrator can request to achieve full

topology information of the network, update each mesh node configuration, decide

migration of EC container to a specific mesh node (via the deploy container button).

Especially, it can even set the MCS of all backhaul links in adaptation to link budget

change due to e.g. rain attenuation. So we evaluate about mmWave backhaul link data

rate and mmWave backhaul link delay.

Assuming that UE is walking toward the upper right EC AP and the UE will start to

play a specific multimedia as it approaches the target AP. If the network can predict

the movement of the UE and migrate the content’s container (Docker) from the

gateway AP to the target AP in advance, the UE can watch the demanded multimedia

Figure 3-5 Map view in the GUI

Figure 3-6 mmWave link measurement on UE


Date: July 2019

Public Deliverable

5G-MiEdge

immediately as it approaches the target AP. Otherwise, the UE needs to play the

multimedia streaming from the Docker located at the gateway AP when it is in the

proximity of the target AP. Of course, for the latter case, the delay is longer since it

includes both backhaul and access delay, while for the former case, the delay is just

incurred by the WiGig access.

3.2.3 Results

To confirm that the mesh network works in even outdoor environment, we conducted

some basic performance test. Table 3-2 shows the backhaul link’s achievable data rate

when the basic MCS 9 is set.

Link Hop number Data rate (Gbps)

Node 1 ⇄ 3 1 1.5

Node 2 ⇄ 4 1 1.5

Node 1 ⇄ 3 ⇄ 4 2 1.5

Node 1 ⟶ 2 1 1.2

Node 2 ⟶ 1 1 1.4

Table 3-3 shows the duration of migrating a sample container with a size of about

5Gbyte over a backhaul link 1→3.

MCS Latency (s)

9 20

5 36

The end-to-end latency between receiver (end user) and EC container was measured

for both cases through the ping command and shown in Table 3-4. The results were

averaged over about 4000 trials.

Type Latency (ms)

w/o EC ≃4

w/ EC ≃0.5

Table 3-2 mmWave backhaul link data rate

Table 3-3 mmWave container migration duration

Table 3-4 End-to-end latency w/ and w/o EC container migration


Date: July 2019

Public Deliverable

5G-MiEdge

3.3 Autonomous Driving

In order to demonstrate the performances of the proposed scenario, i.e., the SDN-based

mmWave V2X network for safe automated driving via coordinating dynamic LiDAR

HD-map communication, we held a proof-of-concept field test in outdoor

environments, where we installed the OBU on a real vehicle, deployed two EC hosts

in RSUs and created a SDN-based network [Li2019].

The Figure 3-7 shows the architecture of the proof of concept, including one OBU,

two RSUs as well as the SDN controller on a remote server. State-of-the-art mmWave

and SDN technologies are applied into this architecture to show their good chemistry.

Moreover, the Figure 3-8 shows the pre-set typical driving scene and path of the OBU,

in which it is going to turn at a corner.

Figure 3-7 PoC system architecture

Figure 3-8 Cooperative perception performed by mmWave V2X


Date: July 2019

Public Deliverable

5G-MiEdge

According to the system design, the SDN-based cooperative perception will be

performed via mmWave V2X along the route of the OBU until it passes the corner.

The whole process is divided into five phases. In the first phase, the OBU just leaves

the starting point. There is no LiDAR data transmission since it is too far from any of

the EC hosts. During the second phase, the OBU enters the coverage of RSU1. The

SDN controller detects its position and triggers the cooperative perception by

connecting RSU1 with RSU2 via mmWave backhaul link, and connecting the OBU

with RSU1 via mmWave V2X. The local dynamic maps of RSU2 and RSU1

simultaneously transmit over the mmWave backhaul and V2X links to the OBU for

integration. Finally, a global dynamic map can be generated at the OBU. Afterwards,

in the third phase, the OBU leaves the coverage of RSU1 so SDN controller

disconnects their link to save power. In the fourth phase, the OBU enters the coverage

of RSU2, but this time, the cooperative perception only occurs between the OBU and

RSU2 via mmWave V2X because the OBU has already passed RSU1, of which the

local dynamic map makes no sense for driving safety. The last phase is quite similar

to the third, i.e., the mmWave V2X link with RSU2 is disconnected by SDN controller

for energy saving.

In order to support such a scenario, the mechanism, which defines the interactive

principles between the data plane and control plane, is described in Figure 3-9.

Through the registration of RSUs, the controller will get knowledge about them (e.g.,

their numbers, positions and mmWave AP information), so that it can prepare the

potential V2X connections for the OBU in advance. As soon as the cooperative

perception is triggered, the context-based flow tables will be assigned by SDN

controller to coordinate the map transmission on EC hosts. Then, the controller will

help the OBU create an mmWave link with the nearest EC host for map reception.

Figure 3-9 Interactive mechanism between data plane and control plane


Date: July 2019

Public Deliverable

5G-MiEdge

Device

Name

Relevant Parameters

Specifications Amount

VLP16 3D

LiDAR

Range: 100 m, Accuracy: +/- 3 cm,

Rotation rate: 5-20 Hz, 16 channels 3

NETGEAR R9000

Tri-Band: 60GHz (4600Mbps), 5GHz

(1733Mbps), 2.4GHz (800Mbps) with 4x

high-performance active antennas

2

mmWave

Antenna Lens model: 5067, Gain: 25.97dBi, Scan

angle: +/- 17deg (Hor.), +/- 4.5deg (Ver.) 2

Panasonic PC

Model: CF-SV OS: Ubuntu 16.04, OVS: 2.8.0, DB

Schema: 7.15.0 2

Acer PC

Model: N15C5 OS: 16.04, ROS: Kinetic, WiGig

Interface: IEEE 802.11, OVS: 2.8.0 1

Vehicle Speed: ≤ 20 km/h (Grey), 0 km/h (Blue) 2

Device

Name

Relevant Parameters

Specifications Amount

WiMAX

NIC

Downlink:120 Mbit/s, Uplink: 60 Mbit/s,

Maximum coverage: 30 miles 4

VAIO PC Model: S13 OS: Ubuntu 16.04, OpenDaylight: Beryllium 1

Table 3-1 Power consumption of mmWave nodes lists the necessary hardware for the

deployment of Data plane. The two 3D LiDARs on RSUs are used to generate HD

maps, and the one on OBU are used to run the Simultaneous Localization and Mapping

(SLAM). Because the NETGEAR router provides 60 GHz mmWave access, we use it

as mmWave AP for the V2X. To constitute the mmWave backhaul between two RSUs,

a pair of mmWave antennas is set up. Two note PCs which support OVS feature allow

the data exchange inside and outside. Another note PC is put on the OBU as it has a

WiGig interface to connect with the mmWave AP at 60 GHz. Figure 3-10 shows their

outdoor deployments. In Table 3-6, besides the PC working as a SDN controller,

WiMAX NICs are used to create a network, which allows the data plane elements

connect with SDN controller to achieve the control logic. Figure 3-11 shows the

hardware deployment of Control plane.

Table 3-5 Hardware for data plane

Table 3-6 Hardware for control plane


Date: July 2019

Public Deliverable

5G-MiEdge

4 Measurements and Evaluation Results

4.1 Dynamic Backhaul Reconfiguration

In this first series of tests, the goal was to test and evaluate the dynamic routing,

configuration and load balancing in the mmWave backhaul network.

4.1.1 Indoor testbed architecture

The used architecture is the same as displayed in Figure 2-2, consisting of 4 mmWave

mesh nodes, with one sender and two connected receivers. There are multiple possible

paths that can be aligned and configured for the traffic between sender and receivers.

When transmitting data from Sender to Receiver 1, the paths are:

1. Sender - N1 - N2 - N4 - Receiver 1


For data transmission between Sender and Receiver 2, the paths are:

1. Sender - N1 - N2 - Receiver 2

Figure 3-10 Outdoor experimental deployment of hardware: Data plane

Figure 3-11 Outdoor experimental deployment of hardware: Control plane


Date: July 2019

Public Deliverable

5G-MiEdge


With the installed power control units, the power state of N2 and N3 is centrally

managed to enable power savings and network optimization.

4.1.2 Definition of KPIs

In this series of tests, we aim to evaluate the following KPIs:

1. End-to-end latency

2. End-to-end throughput

3. End-to-end packet loss

4. Reconfiguration delay

5. Power consumption

To guarantee reproducible and accurate results, each test runs for several iterations.

Results

4.1.2.1 Baseline Backhaul Link Reconfiguration

For the baseline evaluation, targeting the reconfiguration delay KPI, we begin with

switching between multiple backhaul configurations.

In this experiment, node N1 is just using a single interface. It is initially aligned to

node N2, like shown in Figure 4-1. With this configuration, Sender and Receiver 1 can

communicate via the path Sender - N1 - N2 - N4 - Receiver 1, shown as a blue dashed

line, and exchange an 800 Mbps UDP flow. With initial measurements, we calculated

the necessary alignment positions and verified them by reading the RSSI on the links

N1 - N2 and N1 - N3.

After transmitting data in this initial configuration for a specified time, the link N1-N2

is disconnected and the interface in N1 is rotated by 70° to face node N3 and establish

a new link, as shown in Figure 4-2.

Figure 4-1 Initial configuration: N1 mmWave interface aligned with N2


Date: July 2019

Public Deliverable

5G-MiEdge

We repeated this experiment for 15 times. Figure 4-3 shows the cumulative distribution

function (CDF) for the different time intervals required to perform the configuration

operations and re-establish end-to-end connectivity between sender and receiver nodes.

These include the mechanical interface alignment from the initial N1-N2 position to

the final N1-N3 position, the internal link configuration by nodes N1 and N3, and the

detection of the newly established link N1-N3 by the SDN controller. To measure the

interface alignment time, we poll the mechanical movement controller periodically

until we receive a reply that indicates the movement has completed, the internal link

configuration is calculated by computing the maximum link configuration time

between AP and STA, ODL detects a new link when it receives an Link Layer

Discovery Protocol (LLDP) packet from one of the corresponding interfaces and lastly

the traffic interruption time is obtained by calculating the maximum time when there

are no packets exchanged between sender and receiver.

The resuts show a constant link alignment time of approx. 3.06 seconds, the internal

link configuration takes an average of 2.88 seconds, due to the used software that

Figure 4-2 Final configuration: N1 mmWave interface aligned with N3

Figure 4-3 CDF for link reconfiguration time


Date: July 2019

Public Deliverable

5G-MiEdge

requires a restart of the wigig_supplicant. After an average of 3.14 seconds, ODL

detects a new link.

These times combined, add up to a total traffic interruption of 6.5 seconds average, but

this is just the baseline configuration with a single interface on node N1, resembling

unexpected outages. Multiple interfaces per node and a redundant mesh topology will

dramatically decrease these timeouts by simply re-routing traffic.

4.1.2.2 Optimal Steerable mmWave Mesh Backhaul Reconfiguration

To implement any wireless backhaul reconfiguration use-cases, it is crucial that the

wireless backhaul can perform these reconfiguration operations with minimal

disruption on existing traffic. This leads to the question: What is the impact of topology

changes and channel assignment on active traffic? To answer this question, we

conducted a number of experiments with two different backhaul configurations C1 and

C2. As a side effect, these experiments validate the long-term functionality of the

hardware and implemented reconfiguration functions in the entire testbed.

Low Traffic Volume: 500 Mbps UDP

The first experiment in this section is using C1 and C2 as shown in Figure 4-4 and

Figure 4-5 and the traffic between sender and each receiver consists of 500 Mbps UDP.

This transition needs to happen when the controller e.g. detects long-term link

blockage on the link N1-N3 and has to re-route the traffic via node N3 instead.

In C1 all nodes are powered up, the sender is connected to N1 and the receivers to N2

and N4. The links N1-N2 and N2-N4 provide paths between sender and the two

receivers.

The second configuration C2 has the same basic topology, but the links N1-N3, N3-

N4 and N4-N2 provide paths between sender and the two receivers.

Figure 4-4 Low Traffic Volume: mmWave backhaul configuration C1


Date: July 2019

Public Deliverable

5G-MiEdge

Figure 4-6 shows the collected measurements of throughput and latency.

At t = 0, the initial setup is performed and configuration C1 is activated, at t = 10, the

measurements start and the throughput rises to the desired 500 Mbps on both paths,

while the latency is rather constant at approximately 1 ms, with a few visible outliers.

To avoid traffic disruptions, the alignment and link configuration of N1-N3 is

performed at t = 30 and t = 40, before updating the flow rules at t = 42. After a very

Figure 4-5 Low Traffic Volume: mmWave backhaul configuration C2

Figure 4-6 Low Traffic Volume: Throughput and Latency


Date: July 2019

Public Deliverable

5G-MiEdge

short timeout of approx. 50 ms, the data transmission between sender and receivers

continues, this is almost invisible in the Figure. While the latency between sender and

receiver R1 stays the same at approx. 1 ms, it rises to approx. 2 ms for receiver r2,

because now there are two intermediate hops. Table 3-1 shows the average and

standard deviation of throughput and latency for both receivers. There is no packet loss

in this experiment.

Events Host Throughput

(Mbps)

RTT (ms) Loss %

Start - Alignment R1 498.65 (±4.76) 1.23 (±0.37) 0.0

R2 498.44 (±4.73) 0.87 (±0.32) 0.0

Alignment - Path 2 config R1 499.82 (±0.34) 1.21 (±0.29) 0.0

R2 499.64 (±0.45) 0.87 (±0.29) 0.0

Path 2 config. - Fwrd. update R1 499.68 (±0.55) 1.25 (±0.37) 0.0

R2 499.55 (±0.46) 0.89 (±0.34) 0.0

Fwrd. update - End R1 499.51 (±0.83) 1.51 (±0.49) 0.0

R2 499.56 (±0.73) 1.70 (±0.57) 0.0

High Traffic Volume: 900 Mbps UDP

The second test is using a similar setup as the first, but the data throughput is increased

from 500 Mbps UDP to 900 Mbps UDP. This resembles situations in real-world

deployments when the network is fully loaded and additional nodes would be powered

on and integrated into the network.

For this, the first configuration C1 is similar to the first test, using the paths:

1. Sender – N1 – N2 – N4 – Receiver 1

2. Sender – N1 – N2 – Receiver 2

But the final configuration C2 changes slightly, as shown in Figure 4-7. Providing the

paths:

1. Sender – N1 – N3 – N4 – Receiver 1

2. Sender – N1 – N2 – Receiver 2

Table 4-1 Low Traffic: Throughput and Latency, average and standard deviation


Date: July 2019

Public Deliverable

5G-MiEdge

In-between C1 and C2, the links N1 – N3 and N3 – N4 are configured and the flow

rules update to re-route the first flow. For this experiment, the results of the bandwidth

of flows F1 and F2, along with the RTT between the server and both receivers, is shown

in Figure 4-8 and detailed in Table 4-2.

In these results, there are can observe interference between the links, causing

significant spikes in the measured latency, as well as packet loss between sender and

receivers.

Figure 4-7 High Traffic Volume: mmWave backhaul configuration C2

Figure 4-8 High Traffic Volume: Throughput and Latency


Date: July 2019

Public Deliverable

5G-MiEdge


(Mbps)

RTT (ms) Loss %


R2 802.52 (±96.63) 42.77 (±8.90) 10.44

Alignment - Path 2 config R1 725.87 (±95.86) 44.89 (±1.54) 19.35

R2 802.94 (±95.70) 44.73 (±1.63) 10.78

Path 2 config. - Fwrd. update R1 740.66 (±97.12) 44.95 (±1.55) 18.04

R2 796.17 (±96.82) 44.75 (±1.65) 11.76

Fwrd. update - End R1 898.33 (±12.55) 1.67 (±3.32) 0.24

R2 896.89 (±5.50) 1.16 (±4.17) 0.46

The situation normalizes when the flow rules for the second configuration C2 are

applied and the traffic for each receiver is using an individual path throughout the mesh

network. Not only does the throughput rise to almost the desired 900 Mbps UDP, the

latency between sender and the receivers also goes down to an average of approx. 1.2

and 1.7 ms.

Interference caused by suboptimal channel selection

To investigate the interference caused by the relatively high number of links for a small

deployment area and the impact of channel selection, this experiment sets all mmWave

links in a path to use the same channel:

N1 – N2, N2 – N4: Channel 2

N1 – N3, N3 – N4: Channel 3

Figure 4-9 and Table 4-3 show the results, and clearly display the high fluctuation in

latency and throughput caused by interference of the active links. Especially when

comparing the results with those of the previous section that actually use the same

positions and links for all nodes, just different channels.

Table 4-2 High Traffic: Throughput and Latency, average and standard deviation


Date: July 2019

Public Deliverable

5G-MiEdge

In addition to the high latency and low throughput, there is a packet loss of up to 75%,

rendering the resulting network almost unusable, at least when aiming for a satisfying

user experience.


(Mbps)

RTT (ms) Loss %


R2 445.69 (±131.86) 80.09 (±73.61) 49.82

Alignment - Path 2

config

R1 189.58 (±72.67) 291.36 (±244.63) 76.50

R2 479.71 (±132.65) 78.14 (±67.66) 45.87

Path 2 config. - Fwrd.

update

R1 213.28 (±87.62) 263.26 (±202.98) 75.34

R2 470.15 (±143.06) 81.75 (±69.47) 48.30

Fwrd. update - End R1 413.24 (±112.18) 139.11 (±129.10) 53.57

R2 890.01 (±27.12) 2.00 (±14.17) 1.33

Only after applying the flow rules for configuration C2, the values improve slightly,

for receiver 2 they even become acceptable, but still the impact of suboptimal channel

allocation is drastic and highlights the need to optimization strategies when deploying

a dense network.

Figure 4-9 Suboptimal channel selection: Throughput and Latency

Table 4-3 Suboptimal channel selection: Throughput and Latency, average and standard

deviation


Date: July 2019

Public Deliverable

5G-MiEdge

4.1.2.3 Adaptive On/Off Mesh Backhaul Operation

To evaluate the power consumption KPI, we used a simplified architecture. There is

just one receiver connected to N4, receiving data from the sender connected to N1. As

shown in Figure 4-10, the first configuration C1 only includes three active nodes N1,

N2, N4, while node N3 is powered off. The path between sender and receiver is Sender

– N1 – N2 – N4 – Receiver and we transmit an 800 Mbps UDP flow.

Figure 4-11 displays the second configuration C2, with node N2 being powered down

and N3 active. This results in the path Sender – N1 – N3 – N4 – Receiver.

For the evaluation, we switch between the two defined configurations and measure the

resulting throughput, latency, loss. To keep the actual interruption to a bare minimum,

the powered down node is powered up first, the interfaces aligned, the links established

before applying the flow rules to switch between the paths.

In Figure 4-12, the measured throughput and latency are displayed, as well as the

timestamps for configuration events.

Figure 4-10 Adaptive On/Off: mmWave backhaul configuration C1

Figure 4-11 Adaptive On/Off: mmWave backhaul configuration C2


Date: July 2019

Public Deliverable

5G-MiEdge

Table 4-4 shows the exact values for throughput and latency, their standard deviation

and the packet loss between sender and receiver.

Events Throughput (Mbps) RTT (ms) Loss %

Start – Alignment 799.12 (±4.90) 1.29 (±0.25) 0.0

Alignment - Path 2 config. 799.75 (±0.64) 1.28 (±0.23) 0.0

Path 2 config. - Fwrd. Update 799.04 (±2.38) 1.29 (±0.36) 0.0

Fwrd. update - Path 1 config. 799.77 (±0.86) 1.27 (±0.24) 0.0

Path 1 config. - Fwrd. Update 799.55 (±0.87) 1.29 (±0.29) 0.0

Fwrd. update - End 799.63 (±0.73) 1.36 (±1.87) 0.0

As shown in these results, the overall performance was consistent and despite the

installation of new flow rules, there was no loss in performance or data.

4.2 Dynamic Crowd

4.2.1 Results on Prefetching

The testbed introduced prefetching algorithm [Nakamura2019] is evaluated via

downloading time of real traffic data. Container migration was realized automatically

by inputting REST API command to SDN controller at the set time, which is based on

the scheduler registered beforehand by UE. The scheduler was determined based on

user moving speed, throughput of access and backhaul, container start-up time, etc.

which are shown in Table 4-5. Each MEH and UE input REST API commands to SDN

controller to implement the prefetching algorithm based on the scheduler.

Parameter Value

Figure 4-12 Adaptive On/Off: Throughput and Latency

Table 4-4 Adaptive On/Off: Throughput and Latency, average and standard deviation

Table 4-5 Simulation parameters for schedule creation


Date: July 2019

Public Deliverable

5G-MiEdge

Throughput of Access [bps] 1.0 × 109

Throughput of Backhaul [bps] 7.0 × 108

Data size of image [Byte] 2.0 × 109

User speed [m/s] 0.5

Inter-cell distance [m] 25

Container start-up time [s] 40

Waiting time for watching video [s] 20

The schedule of prefetching algorithm and experiment environments are shown in

Figure 4-13 and Figure 4-14 respectively. At first, UE stays at Node1 and connects to

the nearby AP using WiGig. After UE finished downloading 4K-Videos, UE waits a

few seconds for watching the movie. During then, each node starts container migration,

which includes a process of stopping container, transferring container image among

the MEHs and starting up container at another MEH. After watching the video, UE

starts moving from the source node to the destination node. The testbed was designed

to finish container migration just before UE arrives at the next node.

The measurement result is shown in Figure 4-15. In case of EC migration utilizing

prefetching algorithm, it was confirmed that the downloading time of 4K-Video did

not change even if the user moves to the other node. On the other hand, download time

increased as the number of hops increased when the content server is fixedly deployed

on Node 1, in other words, the without edge computing (and migration) case.

Especially, download time increased when user arrived at Node3 for this case. Packet

error rate increased due to multi-hop relay with the increase of re-transmission requests

might be the reason for the worsen phenomenon of w/o prefetching. Detailed

investigation is considered as our future works.

Figure 4-13 Schedule of prefetching algorithm based on the parameters

Figure 4-14 SC-BSs deployment and user movement


Date: July 2019

Public Deliverable

5G-MiEdge

4.2.2 Results on Route Multiplexing

In addition, the testbed is evaluated by backhaul route multiplexing. When intensive

traffic distribution is happened, the architecture applies several backhaul resources to

improve access throughput. By using SDN control, the architecture can realize

adaptive allocation of backhaul resources to follow dynamic variation of traffic

distribution.

In this experiment, EC deploys at node 1 and node 3 in according to Figure 4-16.

Maximum backhaul throughput is about 1 Gbps, and maximum access throughput is

about 2 Gbps, so there is bottleneck in case there is only one backhaul route available

like the left figure. There will be no bottleneck in case there are several backhaul routes

like the right figure. Thus, the effectiveness of mmWave access can be confirmed.

Figure 4-15 Downloading time of 2GB movie data with FTP

Figure 4-16 Architecture of backhaul route-multiplexing


Date: July 2019

Public Deliverable

5G-MiEdge

The measurement result is shown in Table 4-6. In case of “w/o Route-Multiplexing”,

route1 (EC – Node1 – Node3 – Node4 – UE) and route2 (EC – Node3 – Node4 – UE)

use the same backhaul route (Node3 – Node4). As mentioned above, there is bottleneck

in this case. The total throughput is 811 Mbps, which is not the maximum access

throughput but approximately the backhaul link’s throughput. In case of “w/ Route-

Multiplexing”, route1 (EC – Node1 – Node2 – Node4 – UE) and route2 (EC – Node3

– Node4 – UE) use different backhaul routes, therefore there are no bottleneck. The

total throughput achieves 1.56 Gbps, which is much higher than single backhaul

throughput and nearly the maximum access throughput. Comparing these two

scenarios, the total throughput of “w/ Route-Multiplexing” becomes 1.9 times as

higher as that of “w/o Route-Multiplexing”. Therefore, we confirmed the effectiveness

of backhaul route multiplexing using SDN control.

Method Route Throughput Total throughput

w/o Route-

Multiplexing

EC – N1 – N3 – N4 – UE 427 Mbps 811 Mbps

EC – N3 – N4 – UE 384 Mbps

w/ Route-

Multiplexing

EC – N1 – N2 – N4 – UE 816 Mbps 1.56 Gbps

EC – N3 – N4 – UE 743 Mbps

4.2.3 Results on Integration with Internet and Wi-Fi

In the next section, by changing the access method and the number of hops, we again

check the effectiveness of edge computing in mmWave cellular network. Figure 4-17

shows an illustration of the measurement patterns. Three among the four available

nodes, one container having iperf server, and one UE are used in this testbed. In 0-hop

pattern, the container is set in node4, which is the nearest position to UE. In 2-hop

pattern, the container is set in node1 and UE connects to the server via one access link

and two consecutive backhaul links. In the both patterns, two access methods (Wi-Fi,

WiGig) are used. In multi-hop pattern, there is no container in this setting. Node1 is

connected to the Internet and UE connects to an iperf server in Germany via multi-hop

links.

Figure 4-18 shows the results of throughput measurement. In the case of using Wi-Fi

access, throughputs are about 600Mbps with no hops and two hops. These results mean

even if EC is used, Wi-Fi become a bottleneck and it is impossible to achieve

throughput higher than 600 Mbps. On the other hand, in the case of using WiGig access

with two hops and multi-hops, throughputs attain about 900 Mbps and 30 Mbps

respectively. These mean even if WiGig is used, backhaul link becomes a bottleneck

and it is impossible to demonstrate original performance of WiGig. However, by

fetching server in GW node (node1), backhaul bottleneck issue can be resolved. Finally,

in the case of using WiGig access with no intermediate hops, the throughput achieves

the fastest speed of about 1.3 Gbps. This means WiGig can demonstrate its full

potential by setting server in the closest node to UE.

Table 4-6 Throughput of route-multiplexing with iperf3


Date: July 2019

Public Deliverable

5G-MiEdge

4.2.4 Results on Multiple UEs

Then, we demonstrate the benefits of deploying multiple containers in environments

with multiple UEs. Figure 4-19 shows measurement configuration of multi-UE and

multi-container scenario. In addition, Table 4-7 shows throughput measurement results.

mmWave WiGig is used as access link and two UEs connect to node1 and node2

respectively. In case of single container, the container is set in node1. UE1

communicates to the server directly via WiGig access. UE2 simultaneously

communicates to the same server via WiGig access and one backhaul link. Normally

in single-UE scenario, UE1 has throughput of about 1.3 Gbps and UE2 has throughput

of about 900 Mbps. However, in this multi-UE experiment, the throughputs of the UEs

Figure 4-17 Measurement patterns

Figure 4-18 Results of throughput measurement with iperf


Date: July 2019

Public Deliverable

5G-MiEdge

are reduced to about 900 Mbps and 350 Mbps respectively when two UEs

simultaneously access to the server through different routes. On the other hand, in

multi-container scenario, two containers are deployed in node1 and node2 in a

distributed manner. Two UEs connect to the containers directly via WiGig access.

Thereby, the throughputs can achieve higher value compared to the multi-UE scenario

since the effect of backhaul bottleneck and routing degradation can be resolved.

Container pattern UE index Throughput [Gbps]

Single 1 0.90

2 0.35

Multi 1 1.22

2 1.15

4.2.5 Results on blocking mmWave access

Blocking effect on backhaul link can be avoided by reconstructing route or migrating

EC server to edge node in our network configuration. A situation when a human body

blocks mmWave access link is examined in this part. Netgear R9000 Nighthawk

wireless router we use in this examination has a function of beamforming so that

optimal path can be always formed. Figure 4-20 shows measurement environment in

this blocking situation. There are a node with mmWave access point (AP), an UE and

a human by a concrete wall. The UE and the mmWave AP are set at a height of 1m,

the distance between them is 5m. The wall is set in parallel and 1m away from them.

In this environment, the UE downloads data from the mmWave node.

Figure 4-19 Measurement configuration of multi UEs and multi containers

Table 4-7 Throughputs of multi UEs and multi containers


Date: July 2019

Public Deliverable

5G-MiEdge

Figure 4-21 shows the fluctuation of throughput. At first, the throughput has high value

because direct beam path is formed. A few seconds later, the human blocks the path

and the throughput drops sharply. However, by re-adjusting the beam according to BTI

(beam transmission interval), a path using wall reflection is selected, and throughput

returns to the high value. Namely, from this result, blocking mmWave access is not an

issue in environment with walls.

Figure 4-20 Measurement environment in blocking situation

Figure 4-21 Fluctuation of throughput in blocking situation


Date: July 2019

Public Deliverable

5G-MiEdge

4.3 Autonomous Driving

We will give an illustration about how safety this system can provide from the GUI,

which presents the received and merged dynamic HD maps on the EC hosts. As shown

in Figure 4-22 (a), in Phase 1, only OBU map (green points) appears, because the

cooperative perception has not been performed since it is out of the coverage of

mmWave V2X AP. Nevertheless, after the OBU enters the coverage of RSU1, as

shown in Figure 4-22 (b), relying on the cooperative perception with RSU1 local map

(red points) and RSU2 local map (yellow points), its vision is extended to the

surrounding area of the corner. At the same time, an incoming vehicle, which is pointed

by an arrow in Figure 4-22 (b), in the blind spot is detected so that the OBU can take

measures to avoid collision in advance. From Phase 2 in Figure 4-22 (b) to Phase 4 in

Figure 4-22 (c), due to the dynamic topology, the corner environment may have

changed. So cooperative perception performed again with RSU2 local map (yellow

points) to update the global maps.

The end-to-end throughput between the OBU and RSU2 was measured because this

parameter features remote and huge data transmission at the same time. We test the

performances of 2.4 GHz and 60 GHz as candidates of V2X respectively and compare

their performance. As Fig.21 shows, in Phase 1, 3, 5, the V2X communication was not

established because the OBU was moving out of the coverage. When the cooperative

perception was first triggered in Phase 2, the 60 GHz, i.e., mmWave V2X can quickly

provide a data rate of near 1 Gbps while the achievable data rate of 2.4 GHz V2X was

only 90-100 Mbps. In Phase 4, cooperative perception was performed again as SDN

controller detected the OBU was within the range of RSU2. Although there was no

map transmission through the backhaul links because the RSU1 did not engage, the

throughput was almost the same as Phase 1.

a) Phase 1 b) Phase 2 c) Phase 4

Figure 4-22 GUI of received dynamic LiDAR maps


Date: July 2019

Public Deliverable

5G-MiEdge

4.4 Integration of 5G!Pagoda and 5G-MiEdge testbeds

5G!Pagoda is a European/Japanese research project under the H2020 call, like 5G-

MiEdge is. Throughout the projects, we had continued discussions between the

members of both consortiums and planned to investigate possible benefits when

combining our SDN orchestrated mmWave edge computing with their SDN

orchestrated network function virtualization (NFV) [D1.4].

To summarize our discussions, we will show some measurements of the virtual

integration of 5G-MiEdge and 5G!Pagoda networks in Figure 4-24. Wired SDN

constructed by 5G!Pagoda decreases the latency from 400ms to 175ms, and wireless

SDN constructed by 5G-MiEdge decreases it from 4ms to 0.5ms (in a local domestic

network) as shown in Tab. 4-8. Therefore, we confirmed the effectiveness of

wired/wireless SDN and edge computing.

Figure 4-23 comparison of the network performance by using V2X of different frequencies

(2.4/60 GHz) during the PoC


Date: July 2019

Public Deliverable

5G-MiEdge

Method Latency [ms]

Access to foreign server via general network 400~500

Access to foreign server via 5G!Pagoda network 175

Access to local domestic server w/o 5G-MiEdge’s EC 4

Access to local domestic server w/ 5G-MiEdge’s EC 0.5 [Table 3-4]

Figure 4-24 Integration test of 5G-MiEdge and 5G!Pagoda networks

Table 4-8 Comparison of 5G-MiEdge and 5G!Pagoda integration network by latency


Date: July 2019

Public Deliverable

5G-MiEdge

5 Final Demo at EuCNC

As this is the final year of the project, we went to the EuCNC2019 in Valencia, Spain

to exhibit our testbed hardware in a final demo, displayed in Figure 5-1.

Due to the space and transportation constraints, we went for a slightly reduced

architecture, as shown in Figure 5-2.

This final demo is using three mmWave small cells, as shown in Figure 5-3, equipped

with two movable mmWave backhaul interfaces in the top compartments, a

computation device and with nodes N1 and N2 being also equipped with multi-RAT

mmWave/sub 6 GHz access in the lower compartment.

Figure 5-1 Final Demo at EuCNC2019

Figure 5-2 Architecture at EuCNC2019


Date: July 2019

Public Deliverable

5G-MiEdge

The UE is represented by a laptop that connects to the APs via multi-RAT

mmWave/sub 6 GHz and requests data from the EC hosts.

We implemented parts of all previous evaluations in this final demonstration to

summarize our efforts and give an overview of the implemented functionality.

For this demo, we chose a static backhaul topology with three links:

1. N1 – N2

2. N1 – N3

3. N2 – N3

The links were always aligned and connected. The SDN controller, however, was able

to re-route the traffic between these links, depending on the link quality, indicating

obstructions or completely blocked links.

The laptop was constantly requesting content from an edge computing application

running on node N3. In the case of mmWave access link blockage, the multi-RAT

functionality seamlessly switched to the sub 6 GHz access link, resulting in slightly

higher latency and lower throughput, but still ensuring connectivity to the service. For

link failures longer than 3 seconds, the SDN controller orchestrated the laptop to

switch to an AP running on a different node and restore low-latency connectivity at

full speed.

This resulted in four possible routes that the SDN controller automatically selected:

1. UE – N1 – N3

2. UE – N1 – N2 – N3

3. UE – N2 – N3

4. UE – N2 – N1 – N3

Figure 5-3 5G-MiEdge node used at EuCNC 2019


Date: July 2019

Public Deliverable

5G-MiEdge

After running for four days, we were quite satisfied with the performance and

reliability. Despite blocked links in backhaul and access, the connectivity between UE

and EC application was quickly restored, providing a high end-to-end link reliability

and quality of service.

The conference was one of this year highlights and we had many interesting

discussions with visitors and researchers from other projects!


Date: July 2019

Public Deliverable

5G-MiEdge

6 Summary

This deliverable presented the outcome of Task 4.2 ‘Development of common/joint

5G MiEdge Testbed’ and Task 4.3 ‘: Field trials toward Tokyo Olympic 2020’. It is the

final document in WP4 and marks the end of this project.

We conducted successful indoor and outdoor evaluations in our testbeds in Berlin and

Tokyo, not only showing that our developed technologies are ready for events like the

Tokyo 2020 Olympics, but also targeting general 5G deployments and current verticals

as autonomous driving.

The results and evaluation of our developed concepts were based on solid research

from other tasks in WP4, as well as the foundation in WP1, WP2 and WP3.

For the future, we are looking forward to extend our developed prototypes and

achieved results with new concepts, continue our partnership with members of this

consortium and research upcoming technologies!


Date: July 2019

Public Deliverable

5G-MiEdge

7 References

[D4.2] 5G-MiEdge – Deliverable D4.2 “5G MiEdge test-bed integrating mmWave access,

liquid RAN C-plane, and user/application centric orchestration”

[Koslowski2018] Konstantin Koslowski et al “SDN Orchestration to Optimize Meshed Millimeter-Wave

Backhaul Networks for MEC-enhanced eMBB Use Cases”, BMSB2018

[Santos2019] Ricardo Santos et al, “mmWave Backhaul Testbed Configurability Using

Software-Defined Networking”, Hindawi Wireless Communications and Mobile

Computing, Volume 2019, Article ID 8342167

[Tran2019] Gia Khanh Tran et al, “Outdoor Experiment of mmWave Meshed Backhaul for Realtime

Edge Content Delivery,” IEEE WCNC2019, Apr. 2019.

[Fukatsu2019] R. Fukatsu et al, “Millimeter-Wave V2V Communications with Cooperative Perception

for Automated Driving,” IEEE VTC-Spring, May 2019.

[Nakamura2019] M. Nakamura et al, “Performance Evaluation of Prefetching Algorithm for Real-time

Edge Content Delivery in 5G System,” IEEE VTC-Fall, Sep. 2019.

[Li2019] Z. Li et al, “Proof-of-Concept of a SDN Based mmWave V2X Network for Safe

Automated Driving,” IEEE Globecom2019, Dec. 2019.

[D1.4] “Final report on joint EU/JP vision, business models and eco-system impact,” 5G-

MiEdge Deliverables, Jul. 2019.

5g-miedge · 2019. 9. 26. · 2 5g-miedge page 2 the document is proprietary of the 5g-miedge...

Documents