network management challenges and trends in multi-layer and

SUBMISSION TO IEEE COMMUNICATIONS SURVEYS & TUTORIALS 1

Network Management Challenges and Trends inMulti-Layer and Multi-Vendor Settings for

Carrier-Grade NetworksA. Martınez, M. Yannuzzi, V. Lopez, D. Lopez, W. Ramırez, R. Serral-Gracia,

X. Masip-Bruin, M. Maciejewski, J. Altmann

Abstract—The exponential growth of Internet traffic gives norespite to the telecommunications industry, and is visibly short-ening the life-cycle of the technologies used for core networking.To cope with the traffic demand, the industry has primarily fo-cused on the evolution of the data and control planes, and hasrapidly made progress in both subjects. However, the innova-tions in the market have not reached the management plane atthe same speed. This stems from a number of factors, most ofwhich point to the segmentation of competencies in managingmulti-layer infrastructures. Current carrier-grade networks areorganized as multi-layer infrastructures, typically composed oftwo layers: IP routers deployed in tandem with optical transportnodes. In turn, each of the two layers is typically composed ofdevices from different vendors, each of which usually suppliesits own (proprietary) Network Management System (NMS). Inpractice, the lack of broadly accepted mechanisms for enablinginteroperability among the different NMSs has led to the iso-lation of these proprietary systems. As a result, the operationand maintenance tasks on the network are becoming increas-ingly complex, which is leading to duplication of functions, higherOPEX, and significant delays in the coordination of multi-layerprovisioning processes. In this paper, we examine in detail theinteroperability challenges of managing multi-layer and multi-vendor carrier-grade networks, and review the current trendsand recent standards in the area, with strong focus on industrialadvances. We cover the Multi-Technology Operations System In-terface (MTOSI) as well as OpenFlow, and analyze their potentialimpact and reach. We also discuss some of the reasons why rele-vant carrier-grade management proposals have not been able tofulfill the requirements of Internet Service Providers (ISPs), andidentify a set of features that might help pave the way to marketfor new management products.

Index Terms—Networks, management, multi-layer, multi-vendor, IP, optical, interoperability.

I. INTRODUCTION

TO cope with the ever-increasing bandwidth demand, cur-rent carrier-grade networks have evolved to multi-layer

infrastructures, typically composed of IP switching and routing

Manuscript received February 17 2014.A. Martınez, M. Yannuzzi, and R. Serral-Gracia are with the Networking andInformation Technology Lab (NetIT Lab), Spain (emails: {annym, yannuzzi,rserral}@ac.upc.edu).V. Lopez and D. Lopez are with TELEFONICA I+D, Spain (e-mails: {vlopez,diego}@tid.es).W. Ramınez and X. Masip-Bruin are with the Advanced Network Architec-tures Lab (CRAAX), Spain (emails: {wramirez, xmasip}@ac.upc.edu).M. Maciejewski is with ADVA Optical Networking, Poland (e-mail:{MMaciejewski}@advaoptical.com).J. Altmann is with Seoul National University (SNU), Korea (e-mail:{altmann}@acm.org).

devices deployed in tandem with optical transport gear. Indeed,the convergence of IP and optical transport networks has beenat the heart of telecom carriers’ strategies and investments, notonly for improving the scalability and switching efficiency inthe IP core, but also for achieving higher switching capacitiesat lower costs. This trend is actually leading to the utilizationof “more optics” in the network, since carriers are graduallyoffloading transit traffic from expensive high-end routers to-ward cheaper and more energy-efficient optical nodes [1], [2],[3], [4], [5], [6].

For the sake of clarity, in the context of this article we willrefer to the “IP Layer” as the IP framing layer in the TCP/IPreference model [7], i.e., a layer ruled by packet-based switch-ing, whilst the “Transport Layer” refers to the optical networkproviding the physical transmission layer, i.e., a layer ruledby optical-based switching. Observe that the latter is differentfrom the Layer 4 (L4) of the traditional Open Systems Inter-face (OSI) and TCP/IP reference models. Figure 1 illustratesthe mapping between the classical OSI and TCP/IP referencemodels and the layered model of a multi-layer carrier-gradenetwork. Observe that we present three possible approaches forthe layering model of a carrier network, which are representa-tive of different deployment scenarios—mainly indicating thetime progression from left to right. The term Intelligent Opti-cal Transport Network (OTN) in Fig. 1, typically refers to anoptical network endowed with more advanced control planes,such that it can offer rapid circuit provisioning, service flexi-bility, multi-vendor interoperability and enhanced survivability[8].

Overall, carrier-grade networks are currently experiencingconsiderable changes. In the process of evolving to multi-layerinfrastructures, the telecommunications industry has made re-markable advances in the data and control plane technologies.The former is evidenced by the advances made from IP overoptical transmission at 10G to 40G, and now 100G and be-yond, while the latter is reflected in the increasing supply ofequipment supporting cross-vendor interoperability in confor-mance with major standards, such as the Generalized Multi-Protocol Label Switching (GMPLS) [9], and the AutomaticallySwitched Optical Network (ASON) architecture [10]. How-ever, the innovations in the management plane have not beenable to keep that pace. Indeed, the advances in this area lagfar behind carriers’ expectations, and they are making networkmanagement tasks increasingly more complex. This complex-ity lies in part on the rich set of functionalities that network


Network

Transport

Application

TCP/IP Model

Network Interface

IP/MPLS

Multi-Layer Network Model

IP Layer

Transport Layer

Data Link

Physical

OSI Reference Model

Layer 1

SONET/SDH

OTN (DWDM)

Layer 2

NetworkLayer 3

Presentation

Session

TransportLayer 4

Application

Layer 5

Layer 6

Layer 7

IP/MPLS IP/GMPLS

Intelligent OTN (DWDM)

Ethernet

OTN (DWDM)

Fig. 1: Mapping between the classical OSI Reference model, the TCP/IP model, and the Multi-layer network model.

management entails, in addition to the heterogeneity of currentnetwork settings [11].

In a nutshell, network management covers a broad func-tional area that typically includes the Operation, Administra-tion, Maintenance and Provisioning (OAM&P) functions re-quired to monitor, configure, interpret, measure and control thenetwork and the services it offers. Network Management haslong been targeted by carriers as a means for achieving effec-tive and efficient service monitoring and resource administra-tion, reducing operational costs, delivering guaranteed networkperformance, recovering from outages (faults) and maximizingthe control and security of a network. The proper managementof a network results in a secure, reliable and dynamic system,capable of detecting and reacting to critical scenarios that canhighly affect the performance of the network [12].

From a management point of view, the IP and optical trans-port layers are typically composed of devices supplied by dif-ferent vendors, each of which is managed by its own (pro-prietary) Network Management System (NMS). This separa-tion of management systems and tasks, along with the inher-ent technological differences between these two layers, havedeeply segmented the operation of the IP and optical transportnetworks [13]. In addition to this, the lack of widely acceptedmechanisms for interoperation of the different NMSs has led toadministratively independent ecosystems (i.e., from a manage-ment perspective, each layer operates in complete isolation).This raises the need for interoperability between these networkmanagement ecosystems, to enable overall administration andmaintenance of a multi-layer network [14].

In this framework, one of the central questions that arisesis: Why is management interoperability so important in thefirst place? The European Information & CommunicationsTechnology Industry Association (EICTA) defines interoper-ability as “The ability of two or more networks, systems,devices, applications or components, to exchange informa-tion between them and to use the information so exchanged”[15]. Interoperability—in the context of multi-layer networkmanagement—is the key component for enabling communi-cations between the IP and the transport layer managementsystems, thereby allowing coordinated monitoring, provision-ing and execution of cross-layer operations in an automated

fashion. Without such interoperability, cross-layer provisioningtasks are limited to manual means, which is precisely the casenowadays. The downside is that this considerably increases thecomplexity and the costs for operating and maintaining multi-layer networks at the backbone of many large Internet ServiceProviders (ISPs), wherein even simple management tasks re-quire multiple and repetitive manual procedures, leading tolong provisioning time scales.

A closer look into the management interoperability prob-lem in multi-layer networks reveals that it has actually twodimensions. As depicted in Fig. 2, the interoperability prob-lems extend not only vertically between the IP and the opticallayers but also horizontally, i.e., even within a single networklayer. Matters relating to technical aspects, as well as finan-cial and business strategies induce telecom service providers toavoid single vendor dependencies, which yields multi-vendornetwork environments, such as the one shown in Fig. 2. Here-after, we will refer to the vertical interoperability issue in Fig.2 as the Multi-Layer Interoperability (MLI) problem, indi-cating interoperability issues between different managementplanes or layers. Additionally, we will refer to the horizontalinteroperability issue in Fig. 2 as the Multi-Vendor Interoper-ability (MVI) problem, indicating interoperability issues dueto the need of managing devices supplied by different vendorswithin the same layer. Note that the MLI problem is itself sub-ject to the multi-vendor problem, since the devices used forthe IP layer are often purchased from vendors different thanthose that supply the optical nodes. In any case, the MLI prob-lem is broader in scope than the MVI problem, as it covers,and inherently deals with multi-vendor issues across layers.Accordingly, the MVI problem subject of study in this paperrefers to interoperability issues among different managementsystems within the same layer.

Overall, the aim of this article is to provide a comprehensivedescription of the challenges and the interoperability issues inmanaging multi-layer carrier-grade networks. We review thecurrent trends and recent standards in the area, with strongfocus on industrial advances including: the Network Configu-ration Protocol (NETCONF) [16], the Multi-Technology Op-erations System Interface (MTOSI) [17] as well as the im-plications and potential changes derived from the adoption of


IP Network Management System (IP-NMS) /

Operations Support System (OSS)

IP Layer

Transport Layer

Transport Network Management System (T-NMS) / Operations Support System (OSS)




Multi-Vendor Interoperability

(MVI) Issue

Multi-Layer Interoperability

(MLI) Issue

Multi-Layer Interoperability

(MLI) Issue

Multi-Vendor Interoperability

(MVI) Issue

Fig. 2: The Multi-layer and Multi-vendor interoperability problems that make the different NMSs operate in isolation—notethat the Multi-layer problem entails itself Multi-vendor interoperability issues.

OpenFlow [18], among others. We also analyze and discusswhy some solutions in this field have not gained momentum,with the aim of at least partially identifying the set of featuresand requirements for next generation management solutions.In fact, while some of the solutions described herein might notbe part of a new wave of emerging technologies in the field,the problems that these technologies have tried to address re-main largely unsolved. For example, consider NETCONF, themost recent effort of IETF to standardize IP network man-agement configuration. Indeed, it is not a new technology andhas even been implemented by numerous vendors and alreadyreleased in their commercial products. However, in practicemany network administrators are still managing and configur-ing their routers using proprietary Command-Line Interfaces(CLIs). According to industry sources, near 95% of networkdevices are still configured through proprietary CLIs [19]. Thispaper analyzes the reasons why this happens in practice, notonly for NETCONF but also for other technologies, as aneffort to understand which problems remain open, and mostimportantly, determine the most promising paths toward thefuture management of multi-layer carrier-grade networks.

Figure 3 presents a schematic view of the topics addressedin this paper. Note that, we will deal with numerous tech-nologies and the interrelations between them. We will analyzetraditional approaches to multi-layer network management is-sues, delve into the most relevant aspects of newly emergingtrends, and draw the lines for future multi-layer approaches,based on coordinated strategies through promising mediationtechnologies, e.g., supported through Software Defined Net-working (SDN) [20] [21].

The remainder of this paper is organized as follows (cf. Fig.3). Section II describes in detail the MVI and MLI problemsin the context of carrier-grade networks. Section III overviewsthe current strategies in the field, and presents an analysis onhow these solutions can face the interoperability problems innetwork management. Section IV presents a more advancedperspective, covering new research trends in multi-layer net-work management, and also provides insights into the limi-tations of these new lines of work. Section V highlights the

importance of integrating third-party management subsystemsin current multi-layer network management solutions, as en-ablers of specific cross-layer management functionalities. Sec-tion VI, provides the authors’ views on the potential directionsfor future research in the context of multi-layer network man-agement. Finally, Section VII concludes the paper.

II. INTEROPERABILITY ISSUES IN MULTI-LAYER ANDMULTI-VENDOR CARRIER-GRADE NETWORKS

Network management has been the subject of study and oneof the central targets of the telecommunications industry sincethe earliest days of networking. However, the overall manage-ment of multi-layer infrastructures still remains an open fieldfor research, primarily because of the scarce interoperabilitybetween NMSs. In the context of multi-layer networks, man-agement interoperability issues arise mainly due to the hetero-geneity of technologies (i.e., IP and Optical) and the diversityof device manufacturers, coupled with the absence of stan-dardized and broadly accepted mechanisms that enable bothcross-layer and intra-layer communication of NMSs. A clearproof of this situation is the complexity of current OperationsSupport Systems (OSS). OSSs were originally conceived asthe elements bridging business logic for service provision andnetwork management, but in practice they have had to incor-porate several layers of “Umbrella NMSs” (as described laterin section III-A) to provide a minimum degree of workflowautomation. Cutting down OSS complexity is among the mostimportant efficiency objectives of practically all telecom op-erators.

The isolation between the IP and transport managementecosystems is exacerbated by the functional segmentation ofstandardization bodies, which are not necessarily working inthe same direction. While, the Internet Engineering Task Force(IETF) is the organization playing a critical role in the scopeof IP-driven network management standards [22], the Interna-tional Telecommunication Union (ITU) [23], the Optical Inter-networking Forum (OIF) [24], and the TeleManagement Fo-rum (TMF) [25], are the most active organizations working


Multi-Layer Network Management

MLI MVIMain Problems and

Interoperability Issues

- In-house Developments

- Control Plane Technologies

- IPoDWDM

Traditional Strategies to face

MVI and MLI Management

issues

New Trends in ML Network

Management

- Hybrid Nodes- MPLS-TP- Integrated

NMS

Future Trends in ML Network

Management

CoordinatedManagement

Enabling Technologies

• CLI• NETCONF / Yang • Openflow

• SNMP • MTOSI

• GMPLS

• Juniper’s PTX

• CyMS

• Cisco CRS1/3

• Juniper’s M-Series

- PCE- VNTM

3rd Party Management

Subsystems

• PCEP

• LMP

• MTOSI • NETCONF • Semantic Technologies

• SDNs • Openflow • Virtualization

Lessons LearnedObstacles Paths toward solutions

• Coop. Models

• Huawei’s OptiX OSN 9800

Fig. 3: Scheme of the topics covered and organization of thispaper.

toward the development of standards in the field of opticaltransport networks. In this section, we provide a detailed anal-ysis on the MVI and MLI issues for carrier-grade networks andoutline the complexities that emerge as a result of the isolationof their management ecosystems.

A. Multi-Layer Interoperability (MLI) Issues

The MLI issues between the IP and transport NMSs ariseprimarily due to the development of systems targeting indi-vidual network layer functions in the Open Systems Intercon-

nection (OSI) model. While the use of the OSI model hasensured that the technological diversity in lower layers doesnot affect the operation of the upper layers, the lack of mecha-nisms for enabling coordinated management between them hasled to the replication of critical functions. Restoration mech-anisms are a clear example of the redundancy produced byindependent (i.e., per-layer) management functions in multi-layer settings, as they are featured by both layers to restoretraffic onto alternate paths in case of failure. Indeed, the scarceefficiency for protecting and recovering multi-layer networksis the result of the absence of communication mechanismsbetween management ecosystems, which leads to the activa-tion of restoration functions in individual layers based on theutilization of static time thresholds. In typical deployments,transport network restoration is attempted first, and restorationat the IP level is usually delayed by a static time threshold,which can lead to significant losses in case that restoration inthe transport network fails [26].

Let us consider the example shown in Fig. 4, which summa-rizes the possible failure scenarios in a multi-layer network.Link and node failures at the transport layer (denoted as (1),and (2), respectively) can be typically recovered by the opticalnetwork without manual intervention, provided that the spareresources and capacity are sufficient for that end. A failure of arouter (3), on the other hand, can be recovered by the IP/MPLSlayer. However, whenever a failure occurs at the interconnectpoint (4)—i.e., when an inter-layer failure occurs—a coordi-nated strategy is required for safe and efficient recovery. It isworth mentioning that, currently, this coordination is not dy-namically resolved; quite on the contrary, it is predefined andmeticulously pre-configured during network planning cycles.Observe that for failures (3) and (4), if no backup router isavailable at the IP layer, the transport layer could reactively at-tempt to optically bypass the affected router. Unfortunately, thelack of cross-layer management mechanisms does not allowdynamic cross-layer recovery, which leads to an unnecessaryduplication of resources and restoration mechanisms at bothlayers. Indeed, enabling communication between NMSs of dif-ferent layers would derive in much more efficient restorationstrategies, and could help alleviating in the future part of theduplicity in current carrier-grade protection schemes.

Another important aspect is that the MLI issue also hindersthe automation of multi-layer tasks between the managementecosystems of the different network layers. Hence, only man-ual means are attainable for coordinating cross-layer opera-

IP/MPLSLayer

TransportLayer

1

2

4

1 2

3

4

Trigger recovery at the Transport Layer

Implies recovery at theIP/MPLS Layer

Requires coordinatedrecovery.

3

Fig. 4: Typical failure scenarios in multi-layer networks.


tions. By manual means we refer to the human interactionsbetween administrators in each network domain. To illustratethis, consider the multi-layer setting shown in Fig. 5. In prac-tice, for setting up an IP service (e.g., the provision of IPlinks), the Planning or Sales Department must first issue a newservice request to the IP Network Management Department(1), which in turn performs the corresponding actions to checkfor the availability of IP resources (2). If the required resourcesare available at the IP layer, then the administrator will issuea request to the Transport Network Management Departmentfor the set up of new optical paths (3). It is worth highlightingthat, as the IP and Transport networks are typically managedseparately and independently, they require to comply with for-malities and procedures for information exchange, a fact thatoften results in high provisioning timescales. Once the requestis properly received by the Transport Network ManagementDepartment, meaning by this that the request satisfies all therequirements from both parts, the Transport Department com-pletes a series of actions, including confirmation of availableresources and their subsequent provisioning at the transportnetwork (4). After the Transport Department has fulfilled theincoming request, an acknowledgement is sent back to the IPDepartment (5). From this response, the IP Network Manage-ment Department completes the corresponding configurationsof IP devices in their domains (6), and finally, notifies thecompleteness of the IP service request (7). As seen in Fig.5, the manual coordination process is very slow, so even thesimplest cross-layer operation, such as the establishment of asingle IP link, can take hours or even days. As depicted in

Fig. 6, in large carrier-grade networks routers A and B mayperfectly belong to different “IP management domains” (e.g.,when router A is supplied by router vendor VA and router Bby vendor VB). Likewise, Fig. 6 also shows that, the light-path required in the transport layer for supporting the IP linkmay also need to traverse different “Transport managementdomains” (e.g., from vendor VX → VY → VZ).

In the scope of cross-layer operations for multi-layer net-works, services that could be provisioned in a scale of minutes,currently range to the scale of days or weeks, a fact that be-yond its technical and functional implications translates intohigher operational expenditures (OPEX). Moreover, the con-figuration of fixed or dynamic policies for proactive cross-layer operations (e.g., for dynamic traffic management) arenot openly supported by current multi-layer settings1. Indeed,if more advanced cross-layer operations were developed, theycould further contribute to make significant progress in aspectssuch as multi-layer recovery and self-healing. By self-healingwe mean coordinated protection mechanisms that could pro-vide network reliability and automated restoration actions torecover from unplanned failures. Moreover, another significantchallenge is the automatic discovery of interlayer connections,as for now network administrators rely on manually built topo-logical databases. Finally, the advent of third party systemsmakes integration to multi-layer environments more important

1There are some proprietary solutions, but they are naturally constrainedto single vendor settings, for example, Juniper’s solution based on hybridnodes [27] or Huawei’s Hybrid MSTP OSN 7500 II [28]. In Section IV, wewill cover some of these solutions.

Planning or Sales Department

Provision IP Service

1

7

IP Network Management Department

2 ~Days

~Hours

3 5

Transport Network Management Department

~Days

~Hours

4

~Hours Transport Network

Management System

IP Network Management System

6

Fig. 5: Current workflow for provisioning an IP service involving configurations at both layers.



(IP-NMS) / Operations Support System (OSS)

IP Layer

Router A

Router B

VX

VY

VA VB

Transport Network Management System (T-NMS) / Operations Support System

(OSS)


(IP-NMS) / Operations Support System (OSS)



VZ Transport Layer

Fig. 6: An example of network management interoperability issues involving cross-layer resource provisioning of an IP linkbetween routers A and B of a carrier network.

than ever, as current external tools could be combined to au-tomatically interact with the multi-layer environment, so thelatter can benefit from the services of external subsystems,such as multi-layer Path Computation Elements (PCE) [29],monitoring tools, accounting systems, etc.

Accordingly, the absence of standard management inter-faces for inter-layer communication not only results in dupli-cation of network functions and lack of automation for cross-layer provisioning tasks, but also derives in slow provisioningtimescales, lack of proactive policy-based and failure-relatedinteractions, as well as limited integration of third party man-agement subsystems [14].

Although several efforts are underway for overcoming theMLI issues described above, the overall progress is slow. Mostof the solutions proposed thus far have not had enough echoamong ISPs, so it is hard to assert that there is a clear trendtoward tackling the MLI issues at a management level. In Sec-tions III and IV, we will provide an analysis of the main lim-itations of current research trends, in order to set the groundfor future research lines in the area.

B. Multi-Vendor Interoperability (MVI) Issues

Various bodies such as the International TelecommunicationUnion (ITU-T) and the Internet Engineering Task Force (IETF)are working toward developing standard specifications of con-trol and data plane functions enabling interoperability betweendifferent vendors. However, in practice vendors compete witheach other trying to gain higher market share by implement-ing specific non-standard functionalities in their equipment, afact that unquestionably leads to customer lock-in. This in turncreates a buyers dependency on the seller. According to the re-port released by the market research firm Infonetics Researchin the second quarter of 2013 [30], Cisco had the highest mar-ket share for carrier router and switches, followed by Huaweiwho has gained the most market share over the past two years.Overall, Alcatel-Lucent, Cisco, Huawei and Juniper accountfor 90% of the market share, while the remaining 10% is splitamong several other equipment vendors.

The demand for new functionalities has led to vendor-interoperability requirements in the data as well as the controland management planes. A classical example of interoperabil-ity challenges in the data-plane can be seen in the 100G opticalsolutions available in the market today. While standards pro-pose the use of different configurations of single 100G or mul-tiplexed 40G and 10G lightpaths using fixed spectrum slots of50 GHz, vendors such as Ciena, have developed custom 100Goptical solutions that employ larger spectrum slots to providelonger reach, but makes it limited in terms of interoperabilitywith other vendors’ equipment.

MVI issues related to the management plane are primarilyreflected as the lack of standardized interfaces for communi-cation between NMSs from different vendors within a singlelayer. Let us consider the following example to illustrate theMVI problem. As shown in Fig. 6, nowadays, when a ser-vice provider requires to configure a service spanning acrossdifferent vendor (management) domains (e.g., between Ciena[VX ] and Huawei [VY ] at the transport layer) the configura-tion process is done ad-hoc, mainly due to the lack of meansfor interoperation between NMSs, which translates into highoperational and maintenance expenditures. The forces drivingservice providers to buy both Ciena and Huawei are two-fold:firstly, better costs when negotiating prices (lower CAPEX)and secondly, avoid single vendor dependency (which low-ers CAPEX but increases OPEX). This is a typical exampleof the MVI issue, a problem suffered by almost every ser-vice provider, and which will remain present while purchasepolicies remain the same. Another scenario that reveals suchincompatibility between NMSs is the designation of skilledsoftware development teams, committed to develop dedicatedsoftware agents to enable interoperability between NMSs andnetwork devices of different vendors. For instance, this case iswell-known at ADVA Optical Networking, where the manage-ment interoperability issues with Nokia-compliant technologyare tackled through ad hoc software developments.

In this paper, we refer to the MVI problem in two dimen-sions, firstly the communication between NMSs in a singlenetwork layer (i.e., within IP or within the Transport layer)


and secondly, the communication between a NMS and multi-vendor devices (i.e., routers, Optical Cross Connects (OXCs)or any other network elements), for example, for performingconfiguration tasks. This dimension of the MVI problem hasled to the development of standard open interfaces, thereby en-abling hardware independency (e.g., OpenFlow)—a hot topicin the field which will be further discussed in Section VI.

As for the first dimension of this problem, despite the nu-merous efforts for providing multi-vendor support, at presentmany network management software solutions are constrainedto interoperate with other management systems, a problemwhich normally leads to high stresses in internal implemen-tations and increased costs [31]. As mentioned previously,proprietary network management protocols as well as vendor-specific management data representations are the hallmark ofmany manufacturers to distinguish from other vendors. How-ever, these business and market driven decisions result in se-rious interoperability limitations for managing heterogeneousnetworks. This problem goes beyond the scope of standardprotocols for network management and steps into the field ofdevice-vendors, many of which strongly reject the belief ofbeing as good at managing the competition’s products as theirown [32].

As for the second dimension of the MVI problem, let usconsider the following example. In IP networks, the primaryinterface used for configuration is the Command Line Interface(CLI), a vendor-specific technology which can even changebetween devices of a single vendor, e.g., according to theOperating System (OS) version used. Due to the customiza-tion degree per vendor, under this scenario, achieving trulyintegration of multi-vendor device configurations is a ratherchallenging task. Even in the presence of standardized com-munication protocols and mechanisms, the data models usedby different vendors can vary significantly giving place to in-teroperability challenges. For example, the Simple NetworkManagement Protocol (SNMP) [33] [34] is used as a standardprotocol to communicate with devices in order to facilitateinventory management, alarm notification, and performancemonitoring. However, different vendors specify extended Man-agement Information Bases (MIBs) for SNMP to facilitate ad-vanced features that are not described in the standard MIBs.This means that operators have to modify their managementsystems whenever they introduce new devices in the networkor change devices to a different vendor. The explanation forthe prevalence of proprietary approaches in industry, is thatthey are the result of forces driven by vendors rather than byservice providers, thus, it highlights the divergent interest ofboth parties. While operators target the interoperability in het-erogeneous networks, manufacturers clearly target their ownbusiness interests.

Two technologies, the Network Configuration Protocol(NETCONF) [16] and the Multi-Technology Operations Sys-tem Interface (MTOSI) [17] are positioned as promisingtechnologies for the IP and Transport Network Managementecosystems, respectively. On the one hand, NETCONF seemsthe expected standard for remote IP device configuration. Thisprotocol aims to overcome the limitations of SNMP-based con-figurations as well as proprietary CLIs to reduce complex-

ity and lower operational expenditures—according to industrysources network device configuration is actually the greatestcontributor to OPEX [19]. NETCONF is defined for provid-ing configuration state maintenance, concurrent configuration,transaction across devices and roll-back support. Initial imple-mentations of NETCONF in vendor equipment have alreadybeen developed (e.g., see Juniper, Cisco and tail-f [35], [36],[37]). Nevertheless, it is important to note that NETCONFonly foresees the access protocol and configuration of net-work elements, but it does not take into account the definitionof configuration information, i.e., an explicit way of express-ing its payload.

For a complete definition of the configuration model, a datamodeling language is also required, to provide the means fordefining and declaring the network elements particularities(e.g., interfaces, addressing, bandwidth, etc.). In other words,a concrete and precise way of expressing what can be readand written over the configuration is needed. The lack of astandard data model across all vendors led to a new challengearound NETCONF. Indeed, even if NETCONF is implementedby all vendors’ equipment, its future depends on the adoptionof a standard modeling language, allowing a common viewand knowledge of network equipment. The initial implemen-tations of NETCONF relied on proprietary data models, whichin turn raised clear interoperability issues. Examples of datamodels that have been proposed and tested for its use withinthe NETCONF framework are: the XML Schema (XSD) [38],Relax NG [39] and Ontology Web Language (OWL) [40].

More specifically, XML-based languages, such as XSD andRelax NG, were initially considered as potential candidatesfor NETCONF content definition. They were evaluated andcompared with YANG [41], in order to assess their suitabilityas network management data models [39], [42]. These studiescompared language data models according to their level of ex-pressiveness, elements of construction, readability and inter-operability, and revealed that XML Schema languages weresuitable for general constructions, but they had neither theadaptability to model hierarchical data in a clear and con-cise way, nor the level of expressiveness provided by YANG[41]—a data model that has been specifically designed forthe NETCONF protocol. According to these studies, YANGalso outperformed schema languages in terms of readability,due to its likeliness to programming languages. Furthermore,XSD and Relax NG were shown to be too general for domain-specific data modeling in the context of network management.

On the other hand, OWL [43]—a W3C specification for au-thoring ontologies—was also proposed as a modeling languagefor network management information [40]. The proposal is todefine OWL modules for common concepts of any NETCONFconfiguration model, as well as operations and notifications,and to agree on a standard serialization method, while sug-gesting RDF/XML for this purpose. The initiative proposedin [40] also exposes how OWL fulfills NETCONF’s main re-quirements, such as the capability to define NETCONF oper-ations and newly derived ones, error annotation capabilities,human readability, meta-data support, reusability, support tobasic types and relationships, etc. Nevertheless, OWL as wellas other XML-based initial approaches have failed to position


as the best candidates for network configuration data defini-tion either because of their lack of semantics or their intricateuse.

At present, YANG [41] is positioned as the strongest can-didate to a standard data model language for NETCONF. It isthe IETF’s proposed standard to create a common language fordata modeling definition, and although some YANG compli-ant software applications have already been developed, there isstill a long path before YANG and NETCONF become leadingstandards in the network configuration arena.

As for the transport ecosystem, MTOSI emerges as a pro-tocol aimed to overcome the interoperability issues of SNMPand it operates with unified network data models and opera-tions supported by Web Services. However, a limitation is en-visioned for MTOSI. Despite the fact that there is a standard-ized data model specification, the representation of deviceswithin this data model is different, thus, vendor interoperabil-ity becomes a major challenge. In Section VI, we will delveinto the potential of MTOSI for enabling interoperability inmulti-layer infrastructures.

III. TRADITIONAL STRATEGIES TO FACE THE MLI ANDMVI PROBLEM IN NETWORK MANAGEMENT

Initial efforts led by industry and academia to overcomethe MLI and MVI problems in network management haveput into practice several strategies. In-house developments,IP over DWDM (IPoDWDM) [44], and control plane tech-nologies [9], [10], are some of the main solutions currentlydeployed by many large ISPs with the aim of simplifying themissing management functions in multi-layer and multi-vendorsettings. Nevertheless, these developments either solve partialproblems or target them in a temporal and local way, hinder-ing the possibilities of becoming absolute trends in the fieldof multi-layer network management. Either way, they providevaluable foundations for: (i) providing basic support of missing

management functions, (ii) envisioning new requirements forfuture approaches targeting interoperability and (iii) bringingto light the real needs of network operators as they evidencethe existence of numerous interoperability problems.

A. In-house developments

As stated in the previous section, the choice of largetelecommunication service providers to defeat mono-vendorsettings is usually a business and market-driven decision,which has naturally led to heterogeneous network infrastruc-tures. The problems arising from such inherent heterogene-ity have resulted in the development of in-house applications,aimed to locally address the management limitations in multi-vendor and multi-layer settings (see Fig. 7). These platformsare developed as internal network management engines and arealso known as Umbrella NMSs. They are custom-made andthey are typically centralized solutions built to temporally ad-dress the specific needs of network administrators. They followa Manager of Manager (MoM) architecture, an alternative thathas always been available for network managers to “smooth”the limitations around existing management solutions.

Umbrella NMSs lack of flexibility, efficiency or means forexchanging or enabling interoperability between layers in amulti-layer setting. They usually perform as central systemswith no top-to-low layer communication or vice-versa. Asshown in Fig. 7, these umbrella systems typically offer cus-tom interfaces to existing NMSs developed in each networklayer (A). In many other cases, the umbrella NMSs directlyinteract with the network elements (B) to avoid the complex-ity, the high costs and administrative delays that derive fromrequesting new functionalities to be developed over propri-etary NMSs. This allows to achieve the required managementfunctions at lower costs, reduced time-scales, and higher lev-els of customization. In-house developments are not efficientand have brought to light the inadequacies of current network

IP-NMS / OSS

T-NMS / OSS

IP Layer

Transport Layer

IP-NMS / OSS

T-NMS / OSS

A

B

A

B

Interaction with underlying NMS.

Direct Interaction with Network Elements.

A

B

A A

Fig. 7: Facing the interoperability issues with in-house developments.


management tools for targeting the problems that arise in thecontext of multi-layer settings. Umbrella NMSs are just an ex-ample that discloses the fact that current NMSs lag far behindthe requirements faced in practice by network managers.

B. IP over DWDM

IP over dense wavelength-division multiplexing (IPoD-WDM) [45], [46] is a solution for approaching the core net-work architecture in order to support the ever-increasing vol-ume of IP traffic. This architecture proposes the convergenceof IP and DWDM transport technology by integrating colouredtransport interfaces directly in the IP routers and photonicswitching into DWDM platforms. Integration of transpondersonto routing platforms eliminates external layers of transpon-ders between the IP and optical transport layers. The overallgoal is to simplify the network while lowering costs. This in-tegration highly contributes to capital and operational savingsdue to the reduction in the number of network elements, spaceoccupation, and energy consumption. It also leads to simpli-fication of the network by eliminating SDH/SONET boxes.Furthermore, it provides inherent protection capabilities, dueto the knowledge shared between layers, as IP devices canmonitor the optical path. However, the final goal for full multi-layer integration assumes support of the control plane to com-plete cross-layer network provisioning and monitoring—wewill delve into these aspects in the following section.

C. Control Plane Technologies

Several network device suppliers already provide GMPLSsupport in their carrier-grade equipment (e.g., Huawei’s Op-tiX OSN 9800, Juniper’s M-series or Cisco’s CRS-1/3), andsome carriers have even fostered the early adoption of thistechnology. For instance, the major operator in Japan, namely,NTT, has been a leading promoter of the GMPLS technology[47]. Despite these advances, technologies such as GMPLS arestill not widely deployed in practice [48]. The true fact is thattelecom providers are not under a big pressure for deployingGMPLS—at least not until the latter acquires more momen-tum. This means that, in the meanwhile, IPoDWDM solutionsare restricted to scenarios where mainly manual transport pathsare established, arising clear interoperability issues betweenlayers. Indeed, the proprietary nature of most IPoDWDM so-lutions, conditions the network infrastructure in many cases tomono-vendor settings. Integrated management under this par-ticular scenario would be restricted to end-to-end provisioningfor a single-vendor solution.

In addition, signal compatibility is also a limitation of thesesolutions, since standardization efforts are still required toensure a perfect match between transponders and transportequipment. Accordingly, the value of these solutions is lim-ited in scope to signal compatibility and interoperability con-straints. The lack of standardization at the signal transmittedby the transponder is an important issue in the integration oftransponders at the IP/MPLS cards [44]. There are some stan-dardization efforts at the ITU G698.2 [49] to solve the prob-lems of the so called “Black Link”. A Black Link is defined asa link where the transponders (source and destination) and the

intermediate optical equipment may be from the same or fromdifferent vendors. This standard defines the signal parametersfor the input and output interfaces in such network. There aretwo different kinds of wavelength connections: (1) “FriendlyWavelength” and (2) “Alien Wavelength”. A Friendly Wave-length is a lambda connection not created in the optical system,but it is known and managed by the optical management sys-tem. This Friendly Wavelength may be created by a router, butthe optical management system exchanges information withthe router to know the status of the connection. Usually this isdone in mono-vendor solutions with virtual transponders fromthe optical management system point of view. On the otherhand, an Alien Wavelength is a connection created outside theoptical domain (e.g., from an IP Router), but it is not man-aged by the optical network management system, hence, theDWDM system has no early knowledge of signal parameters(e.g., bandwidth, wavelength). In such scenario, the opticalmanagement system is signaled to configure the intermediateOXCs, but it does not have information about the signal qual-ity. Hence, intermediate integrated solutions open new issuessuch as the optical signal information exchange between theIP/MPLS and the optical management systems.

Overall, IPoDWDM strategies have been driven by sev-eral device manufacturers (e.g., Cisco, Juniper and Alcatel-Lucent) setting the trend of next generation networks. Whetherthe management of a network with IP/WDM integration withcoloured interfaces is simpler than todays network manage-ment is still an open issue, as most routers do not deal withtransport specific issues and some problems are still open toresolution.

IV. NEW TRENDS IN MULTI-LAYER NETWORKMANAGEMENT AND CHALLENGES FACED

Aside from the strategies traditionally deployed by serviceproviders to face the MLI and MVI problems, recently, newtrends have emerged in the field of multi-layer network man-agement. These solutions follow one of two lines. They eitherrepresent new ways of approaching the multi-layer networkmanagement problem [50], [51] or they feature new emergingtechnologies [27], [52], which demand new forms of interac-tion and pose different challenges from the management pointof view.

While service providers pursue new trends for targeting theconvergence of IP and Optical layers, research efforts shallconsider all the newly emerging challenges and complexitiesin order to enable interaction between layers at the highestlevel of abstraction. Thus, solutions within the scope of multi-layer network management should provide the flexibility fordelivering true full systems not compromised to specific tech-nologies.

To this end, we present an analysis on new trends in thescope of multi-layer networks. We firstly introduce MPLS-TPa profile of MPLS for transport networks, then we commenton various solutions based on integrated network managementsystems for multi-layer networks, such as the solution devel-oped by Cyan [50], and finally, we delve into hybrid solutionsfor achieving IP over Optical integration, e.g., Juniper’s PTX[27] or Huawei’s Hybrid MSTP OSN 7500 II [28].


A. MPLS-Transport Profile (MPLS-TP)

Multi-Protocol Label Switching Transport Profile (MPLS-TP) [51] has been built as a joint effort between ITU-T andIETF. Its goal is to extend and enhance the concepts of MPLS[53] and create a profile that can be used in the context oftransport networks. MPLS-TP not only adds a set of exten-sions to MPLS, but also disables those capabilities neitherrequired nor consistent with transport technologies. An exam-ple of such suppressed capabilities are Label Switched Path(LSP) merge [54], Penultimate Hop Popping (PHP) [55] andEqual Cost Multi Path (ECMP) [56]. These features are re-moved because they hinder the Operation, Administration andMaintenance (OAM) functions, which are mandatory for trans-port networks. For example, the PHP functionality, which con-sists on removing the label at the penultimate hop—i.e., be-fore reaching its destination—would remove significant infor-mation required by OAM functions to work accurately. An-other major difference between both technologies is that, un-like MPLS, MPLS-TP LSPs are bidirectional, therefore, theforward and backward LSPs follow the same path. This pro-vides the ability to communicate back to the source if anyproblem is encountered, thus simplifying troubleshooting.

In a nutshell, MPLS-TP aims to enable MPLS on transportnetworks, while keeping their operation as close as possibleto already existing transport technologies (e.g., SONET/SDHnetworks). To this end, MPLS-TP—like is the case of mosttransport network platforms—is not constraint to conveyingIP packets, meaning that, MPLS-TP functionality can befully provided regardless of the packet-layer technology used.MPLS-TP has been designed to provide a rich set of controlplane-independent OAM features. In MPLS-TP all OAM func-tionalities run inband, hence, OAM packets are sent over thesame path of the user payload in the MPLS-TP forwardingplane to manage and monitor the transport network and theservices being delivered. Some of the OAM features include,Continuity Check (CC), Connectivity Verification (CV), de-lay and loss measurements as well as fault notification. Anoverview of the MPLS-TP OAM framework can be found in[57].

In addition to OAM enhancements, MPLS-TP provides twomodes of operation. Firstly, configuration of LSPs based onnetwork management plane technologies, which are usually re-ferred to as “static” configuration (i.e., not based on a dynamiccontrol-plane), and secondly, dynamic provisioning based oncontrol plane technologies (e.g., GMPLS). The main advan-tage of the former approach is that MPLS-TP networks canbe managed through a centralized NMS as in traditional trans-port environments (e.g., in ROADM-based optical networks),which makes it quite appealing for network operators whoare used to manage and provision their services in that way.The latter approach, suggests the use of the GMPLS controlplane to provide automatic setup of MPLS-TP LSPs. For moredetails on MPLS-TP’s components and its technical specifica-tions the reader is referred to [51], [57], [58], [59], [60].

In light of all this, MPLS-TP is foreseen as a technologyto leverage traditional transport technology to support packet-based services in a more efficient way, while embracing the

features of traditional transport environments (e.g., QoS, cen-tralized operation, etc.). Over the recent years, network devicemanufacturers and network operators have progressively be-gun to provide support to standard-based MPLS-TP solutions(e.g., Cisco, Nokia Siemens Networks, ZTE, Verizon, BhartiAirtel, Huawei, Telecom Italia, China Mobile, etc. [61]), whileearly applications have already been discussed and deployed.Service Providers are considering the migration of traditionalSONET/SDH networks to packet transport technology. Alongthis path, MPLS-TP appears to be a well-suited technologyto overcome the inefficiency of these networks when trans-porting packets—basically due to constant bit rates even inthe absence of traffic. Indeed, plans for deploying dynamicMPLS-TP over OTN/DWDM in Verizon’s core network werereported in [61]. The goal was to provide connectivity to edgeservices by interconnecting Ethernet and IP/MPLS domainsthrough MPLS-TP.

Furthermore, MPLS-TP has been considered a candidate forreplacing Ethernet in the access network. Data plane compat-ibility between MPLS-TP and IP/MPLS could make interop-erability with the backbone of large ISPs rather simple, whileproviding end-to-end OAM. In this context, the selling point isthat by deploying MPLS-TP—along with already existing de-ployments of IP/MPLS—ISPs will have simple and consistentways of provisioning and managing their networks edge-to-edge. However, technical comparisons in 2012 between bothtechnologies seem to favor Ethernet, claiming that MPLS-TPstill lacks of maturity and security to yet dominate access net-works [62]. If MPLS-TP is ready to replace Ethernet or otherpacket transport technologies in the access or the core networkis yet to be seen.

Overall, one of the driving forces of MPLS-TP is the possi-bility of deploying a unified MPLS strategy, wherein ISPs canuse MPLS in its different flavors (i.e., IP/MPLS, MPLS-TP,etc.) from core to aggregation and access to improve end-to-end convergence. At the same time, the dual mode of operationof MPLS-TP brings flexibility into the design of transport net-works as static configurations may be used while control planetechnologies are not yet in place or required. However, the re-quirements in the scope of network management go beyondthe provisioning functionality and instead require even morecomplex interactions (e.g., scheduling, alarm correlation, andself-healing, among others) to really address the key issues inmulti-layer network management.

B. Integrated Multi-Layer Network Management SystemsAn integrated NMS for multi-layer networks has been for

some time among the research interests of academics [63],[64], [65] and most recently launched by Cyan under the fig-ure of Cyan’s Multi-Layer Management System (CyMS) [50].The aim of these solutions is to provide a unified networkmanagement plane to effectively manage (i.e., configure, pro-vision, monitor, etc.) multi-layer networks regardless of thetechnological differences.

Early works such as [64] and [65] proposed an integratedNMS capable of providing multi-layer connectivity servicesbased on the use of management functions supported by sin-gle layer control plane technologies—whenever available. This


approach mainly focused on supporting basic configurationmanagement functions to provide end-to-end IP connectivityderived from Service Level Agreements (SLAs). In addition,some of the same authors devised a policy-based architecture[66] supporting security policy definition, storage and enforce-ment. Moreover, the authors in [63] developed a prototypeimplementation of an integrated platform based on standardtechnologies, to address the issues of multi-layer network man-agement. In that research initiative, the authors defined theirown XML-based management information modeling language,adopted a new XML-based management protocol and devel-oped mediation modules to interact with legacy protocols (e.g.,SNMP). They also featured policy-based management capa-bilities while providing an open interface through which basicsupport to services was provided. That prototype was assessedunder a simulation environment to prove support to multi-layernetwork optimization tasks, fault management and restorationfunctionality.

Beyond the scope of research, Cyan has actually commer-cialized its product release of an integrated network manage-ment solution, namely, CyMS. CyMS provides a software so-lution for integrated three-dimensional view of the physicaland logical connections in all network layers. It is adaptedand designed to comply with TMF Multi-Technology NetworkManagement (MTNM) and ITU G.800 principles [67]. Addi-tionally, it features three-dimensional heat maps for enablingproactive network monitoring and control, while gradient-colorcoding indicates the existence of critical conditions—for in-stance, for anticipating future faults or service degradation.Indeed, CyMS provides enhanced functionalities for Cyan-compliant devices. However, its potential and dynamics islimited to its own optical gear. This proprietary solution isrestricted in scope to multi-vendor environments, a fact thatmakes it an unsuitable solution in the context of most com-mon and heterogeneous network deployments. The constrainedmulti-vendor reach also contributes to high costs of mainte-nance and updates due to new technologies or requirements.

C. Hybrid NodeSeveral hybrid solutions drawn from industry and academia

have emerged in the field of multi-layer networks [27], [28],[68], [69]. The Juniper PTX converged supercore [27] unveiledon March 2011 is Juniper’s hybrid solution to multi-layer net-works. Juniper’s PTX stands for “packet transport switch”,and, unlike the IPoDWDM solutions described earlier in Sec-tion III, this hybrid approach takes integration a step further bycombining optical and electronic technologies in a unique box.The combination of hardware and software to develop a singlehybrid solution aims to provide integration of IP/MPLS andoptical control and management planes. This integration al-lows to enable coordinated provisioning, planning and model-ing, avoiding traditional duplication of management functions.This transport strategy assumes that IP and Optical layers willno longer be isolated ecosystems. In this sense, Juniper’s PTXpoints in the direction of an integrated approach for multi-layer network management. The convergence of packet/opticallayers within a single platform derives in seamless integra-tion of their control planes based on GMPLS technology and

UNI+, which facilitates multi-layer provisioning, managementand restoration. However, an integrated approach aligned tothe foundations of Juniper’s PTX converged supercore has anumber of limitations: (i) integration is constrained to a sin-gle vendor—at least with current available technology, hencenot solving the MVI problem; (ii) it represents a disruptiveapproach which requires of new network infrastructures; and(iii) ISPs will quite likely keep buying optical equipment toother transport vendors as well as routing devices to other IPmanufacturers, meaning that, this would also have importantimplications from the business and operational points of view.

Moreover, other hybrid approaches have been researched inacademia [68], [69]. The authors in [69] developed a hybridoptoelectronic router in which optical and electrical technolo-gies and complementary metal-oxide-semiconductor (CMOS)electronics are combined into a single router. They demon-strated that such an architecture enables reduction of powerconsumption and latency while still providing the capabilitiesrequired in optical packet switching. R. Cafini et. al. [68] pro-posed a modular programmable router architecture to providedynamic management and configuration of services and re-sources. Indeed, a relevant contribution of this work is to con-sider network programmability to achieve dynamic networklayer functions. The value of featuring network programma-bility is that it has actually become a must for next-generationnetworks, and is the main driver for Software Defined Net-working (SDN)—more details on network programmabilityand SDNs as a means to provide flexibility and opennessfor network infrastructures are given in Section VI-C. Cer-tainly all these approaches for developing hybrid devices arealigned with the belief of improving network management andresource consumption while integrating the management andcontrol planes of different network layers. Conversely, coor-dinated strategies diverge from this view, based on the beliefthat an intermediate system should “coordinate” functions inmulti-layer environments. In Section VI, we will delve into thefuture trends in multi-layer network management, and contrastintegrated approaches against coordinated ones.

V. THE ROLE OF THIRD PARTY MANAGEMENTSUBSYSTEMS IN MULTI-LAYER NETWORKS

The advent of complementary management subsystems hasenabled specific cross-layer management functionalities in thecontext of multi-layer networks. The Path Computation Ele-ment (PCE) [29] and the Virtual Network Topology Manager(VNTM) [70] address cross-layer path computation and cross-layer topology management, respectively. In this section, wewill describe the basics and outline the reach of these manage-ment subsystems in the context of multi-layer network man-agement.

A. Path Computation Element (PCE)

The Path Computation Element (PCE) [29] is a standardizedsolution for facilitating optimal constraint-based path compu-tation in (G)MPLS networks. In this architecture, a Path Com-putation Client (PCC) can request the computation of a pathunder specific constraints to the PCE, which in turn uses its


Traffic Engineering Database (TED) to compute the requestedpaths. The communications between the PCC and the PCEhave been standardized by the IETF in the form of the PathComputation Element Protocol (PCEP) [71]. Note that, pro-viding a detailed description of the PCE is out of the scopeof this paper. Instead, our main focus is to position the PCEin the context of multi-layer settings. Readers are referred to[72], [73] for further details on this subject.

There are different choices for the implementation of thePCE, wherein a PCE server can be implemented within a La-bel Switched Router or as an external PCE that is implementedas a third-party subsystem. The use of a centralized server (i.e.,an external PCE) is especially beneficial in networks requiringcomplex path computation such as Wavelength Switched Opti-cal Networks (WSON). In these networks, the implementationof specific functions for path computation with high complex-ity in each switch can drive up equipment cost. On the otherhand, centralized servers on dedicated hardware designed forpath computation would be a more cost-effective solution, es-pecially in large networks. A unique selling point of the PCEarchitecture is its ability to extend path computation capabili-ties across multiple domains, including multi-layer networks.PCEs preserve topology confidentiality, which is essential incommercial network scenarios. Also, the decoupling of pathcomputation from network devices means that operators canemploy third-party boxes and can control path computationmechanisms and policies with ease, even in a multi-vendorscenario.

To date, the PCE protocol has been standardized for use inMPLS networks and current standardization work is focusingon extending the protocol for supporting networks using theGMPLS control plane [74], and for wavelength assignment inWSON networks [75]. Extensions to the PCE protocol havealso been proposed to compute optimal inter-domain pathson a fixed domain chain using the Backward Recursive PathComputation (BRPC) [76].

Current standardization work is also focusing on the useof PCE for multi-layer path computation [70]. PCE has po-sitioned as the candidate solution to overcome the limita-tions of such networks for providing effective multi-layer TE,which are primarily attributed to the lack of shared networkresource knowledge between layers [77]. In this network sce-nario, multi-domain path computation is not a high priority,but multi-layer path computation is especially relevant as mostlarge carriers typically run a transport as well as an IP/MPLSinfrastructure, and would benefit significantly from automatedmulti-layer path computation. While no definitive solution formulti-layer path computation with the PCE is available today,[70] defines three different mechanisms for the same, namely(i) Single PCE Inter-Layer Path Computation, (ii) MultiplePCE with Inter-Layer Path Computation and (iii) Multiple PCEwithout Inter-Layer Path Computation.

The Single PCE Inter-Layer Path Computation uses onePCE to compute paths between multiple layers. The PCE hasthe global knowledge of all topologies and can compute pathsacross different layers. The Multiple PCE with Inter-LayerPCE communication approach uses PCEs on each layer, hav-ing topological visibility restricted to their own layer. This

model adapts to the Inter-Domain path computation scenario,where different PCEs are chained to compute a strict pathfrom source to destination. In the case of PCE without Inter-Layer PCE communication, each PCE computes loose pathsfrom ingress to egress LSP, in this case higher level LSP tolower level LSP, building the path traversing every PCE untilthe destination is reached—referring again to the PCE Proto-col model of Inter-Domain path computation with loose hops.Out of these mechanisms, the single multi-layer PCE has thebest performance, but the Multiple PCE with Inter-Layer PCEcommunication approach is better suited for most carrier-gradenetworks, given the administrative segmentation between theIP/MPLS and transport network in most carrier organizations.The implementation of the same is demonstrated in [78] wherethe authors use the existing standards to facilitate inter-layerpath computation in a MPLS over WSON network scenario.

Aside from the already mentioned schemes for multi-layerpath computation, the algorithmic issue for computing suchpaths is of utmost importance. The main challenge that multi-layer path computation algorithms face is combining differentlayer-specific constraints to find optimal or near-optimal cross-layer paths [79]. The majority of constraints that condition theset up of a lightpath in an optical network (e.g., wavelengthcontinuity, attenuation, etc.) demand solutions that differ fromtraditional circuit-based computation methods. In light of this,algorithmic solutions in the context of multi-layer networksmust take into account other variants, for example, networkdevice capabilities for performing multi-switching and wave-length conversion in order to compute cross-layer paths undergiven constraints. In the literature, various multi-layer pathcomputation algorithms have been proposed [80], [81], [79],[82], [83], [84], [85].

In [80], [81] B. Jabbari et. al. discuss on the constraintsand possible solutions for computing traffic engineering pathsin multi-layer switched networks. They propose a solution inwhich a network graph is transformed into a channel graph thatexplicitly exposes the constraints of nodes and links, whichotherwise are not visible. Authors in [82] extend this ap-proach by combining the transformation technique with a sim-ple heuristic aimed to cope with the increased complexity ofthe new graph. They also introduce and evaluate a ConstrainedBreadth First Search (C-BSF) technique where constraints areevaluated on-a-fly fashion based on the BSF search algorithm.Authors in [83] developed an heuristic called Min-phys-hoprouting and a wavelength assignment algorithm, which assignsa weight to each optical path that corresponds to the numberof physical links that comprise it. Based on the belief thatan efficient path goes over the minimum number of physi-cal network devices, the least-cost path is chosen. One of themain limitations of this algorithm is that it does not considernetwork nodes capabilities such as Optical-Electrical-Optical(OEO) conversion or wavelength conversion as considered byother algorithms [80], [81].

In Table I, we summarize the main approaches toward solv-ing the algorithmic issue of multi-layer path computation. Notethat, this table does not intend to provide an exhaustive studyon multi-layer path computation algorithms. Instead, it pro-


Constraint Type Proposed Path Computation Algorithms

Based on bandwidth requirements

Prunable Based on protection requirements• Several variants of CSPF (Constrained Shortest Path First) [86]

Based on policy constraint requirements

Additive (e.g., attenuation, dispersion, delay,etc.)

• KSP (K Shortest Paths) [87], [88], [89]

Non-PrunableNon-Additive (e.g., wavelength continuity,switching capability, etc.)

• Common Vector [80]• Constrained Breadth First Search (C-BFS) [82]• Channel Graph-based solution [80], [81]• Label-Layer Graph-based solution [82]• Auxiliary Graphs [90], [91]• Dynamic VNT Configuration Algorithm [79]

TABLE I: Multi-layer Path Computation Algorithms.

vides an introductory and representative list of solid contribu-tions in the field. We have categorized constraints into prun-able and non-prunable classes as defined by authors in [80].The prunable category refers to all those constraints that canbe met by simply discarding the elements that do not com-ply with the required feature from the path computation pro-cess, e.g., bandwidth requirements, wherein potential paths notmeeting a bandwidth constraint can be excluded from the pathsearch process. The non-prunable category, on the other hand,usually comprises the set of constraints that require more com-plex computation strategies taking into account multiple net-work attributes along the whole path. For instance, in net-works where certain nodes are endowed with wavelength con-verters, the lack of a common wavelength along the entirepath is not sufficient to discard the latter as a potential can-didate, since we must also assess the wavelength conversioncapabilities at intermediate nodes along the path. Indeed, de-termining whether a constraint is prunable or not is also sub-ject to design requirements. Multi-constraint path computationalgorithms generally follow one of two approaches, namely,computing paths over the raw network graph and then per-forming robust runtime constraint evaluation (e.g., algorithmsbased on linear programming), or graph transformation tech-niques, where network graphs are transformed into elaboratedgraphs capable of representing a set of given constraints [82].For more details on the algorithmic issues we refer readers tothe references found in Table I.

Note that, despite of all the ongoing standardization effortsin the field of cooperative path computation—some alreadydiscussed herein—many issues still remain open for research.Consider for example, the computation of a cooperative pathbetween several PCE’s based on proprietary objective func-tions operating on non-standard constraints. Current frame-work does support encoding of standard and proprietary ob-jective functions [92]. However, there are no available mecha-nisms in the current definition of PCEP for conveying vendor-specific information on which proprietary objective functionsrely. Authors in [93] have recently proposed a mechanism forconveying vendor-specific constraints in PCEP in which theydefine a dedicated object—the “vendor information object”—to convey vendor-specific information. In light of this, thereis still many to be done in the scope of multi-layer path com-

putation. The realization of all these efforts and proposals areindeed required in the way for achieving a truly mature tech-nology suitable to the complex requirements of multi-layerand multi-vendor carrier-grade infrastructures.

B. Virtual Network Topology Manager (VNTM)

The Virtual Network Topology (VNT) is the network topol-ogy formed by lower layer LSPs and the logical view of theseconnections in the upper layer [52]. The relationship betweenboth layers is created with “virtual TE links” [94]. The vir-tual TE links are potential connections between two nodes atthe upper layer, which are based on possible connections atthe lower layer (i.e., not fully provisioned LSPs). The VirtualNetwork Topology Manager (VNTM) is an entity in charge ofmaintaining the topology of the upper layer by connections inthe lower layer [70].

To optimize the overall use of network resources in multi-layer environments, there are basically two requirements: (i)a cross-layer path computation strategy to compute end-to-end inter-layer paths and (ii) mechanisms to control and man-age the VNT by provisioning and releasing connections in thelower layer. In this regard, the VNTM and the PCE are bothkey entities for coordinating and managing multi-layer paths.While the PCE is responsible for computing the path betweenendpoints, the VNTM initiates the signaling for circuit setupor decommissioning in the transport layer [78].

There are two possible cooperation models for inter-layerpath computation, namely, the PCE-VNTM or the NMS-VNTMcooperation models [70]. To illustrate the relationship betweenthe PCE and the VNTM for multi-layer path computation in aPCE-VNTM model, let us consider the example shown in Fig.8. In this case, a single PCE is used to compute paths betweenboth layers (i.e., a single inter-layer path computation strategy,as described in the previous section). Let us assume that, thesingle-layer PCE fails to compute an inter-layer path becauseno logical connection is set up in the upper layer (IP; PCE). Insuch conditions, the PCE can request or suggest to the VNTMadditional connections in the lower layer (OXC; VNTM). If thePCE has visibility of the lower layer topology it can explicitlysuggest a given path or it can just exchange information onthe upper layer request and wait for the advertisement of the


Transport Network Management System (T-

NMS) / Operations Support System (OSS)

IP Layer

Transport Layer

IP Router

Optical Cross-Connect (OXC)



PCEP

Not Defined

Fig. 8: An example showing one possible PCE/VNTMconfiguration.

new virtual TE link. Moreover, the VNTM could change theconnections to the upper layer if its policies indicate that abetter configuration exists. The operation mode of the VNTMcould be any solution even using another PCE to computelower layer LSPs.

Another approach to inter-layer path computation is whenthe VNTM is part of the NMS (NMS-VNTM model [70]) (cf.,Fig. 9). This model assumes that the VNTM can be embed-ded within the NMS to cooperate in the set up of lower layerLSPs. In this scenario, the NMS requests the PCE to computea path and upon receiving the result of such computation, theNMS is able to request the VNTM to create a new connec-tion. The interface to request a connection is not currentlydescribed or defined in any RFC (this is why we representthe interface between the NMS and the VNTM in Fig. 9 withan interrogation mark). In [70] a TMF standard interface isproposed, while authors in [78] propose the utilization of thePCEP protocol with a new message to suggest a new route tothe VNTM, so the VNTM can decide, based on its policies,if creating the connection or not.

Based on the previous definition of the VNTM, the virtualTE link creation is done from the source IP/MPLS router tothe destination router. This means that the VNTM must haveinformation not only about the lower layer, but also about theinterlayer connections between the IP/MPLS routers and theOXCs. However, automatically building and accurately keep-ing updated the inter-layer connections remains an open issue.State-of-the-art protocols, such as the Link Management Pro-tocol (LMP) [95], offer palliative solutions. This point-to-pointprotocol, defined by the IETF, is used by GMPLS devices tofeature link discovery, i.e., determine data plane connectivityof network nodes to and from its neighbors. Link verificationand fault isolation are also featured by this protocol to checkon the status of links and to isolate faults that may occur,respectively [96]. Nevertheless, LMP is not widely used byISPs mainly because current deployed multi-layer networksdo not provide full integration of GMPLS technology. Actu-

ally, in practice most ISPs remain relaying on databases, mostof which require human intervention for feeding and updatinginter-layer topological data.

In summary, the emergence of management subsystems wasenvisioned to simplify and partially fulfill a set of manage-ment functionalities that were not covered by existing solu-tions. This makes mandatory to consider the integration andfuture requirements of third-party subsystems in multi-layermanagement infrastructures.

VI. FUTURE TRENDS IN MULTI-LAYER NETWORKMANAGEMENT

A. Coordination vs. Integration

Integrated and coordinated approaches both offer a pathfor addressing current multi-layer network management chal-lenges. The integrated approach (cf. Fig. 10a) proposes tounify the two separate NMSs into a single entity, enablingmanagement automation and reduction of operational expen-ditures. This figure shows two possible configurations. In thefirst one, the IP and Optical devices are part of separatelayers—as in traditional multi-layer deployments—but theirmanagement planes are unified under a common NMS. In thesecond configuration, the IP and Optical equipment are in-tegrated into a single box with a unified management plane.Examples of the former are the integrated control plane andmanagement plane frameworks [50], [63], [64], [65], and forthe latter, the case of Juniper’s hybrid node [27].

Unfortunately, these approaches pose complex challengesboth technically and operationally. For instance, multi-layernetwork management issues cannot be fully addressed by theintegration of network control planes per-se, as they do notfulfill all management competencies. Therefore, under thisscenario, the management planes of both layers would alsorequire a level of integration; otherwise, cross-layer servicemanagement operations (i.e., provisioning, monitoring, alarmcorrelation, scheduling, etc.) will not be possible. Furthermore,the integration of network management systems aggregates ameasure of disruption. More specifically, the integration of the

PCENMS

VNTM

LSRH1

LSRH2

LSRL1

LSRL2

LSRH3

LSRH4

PCEP

?

Fig. 9: Another example of a PCE/VNTM configuration witha VNTM entity embedded within the NMS.


IP Layer

Transport Layer

IP + Optical Transport

Transport Layer

IP Layer

IP-NMS

Mediator

(a) Integrated Network Management (b) Coordinated Network Management

Integrated ML-NMS

Integrated ML-NMS

T-NMS

Fig. 10: Integrated vs. coordinated approaches for multi-layer network management.

IP-NMS and T-NMS entails a change of perspective in cur-rent carriers’ practices, and therefore, a major investment mustbe done to acquire and adapt to the new management envi-ronment. The traditional separation between the IP and trans-port network layers has demonstrated significant resistance tosuch a game change, given its implications on the operational,functional and business strategies. Moreover, a successful net-work management integration can only be useful for a singledomain, since domain administrators are reluctant to sharingperformance information about their networks.

In light of all these limitations, coordinated network ap-proaches (cf. Fig. 10b) are positioned as reliable solutions formulti-layer network management. Coordination of cross-layernetwork operations can help not only to reduce operational andcapital expenses, but also to significantly simplify operationalprocesses and reduce administrative burdens. A coordinatedapproach is based on the idea of a mediator system (shownin Fig. 10b) capable of overcoming the barriers of protocoldifferences and network equipment heterogeneity. A mediatorcan address the MLI and MVI issues without requiring ma-jor changes in network practices or the technologies currentlybeing used by network operators on both layers. Additionally,the coordinated approach can easily be adapted to meet fu-ture application demands, as its adoption does not cause anydisruption to current network practices.

One emerging initiative in this direction is the ONE Adapter[97], a solution for enabling interoperability in a coordinatedfashion. The ONE adapter is based on a mediator model, whichhas been designed to operate between the IP/MPLS and thetransport layer NMSs. It allows system automation and coor-dination of cross-layer network management functions. Thisinitiative proposes a non-disruptive solution that only requiresto interface with the IP-NMS and the T-NMS. Furthermore,it does not require changes ro current carriers’ practices. TheONE adapter is particularly useful for management actionssuch as coordinated self-healing, coordinated IP traffic offload-ing, and coordinated service provisioning. The ONE approach

also allows coordination of network management functions ina multi-vendor environment, thus addressing the MVI problemas well. One of the key features in ONE is its ontology-drivennature, which enables syntactic adaptations based on sharedsemantic knowledge of different vendor spaces.

Table II summarizes the advantages and drawbacks of co-ordinated versus integrated multi-layer network management.

B. Enabling Technologies for Coordinated ApproachesIn addition to the numerous advantages and benefits that

a coordinated approach can provide to address the MLI andMVI problems in the context of network management, we haveidentified a number of emerging technologies that can assistin the implementation of truly coordinated platforms.

Beyond the scope of traditional mechanisms for networkmanagement, there are a number of technologies capableof enabling enhanced functionalities to this critical areaof networking. MTOSI [17], NETCONF [16], semanticapproaches and OpenFlow [18] are among the most relevantenablers for providing support to the field of multi-layernetwork management. Despite the fact that some of thesetechnologies (e.g., semantic technologies and OpenFlow)were not initially designed for the purposes of network man-agement, they have an interesting potential that can ease andaddress the management interoperability gap in multi-layerand multi-vendor settings. In this section, we will provideinsights on the potential integration of these technologies inorder to understand what makes them appealing for theirapplicability in the area of coordinated Multi-Layer NetworkManagement (MLNM). It is worth highlighting that given theimpact of OpenFlow, this technology will be further analyzedin Section VI-C as part of the Software Defined Networking(SDN) umbrella.

NETCONF

To fill the existing gap in the configuration of IP networkequipment, in early 2003 the IETF created a working group to


Approach Advantages Drawbacks

Integrated

• It reduces the management and operational costs.• It centralizes management tasks (single point of opera-

tion).• It provides overall view of the multi-layer network infras-

tructure.

• It requires big changes in the network structure and incurrent practices.

• It is hard to adapt to meet future application demands.• It hardly deals with multi-vendor interoperability issues.• Its adoption disrupts the networks’ regular activities.• It is difficult to deploy.• It requires to be integrated in all layers.• It has high deployment costs.

Coordinated

• It requires neither big changes in the network structure norin current operational practices and business workflows.

• It can be easily adapted to any type of environment tomeet the future application demand.

• It can easily deal with the multi-vendor interoperabilityissues.

• Its adaption does not disrupt the networks regular activi-ties.

• Its deployment is easy and needs only to be interfacedwith the IP and the transport NMSs.

• It allows functionalities, which do not exist in the controlplane frameworks.

• Its functions are domain and layer independent.

• Cannot directly manage the networks nor it can makechanges to the IP layer or the transport layer without theintervention of the NMSs at each layer.

• Its normal workings depend on the responsiveness of thirdparty systems (e.g., PCE) as well as the IP and the trans-port layer NMS.

TABLE II: Advantages and drawbacks of integrated vs. coordinated multi-layer network management solutions.

define and develop a standard configuration protocol. The re-sult of this effort is the Network Configuration Protocol (NET-CONF), specified in [16] as an IETF standard. NETCONF wasdefined to cope with the needs of providing configuration statemaintenance, transactional-safe operations across multiple de-vices, separation of configuration from operational data, con-currency, consistency, and support to multiple configurations,in a standard and easy way of use. This set of requirementsgather the most remarkable characteristics of CLI-based mech-anisms along with the desired features of network providersin the scope of configuration management [98].

Some of the key features of NETCONF are: (i) the abilityto distinguish configuration data from operational data, i.e.,variables that can be set by the administrator from statistics,alarms, notifications, etc.; (ii) support to transactions, meaningthat, it ensures completion of configuration tasks—not only ona device basis but even, on a network basis—otherwise, roll-back operations are automatically performed; (iii) transportprotocol independence; (iv) support to configuration locking;and (v) filtering mechanisms that enable selected retrieval ofconfiguration data [19]. Most importantly, NETCONF featuresautomated ordering of operations, which means that the com-plexity of task sequence ordering is pushed from the opera-tor’s side into the device. Its design is flexible and extensi-ble enough to be implemented and deployed by all vendors.Note that this is not an advantage over other protocols, suchas SNMP, but is a must if NETCONF intends to become awidely adopted standard.

The complexities of SNMP and the proprietary natureof CLI-based mechanisms, have pushed toward the initialimplementation of NETCONF in IP/MPLS vendor’s equip-ment (e.g., Juniper and Cisco) as a means to guarantee theperformance of the configuration procedure. The inclusion of

NETCONF will enable operators to deal in a standard andreliable way with multi-vendor infrastructures, significantlydecreasing the operational expenditures. However, as exposedearlier in Section II-B, the definition of network managementinformation within NETCONF raises clear interoperabilityissues that still represent an important limitation to its wideacceptance and deployment. Most recently, YANG [41]—theIETF’s proposal for a standard data model—has becomethe strongest candidate for the formal representation ofnetwork configuration management information. It provideswell-defined abstractions of the network resources that canbe configured or manipulated by a network administrator,including both devices and services. Currently, the IETF isworking on the definition of standard YANG modules, towhich vendors are expected to comply with in the future.To date, several internet-drafts have been introduced for thedefinition of the interface, IP, routing, SNMP and systemmanagement data models [99]. However, there is still along path before NETCONF and YANG become maturetechnologies and established as the default standard for IPnetwork management configuration. Though, the future ofNETCONF and YANG is promising and the interest ofnetwork device vendors is growing, almost eight years afterits initial proposal, CLI-based mechanisms continue to bethe preferred way for configuring network devices. For thisreason, and based on the fact that a solution is required forcurrent network settings, we envision semantic technologiesas a promising enabler for the integration of dissimilarprotocols in the scope of network management. We proceedto briefly describe the potential of these technologies in thescope of a coordinated platform dealing with heterogeneousprotocols.


Semantic Adaptations

Semantic adaptations serve as a promising approach for en-abling future coordination of multi-layer management. Whencombined with current available technologies (e.g., ontologies,data mining, artificial intelligence, natural language process-ing, etc.), semantic approaches can provide the means forachieving true interoperability in the configuration manage-ment of multi-vendor environments.

Semantic technologies can be envisioned as integratorsacross multi-vendor scenarios, providing the means for car-rying out the mappings between data formats (e.g., CLI com-mands) of different device manufacturers based on the com-mon semantics which define the configuration domain. In thecontext of IP device configuration, the semantics could be ex-tracted, for example, from the “help set” provided to usersin the form of text through the CLI environment or config-uration manuals [97]. Regardless of the heterogeneity of de-vice and configuration environments, configuration commandsshare common semantics (i.e., meaning) intrinsic to the do-main of interest. Indeed, semantic approaches can significantlycontribute to bridge the gap between heterogeneous devices byproviding an agnostic view of configurable network resources.

The benefits that semantic approaches can provide go be-yond the scope of seamless device configuration, and can bedevised for any other scope in which the existence of di-verse protocols or mechanisms lead to heterogeneous scenar-ios. They provide an evolutionary solution to current MVIproblems through an intelligent, flexible, and extensible ap-proach. Semantic adaptations can thus improve and enhancethe experience of the final user, abstracting network managersfrom the particularities of each language or format, therebyremoving the complexity that is left in the hands of operators(which is error-prone and highly restricts the level of automa-tion of management tasks).

Semantic technologies can be combined with machine learn-ing techniques [101] to develop software agents capable of se-mantically and syntactically adapting configurations for easingtasks in multi-layer and multi-vendor environments. In the lit-erature, several research efforts can be found aligned with theuse of semantic technologies to address the problems inher-ent to the network management domain [102], [103], [104],[105]. For example, in [103], [104] H. Xui and D. Xiaoconsider the application of ontology languages to bring in-telligence into the network management plane to enable au-tomated configuration. Authors propose a general model inwhich three ontology-related languages are considered. Theypropose, firstly, the Ontology Web Language (OWL) [43] formodeling the domain knowledge, secondly, the Semantic WebRule Language (SWRL) [106] to add behavior information—such as axioms and constraints—and finally, OWL-S [107]—asemantic markup language for Web Services—to automate theexecution logic based on the use of Web-Services. In short,authors envision the use of Semantic Web Services for automa-tion of network management and propose standardization ofnetwork management information on this basis. Moreover, inthe work led by J. Lopez et al. in [105], authors thoroughlyanalyze and compare several management information defini-

tion languages on the basis of their semantic expressiveness.The goal is to integrate different management models basedon ontological mapping and merging techniques to create aglobal management ontology encompassing a huge set of in-formation models (e.g., Structure of Management Informationversion 2 - SMIv2, Managed Object Format / Common Infor-mation Model - MOF/CIM, etc.). Finally, A. K. Y. Wong etal. [102] proposed a generic ontology-driven solution to theinteroperability problem in network management. In this ap-proach, ontology concepts are matched based on the similaritymeasure obtained through a novel function developed by theauthors.

As seen, semantic technologies have already been consid-ered in previous research studies to address the interoperabilityissues in network management as well as in many otherfields (e.g., health care, product design, biomedicine, etc.).Nevertheless, there are still many open research lines thatmust be explored in order to exploit and properly use theresources offered by device manufacturers.

MTOSI

In the field of optical transport networks, MTOSI [17] hasemerged with great force as a technology for enabling standardcommunication between multiple OSS (e.g., Element Manage-ment Systems, Network Management Systems, Service Ac-tivation Systems, Fault Management Systems, etc.), coveringboth service and resource management. This standard has beenspecified by the TeleManagement Forum (TMF) to providea common interface to manage multi-technology networks.It is an XML/Web Service-based interface, that is indepen-dent from the underlying transport technology. MTOSI enablesinteroperability between the Service, Network and ElementManagement Layers of the Telecommunications ManagementNetwork (TMN) layering model.

The general specifications of the interface maintain thatMTOSI will be used between OSSs within the same admin-istration [100]. Hence, it covers the interoperability require-ments of current Service Providers which use multiple man-agement systems to manage complex multi-vendor and multi-technology networks.

To illustrate the MTOSI reference architecture let us con-sider the example shown in Fig. 11. This figure provides aview of the communication between different OSSs enabledthrough MTOSI at different levels. The interaction betweenthe management systems is done over the Common Commu-nication Vehicle (CCV). As depicted in the figure, in typicalnetwork settings (i.e., a multi-vendor scenario) a NMS—orother type of OSS—directly connected to the CCV can beat the same time managing underlying Element ManagementSystems (EMSs), which in turn manage network elements ofa single vendor based on other management protocols (e.g.,SNMP, TL1, CMIP, etc.). For this example, a Fault Manage-ment System providing an MTOSI-based interface can retrieveinventory from an Inventory OS to effectively fulfill its man-agement functions.

During the last few years, MTOSI seemed to become thefuture standard for enabling interoperability between OSSs in


Common Communications Vehicle -

TL1 SNMP SNMP SNMP

Fig. 11: MTOSI reference architecture example (adapted from “Framework DDP BA TMF518 FMW Version 1.2” [100]).

the transport layer. This web-based interface was positioned asa rather appealing technology for overcoming the limitationsin the scope of network management. However, a change inperspective in the field has made MTOSI lose momentum andinstead, attention has turned most recently to Software De-fined Networking (SDN), a new emerging technology with thepower for taking the management solution to the next level.

SDN has become an increasingly popular concept whosepotential certainly opens the field for research and innovation.For this reason, in the next section we will provide insights onSDNs capabilities and on the challenges from a managementpoint of view.

C. Software Defined Networks

We could say that Software Defined Networking (SDN)is the current fulfillment of an old-time promise: providingthe possibility of programmability of network functionalities,while offering a clear path for network management to followthe same direction of other Information Technology (IT) fieldstoward virtualization.

The idea of a flexible real-time control of network functionshas been considered several times in the past, with proposalsranging from the use of “programming packets” to transmit thedesired behavior to network boxes, up to the implementationof adaptive control both in software and hardware. However,the combination of radical decoupling and open interfaces thatconstitutes the kernel of SDN is a novel proposal that hasgained a strong momentum, especially, with the advent of aprotocol that demonstrated the feasibility of this approach, andallowed the deployment of real-scale SDN-based networks:OpenFlow [18].

The complexity and lack of flexibility of standard networkdevices has made network experimentation and innovationhighly difficult at all scales for academic researchers. Anychange to the software embedded in each device had to be

coordinated between vendors in order to make the distributedcontrol algorithms interoperable. Therefore, evolving at thepace required by research and experimentation was extremelydifficult. In this context, OpenFlow was born as a cornerstone.The first step was to develop the ability to program switchesfrom a remote controller. Realizing that this implied externalsoftware-based control of the data plane, bypassing traditionalL2 and L3 protocols and associated configurations, was a nat-ural consequence.

Software Defined Networking relies on two main assump-tions. The first is a radical separation between the control andthe data planes, located in two (most often physically) indepen-dent entities: the controller, in charge of the control plane, andthe switches, responsible for the functions in the data plane.The choice of singular and plural in the definition above iscompletely intentional: although not required by the model,the obvious deployment consists of a single controller tak-ing care of several switches in a certain realm. The secondassumption is the availability of an open protocol betweencontrollers and switches, allowing for a free combination ofelements from different vendors to provide network functions,and of an open interface to the control plane, so the controllercan be uniformly accessed by other components participatingin the network, such as sources of network intelligence or ap-plications in general.

The most widely deployed SDN protocol, OpenFlow, isbased on the definition of rules from the controller to be ap-plied by the switches when receiving packets. Rules are firedby matching certain parts of the packets (or the path they ar-rive through), and contain actions to be applied to those pack-ets, such as forwarding them to a certain path, making somechanges to them, or even discarding them.

In summary, in SDN control decisions are taken by a cen-tral element, while switching decisions are actually appliedby distributed elements. A common protocol allows the con-troller to communicate its decisions to the switches. Having


this central element translates into the possibility of abstract-ing the network into a single element, as it becomes the one incharge of the whole network behavior. Furthermore, the com-mon protocol acts in a similar way to a processor instructionset controlling its registries, processing units and peripherals,and therefore the network becomes a programmable entity,suitable to be controlled in the same way as any other ele-ment in the whole computing infrastructure.

A general architecture for SDN-based network managementis proposed in [108] (shown in Fig. 12), where an SDN Medi-ator communicates with applications and services, and trans-lates requests to the physical network components. This me-diator relies on a database containing network topology andcomponent information, and it is able to provide a fully virtu-alized view to applications and services. The mediator controlsseveral processes:

• Discovery. It enables the SDN users to discover and reg-ister to the Service Mediator. As a part of the discoveryprocess, the SDN users may negotiate capabilities withthe service mediator.

• Provisioning. It allows the Service Mediator to provisionthe underlying network resources. While in principle theprovisioning process should rely on OpenFlow, it is con-ceivable the use of other protocols to create or adjusttraffic engineering connections.

• Monitoring. This allows the Service Mediator to interfacewith the underlying network to gather topology informa-tion at an abstract level, and detect the network failuresthat may impact applications and services.

A more radical approach is taken by a group of the originalOpenFlow proponents [111], who present a unified controlfor packet and transport networks, claiming that with sepa-ration of data and control, and the treatment of packets asflows, together with the introduction of circuit-flow featuresin the OpenFlow protocol, a unified architecture becomes re-alizable for converged packet-circuit networks. OpenFlow ab-stracts each data-plane switch as a flow table. It allows the

IP Layer

Transport Layer

Applications and Services

Provisioning

Discovery

Network Topology & Resource Database

SDN Service Mediator

Monitoring

Fig. 12: Mediator architecture for SDN-based management.

definition of a flow to be any combination of L2-L4 packetheaders for packet flows, as well as L0-L1 circuit parametersfor circuit flows.

Whether a mediator-based or a SDN-only approach is fol-lowed, it is important to remark four salient characteristicsimplied by the use of SDN for multi-layer network manage-ment:

• Since the network elements are controlled via a (few) uni-form protocol(s), the management database can be greatlysimplified.

• Management commands and/or configuration are alwaystranslated into pairs of the type (device,rule), andit is possible to override layer separations, device originalfunctions, and other potential limitations in other models.A multi-layer management plane is a natural consequenceof applying SDN, even if not a full integration is pursued.

• The northbound API can be adapted to provide differentmanagement views to the application services, not limitedby individual elements, links or layers being managed.The network functions can be fully virtualized.

• This virtualization possibilities translate into an easy ex-tensibility of the configuration programmatic interface,able to being updated according to the application/serviceneeds, even in a real-time or ad-hoc manner.

Another important aspect is that with the advent of SDNs,the range of multi-layer capabilities has significantly increased,since open programmability enables that new applications cantake control of network resources across all layers in the OSIstack (cf. Fig. 1 ). Observe that, in this paper, we have focusedon multi-layer capabilities across the transport and networklayers of a typical carrier-grade network, but SDNs also bringa plethora of possibilities to upper layers, allowing applica-tions to intelligently combine network functions at differentlayers. Consider for instance some of the recent research ad-vances combining the utilization of OpenFlow and the Multi-Path Transport Control Protocol (MPTCP). In this particularexample, the goal is to intelligently and transparently assigntraffic to multiple paths in order to improve the resilience, thestability, and the performance of different types of communica-tions [112], [113], [114], [115], [116]. Another interesting ap-proach can be found in [117], where a cross-layer cooperationmodule is defined between MPTCP and the Locator/IdentifierSeparation Protocol (LISP). In this example, the goal of theauthors is to improve both the performance and the transfertime between the endpoints.

An example involving even a higher number of layers canbe found in [118], wherein the authors propose a methodol-ogy to extend Open VSwitch (OVS)—an open-source Open-Flow switch—to support L4-L7 service awareness. Moreover,SDN for transport networks [119] [120] —in correspondenceto our earlier discussions—has recently gained a lot of at-tention, and is considered one of the potential candidates forenabling packet/optical integration, and thereby, improve ex-isting multi-layer path provisioning techniques [121], [122],[123]. Indeed, there are ongoing efforts within the Open Net-working Foundation (ONF), for standardizing transport exten-sions to OpenFlow so as to enable SDN in optical networks.


Transport Layer

2009 2010 2011 2012 2013 …

Transport Layer

T-NMS T-NMS

T-NMS

OpenFlow Controller

Fig. 13: Network management trend: SDN over MTOSI.

In the meanwhile, several approaches to SDN for the transportlayer have already been developed. For example, ADVA, in ajoint effort with IBM and the Marist College, have recentlydemonstrated an SDN solution for transport networks, mainlytargeting the dynamic set up and tear down of wavelengths be-tween data centers. Overall, SDN has become a revolutionaryparadigm shift in the telecom field, and it is expected to havea significant impact on our conception of networking. The po-tential is remarkable, but the status of SDN-based applicationsenabling multi-layer functions in carrier-grade networks is stillin a very early stage of development, so substantial researchis needed before we can start witnessing the deployment ofsolutions in operational networks.

It is unquestionable that the advent of SDNs has weakenedthe power of MTOSI, which, only a few years ago, was posi-tioned as the predominant trend for enabling transport networkmanagement interoperability. The paradigmatic differences be-tween the two approaches has shifted the tendency towardSDNs. As shown in Fig. 13, this shift suggests that, in thefuture, the interest will no longer be focused on the communi-cation between multiple OSS (i.e., the MTOSI paradigm), butinstead, the new paradigm will be the one-to-multiple approachfollowed by SDNs for achieving true network programmabilityand virtualization through open and clear APIs.

VII. CONCLUSION AND LESSONS LEARNED

The multi-layer core infrastructures of large ISPs haveevolved as administratively separated ecosystems. Thisbusiness-driven separation has led to the operational and man-agerial isolation of both layers. The lack of mechanisms en-abling communication from a management perspective hasclearly derived in interoperability issues between layers. Sev-eral initiatives can be found in the fields of control planeand data plane technologies, though none of these is capa-ble of addressing the needs and requirements that arise fromthe management point of view. In addition to this, the emer-gence of new trends in the field of IP over Optical transport

(e.g., hybrid nodes, proprietary multi-layer NMSs, etc.) posenew challenges in the management of heterogeneous networkenvironments.

The integration and coordination of network managementsolutions in the context of multi-layer networks are amongthe most predominant approaches for overcoming the isola-tion between the management ecosystems. These approachescan enable inter-layer interoperability, which can in turn sig-nificantly reduce the operational and capital expenses whilefacilitating a number of complex orchestrations required formanagement operations. Our research has identified that co-ordinated approaches seem to be the most suitable alternative,especially, for achieving automation of cross-layer operations,avoiding the redundancy of management functions, reducingoperational and capital expenses, providing a smooth evolu-tionary practice, while offering higher flexibility. Examplessuch as the ONE Adapter in [97], as well as the mediationscheme for SDN-based management presented in Fig. 12, areindicative initiatives that envision the direction of a mediationmodel for truly accomplishing interoperability through the co-ordination of both layers.

In Table III we summarize the obstacles and pitfalls, thepaths toward solutions, and the lessons learned in the man-agement field of multi-layer networks. The aim is to providea closer look into the main challenges, and summarize theissues that must be addressed by the research community interms of future multi-layer management solutions. The futureof network management for multi-layer network settings isnot a clear panorama. However, the emergence of NETCONF,MTOSI, SDN supported through OpenFlow, as well as theconsolidation of Web Services and many other new trends inthe field of IP over Optical, are key indicators that a change ofperspective is required to address the isolation of managementecosystems and improve multi-layer performance.

ACKNOWLEDGEMENTS

This work was supported in part by the European Com-mission through the ONE project in the FP7 Program under


contracts number INFSO-ICT-258300. UPC authors also ac-knowledge the support received from the Spanish Ministryof Science and Innovation under contracts TEC2009-07041and TEC2012-34682, and by the Catalan Government undercontract 2009 SGR1508. Seoul National University authorshave been funded by International Research & DevelopmentProgram of the National Research Foundation of Korea ofthe Ministry of Education, Science and Technology of Korea(Grant number: K21001001625-10B1300-03310).

REFERENCES

[1] J. Gabeiras, V. Lopez, J. Aracil, J. Fernandez-Palacios, C. Garcıa,O. Gonzalez de Dios, F. Jimenez Chico, and J. Hernandez. Is multilayernetworking feasible? Optical Switching and Networking, 6(2):129–140,April 2009.

[2] T. Lehman, X. Yang, N. Ghani, F. Gu, C. Guok, I. Monga, and B. Tier-ney. Multilayer Networks: An Architecture Framework. IEEE Com-munications Magazine, pages 122–130, May 2011.

[3] A. Ahmad, A. Bianco, E. Bonetto, L. Chiaraviglio, and F. Idzikowski.Energy-aware design of multilayer core networks [invited]. OpticalCommunications and Networking, IEEE/OSA Journal of, 5(10):A127–A143, Oct 2013.

[4] F. Idzikowski, E. Bonetto, L. Chiaraviglio, A. Cianfrani, A. Coiro,R. Duque, F. Jimenez, E. Le Rouzic, F. Musumeci, W. Van Heddeghem,J. Lopez Vizcaino, and Yabin Ye. Trend in energy-aware adaptive

Obstacles and Pitfalls Paths toward Solutions Lessons Learned

Absence of standards and/orbroadly accepted mechanismsto enable network managementinteroperability

• An early solution to the interoperabil-ity problem in network management wasSNMP [33]. Despite being the de-factoprotocol for network monitoring in IP-based networks, SNMP has failed to ful-fill other scopes of network management,such as device configuration.

• Most recently, NETCONF [16] is en-visioned for device configuration andMTOSI [17] for OSS interoperability intransport networks.

• OpenFlow has arised in the context ofSDNs [18].

The absence of standard protocols prevents automation of cross-layermanagement operations, leading to manual management of currentmulti-layer infrastructures. Standard protocols for network monitoring(e.g., SNMP and most recently Web Services) and network device con-figuration (e.g., NETCONF or OpenFlow) are not sufficient to enablemanagement interoperability across multi-layer platforms. For instance,Network Managers require means (i.e., mechanisms, platforms) to coor-dinate between both layers to optimize resource usage, avoid duplicationof network functions and automate/execute cross-layer workflows.

Segmentation of Standardiza-tion Bodies

• For instance, MPLS-TP [58]

Solutions to multi-layer issues require of standardization bodies behindeach domain (i.e., IP and Optical) to be aligned and to develop jointefforts to generate fully-compliant requirements and solutions to bothtechnology layers. An example to this, is the MPLS-TP technologywhich begun as an ITU-T effort under the name of T-MPLS. However,IETF—the developers of MPLS standards—determined a set of incon-sistencies between T-MPLS and native MPLS. They requested to extendthe IETF’s MPLS technologies to packet transport networks through theIETF Standards Process in a joint effort between both parties to con-solidate a solution aligned to the IETF standards and fulfilling ITU-Trequirements.

Poor Coordination across lay-ers

• In-house developments are early attemptsto coordinate multi-layer tasks in a static(i.e., pre-configured) way.

• Most recently, the ONE Adapter [97] hasapproached multi-layer management in acoordinated fashion.

• Coordinated systems based on SDN’s(mediator models).

The lack of coordination between layers in multi-layer networks has ledto duplication of network functions and long provisioning timescales. In-house developments have been for long time a way to flatten the issuesin multi-layer management. However, the lack of flexibility makes themuseful in scenarios where fixed solutions are sufficient. However, multi-layer management requires much more flexible, dynamic, programmable(i.e., configurable) solutions capable of overcoming the barriers of layersegmentation.

Lack of Cross-Layer NetworkManagement Automation

• For instance, GMPLS control plane [9].

GMPLS is positioned as the unified control plane solution for dynamicpath provisioning in multi-layer networks. However, neither these so-lutions are widely deployed nor control plane technologies are capa-ble of addressing all the needs and requirements from a managementperspective (e.g., proactive execution of policy-based workflows withcross-layer components, IP device configuration, etc.).

Limitations for Proactive En-forcement of Policy-based Man-agement in Multi-Vendor set-tings

• In-house developments (i.e., umbrellaNMSs).

Network programmability is a must in order to develop scalable solu-tions capable of adapting to the changing nature of network behavior.Network administrators require of flexible solutions that enable on-the-fly policy enforcement to bring dynamics to the multi-layer environ-ment.

Overburden of multi-layer func-tions which add on the basisof their complexity (e.g., compu-tation complexity) - (embeddedcomplexity)

• Outsourcing of multi-layer functions, ex-amples are the Path Computation Ele-ment (PCE) [29] and the Virtual NetworkTopology Manager (VNTM) [52].

• Virtualization and Network FunctionsVirtualization (NFV) [109].

The advent of third-party (i.e., external) management subsystems, rep-resent a unique opportunity to be integrated into future solutions to helpsolving cross-layer issues. For example, a PCE for outsourcing compu-tation of multi-layer paths. In this field, yet some issues still remainopen. For instance, the communication protocol between entities forcoordination of cross-layer functions (e.g., the communication betweenthe NMS and the VNTM in a cooperative model for multi-layer pathcomputation has not been yet defined).


Obstacles and Pitfalls Paths toward Solutions Lessons Learned

Inadequate discovery mecha-nisms of Inter-layer connections

• Manual Topological Databases.Manual topological databases are error-prone, hard to maintain andpresent poor scalability features. Inter-layer discovery remains an openresearch challenge and requires of automated solutions capable of over-coming the restricted view (i.e., shared information) between layers.

Deficient mechanisms for seam-less network device configura-tion (e.g., the preferred config-uration mechanism in IP-basednetworks is CLI)

• NETCONF [16], a standard-based solu-tion to the configuration issue in IP-basednetworks.

• OpenFlow [18].

Standardization of network management protocols is not sufficient toovercome the configuration heterogeneity issue in multi-vendor environ-ments. In this view, standardization of data models is equally relevant tothe configuration domain to comply with a standard view of the networkelements.

Organizational Barriers (i.e.,Department segmentation)

• Integrated Network Management Solu-tions, e.g., Cyan’s CyMS [50] or Juniper’sPTX hybrid node [110].

Integrated solutions have important implications on current practices.On the one hand, the traditional separation between the IP and OpticalDepartments is reluctant to a game change from the operational, func-tional and business perspectives. To this end, emerging solutions shouldseek for non-disruptive approaches from the business model point ofview. On the other hand, current integrated solutions are subject to sin-gle vendor scenarios, a non-desired feature by network operators.

High Operational and Capi-tal Expenditures for managingMulti-Layer Settings

• Network Functions Virtualization (NFV)[109] and Software Defined Networking(SDN) [108].

• IPoDWDM solutions integrate transpon-ders directly into the IP routers enablingIP equipment to transmit ITU-compatiblecoloured wavelengths directly to the op-tical gear.

IPoDWDM solutions claim to reduce costs generously due to simplifi-cation of the network, at the cost of moving lower layer complexitiesto the upper layer, since routers have never had to deal with wave-length issues. This is somehow debatable as an assertion of this typedepends on many factors (e.g., is implementation-dependent). Anyway,newly emerging solutions require to lower OPEX and CAPEX whilemaintaining the simplicity of network operation. Moreover, featured vir-tualization functions are undoubtedly of huge interest for ISPs, and willpotentially contribute to significantly lower costs (CAPEX), while thebenefits that SDNs technologies bring along to the field of networkmanagement will impact on the OPEX.

TABLE III: Obstacles and pitfalls, paths toward solutions, and lessons learned for managing multi-layer and multi-vendorsettings.

routing solutions. Communications Magazine, IEEE, 51(11):94–104,November 2013.

[5] G. Rizzelli, A. Morea, M. Tornatore, and O. Rival. Energy efficienttraffic-aware design of on-off multi-layer translucent optical networks.Computer Networks, 56(10):2443 – 2455, 2012. Green communicationnetworks.

[6] M.Z. Feng, K. Hinton, R.W.A. Ayre, and RodneyS. Tucker. Energyefficiency in optical ip networks with multi-layer switching. In Opti-cal Fiber Communication Conference and Exposition (OFC/NFOEC),2011 and the National Fiber Optic Engineers Conference, pages 1–3,March 2011.

[7] D. Comer. Internetworking With TCP/IP Volume 1: Principles Proto-cols, and Architecture, 6th edition . Addison-Wesley, 2013.

[8] R. Munoz Gonzalez. GMPLS-based distributed lightpath provision-ing and protection schemes for metropolitan R-OADM DPRings. PhDthesis, Universitat Politecnica de Catalunya, 2005.

[9] E. Mannie. Generalized Multiprotocol Label Switching (GMPLS) Ar-chitecture. RFC 3945, IETF, October 2004.

[10] ITU-T. G.8080/Y.1304: Architecture for the automatically switched op-tical network (ASON). International Telecommunications Union, 2012.

[11] T. Benson, A. Akella, and D. Maltz. Unraveling the complexity ofnetwork management. In Proceedings of the 6th USENIX symposium onNetworked systems design and implementation, NSDI’09, pages 335–348, Berkeley, CA, USA, 2009. USENIX Association.

[12] M. Subramanian, T.A. Gonsalves, and N.U. Rani. Network Manage-ment: Principles and Practice. Dorling Kindersley, 2010.

[13] L. Mao. IP backbone calls for unified NMS. Huawei Communication,2008.

[14] M. Yannuzzi, X. Masip Bruin, O. Gonzalez de Dios, C. Garcıa Argos,M. Maciejewski, and J. Altmann. Bridging the Interoperability GapBetween the Internet and Optical Network Management Systems. InNetwork and Optical Communications (NOC) Conference, 2011.

[15] European Information and Communications Technology Industry As-sociation. EICTA Interoperability White Paper. 2004.

[16] R. Enns, M. Bjorklund, J. Schoenwaelder, and A. Bierman. NetworkConfiguration Protocol (NETCONF). RFC 6241, IETF, June 2011.http://tools.ietf.org/html/rfc6241.

[17] Telemanagement Forum MTOSI Web Page. http://www.tmforum.org/MultiTechnologyOperations/2319/home.html.

[18] Open Networking Foundation. OpenFlow Switch Specification. Version1.1.0. http://www.openflow.org/documents/openflow-spec-v1.1.0.pdf.

[19] C. Chappell. The Business Case for NETCONF/YANG in NetworkDevices. White Paper, Heavy Reading, October 2013.

[20] T. Nadeau and K. Gray. SDN: Software Defined Networks An Author-itative Review of Network Programmability Technologies. O’ReillyMedia, 2013.

[21] Software-Defined Networking: The New Norm for Networks. WhitePaper, Open Networking Foundation (ONF), April 2012.

[22] The Internet Engineering Task Force Official Web Site. http://www.ietf.org/.

[23] The International Telecommunication Union Official Web Site. http://www.itu.int/.

[24] The Optical Internetworking Forum Official Web Site. http://www.oiforum.com/.

[25] Telemanagement Forum Official Web Site. http://www.tmforum.org/.[26] F. Munoz, V. Lopez, O.G. de Dios, and J.P. Fernandez-Palacios. Multi-

layer restoration in hierarchical IP/MPLS over WSON networks. InNetworks and Optical Communications (NOC), 2012 17th EuropeanConference on, pages 1–6, June 2012.

[27] Juniper PTX Official Site. http://www.juniper.net/us/en/dm/supercore/.[28] Huawei’s OptiX OSN 7500 II. http://www.huawei.com/en/products/

transport-network/hybrid-mstp/osn7500II/index.htm.[29] A. Farrel, J. P. Vasseur, and J. Ash. A Path Computation Element-

Based Architecture. RFC 4655, IETF, August 2006. http://tools.ietf.org/rfc/rfc4655.txt.

[30] Infonetics Research - Service Provider Routers andSwitches report. http://www.infonetics.com/pr/2013/2Q13-Service-Provider-Routers-Switches-Market-Highlights.asp.

[31] A. Gupta. Network Management: Current Trends and Future Perspec-tives. Journal of Network and Systems Management, 14(4):483–491,December 2006.

[32] T. McElligott. The challenge of managing multivendor networks.Billing and OSS World, 2008.

[33] J. Case, M. Fedor, M. Schoffstall, and J. Davin. Simple NetworkManagement Protocol (SNMP). RFC 1157, Internet Engineering TaskForce, May 1990.

[34] D. Mauro and K. Schmidt. Essential SNMP, 2nd Edition. O’ReillyMedia, 2005.


[35] JUNOS OS NETCONF XML Management Protocol Guide, Release11.4. Technical Documentation, JUNIPER, October 2011.

[36] Cisco Networking Services Configuration Guide - Network Configura-tion Protocol. Technical Documentation, CISCO, March 2013.

[37] Tail-f Systems: Build On-Device Management Applications withConfD. http://www.tail-f.com/on-device-configuration-management/.

[38] S. Chisholm, A. Clemm, and J. Tjong. Using XML Schema to defineNETCONF Content. Internet-Draft, Network Working Group, 2008.

[39] H. Cui, B. Zhang, G. Li, X. Gao, and Y. Li. Contrast Analysis of NET-CONF Modeling Languages: XML Schema, Relax NG and YANG. InCommunication Software and Networks, 2009. ICCSN ’09. Interna-tional Conference on, pages 322–326, 2009.

[40] L. Johansson. NETCONF Configuration Data Modeling using OWL.Internet-draft, Internet Engineering Task Force (IETF), 2008.

[41] M. Bjorklund. YANG - A Data Modeling Language for the NetworkConfiguration Protocol (NETCONF). RFC 6020, IETF, October 2010.http://www.ietf.org/rfc/rfc6020.txt.

[42] H. Xu and D. Xiao. Data modeling for NETCONF-based networkmanagement: XML schema or YANG. In Communication Technology,2008. ICCT 2008. 11th IEEE International Conference on, pages 561–564, 2008.

[43] OWL Web Ontology Language Overview. http://www.w3.org/TR/owl-features/.

[44] Cisco Systems. Cisco active network abstraction 3.7.2 theory of oper-ations guide. Technical report, Cisco Systems, 2012.

[45] IPoDWDM. http://www.cisco.com/en/US/netsol/ns1192/networkingsolutions solution.html.

[46] Juniper Networks. White paper: Improving network efficiency, relia-bility, and operations with IPoDWDM. Technical report, Juniper Net-works, 2010.

[47] NTT Com Demonstrates GMPLS over 40 Gbps Network. Technicalreport, NTT Communications, June 2006.

[48] S. Das, G. Parulkar, and N. McKeown. Why OpenFlow/SDN Can Suc-ceed Where GMPLS Failed. In European Conference and Exhibitionon Optical Communication, volume 1, 2012.

[49] International Telecommunication Union (ITU). RecommendationG.698.2: Amplified multichannel dense wavelength division multiplex-ing applications with single channel optical interfaces, November 2009.

[50] CyMS Multi-Layer Management System. http://cyaninc.com/cyms/cyan-cyms.

[51] M. Bocci, S. Bryant, D. Frost, L. Levrau, and L. Berger. A Frameworkfor MPLS in Transport Networks. RFC 5921, Internet Engineering TaskForce (IETF), July 2010. http://tools.ietf.org/html/rfc5921.

[52] K. Shiomoto and et al. Requirements for GMPLS-based multi-regionand multi-layer networks (MRN/MLN). RFC 5212, IETF, July 2008.

[53] MPLS Working Group. http://datatracker.ietf.org/wg/mpls/.[54] L. Lei and S. Sampalli. Distributed Online LSP Merging Algorithms

for MPLS-TE. In Hermann Meer and Nina Bhatti, editors, Qualityof Service IWQoS 2005, volume 3552 of Lecture Notes in ComputerScience, pages 366–368. Springer Berlin Heidelberg, 2005.

[55] L. Lobo. MPLS Configuration on Cisco IOS Software. Cisco Press,2005.

[56] G. Swallow, S. Bryant, and L. Andersson. Avoiding Equal Cost Mul-tipath Treatment in MPLS Networks. RFC 4928, Network WorkingGroup, June 2007.

[57] I. Busi and D. Allan. Operations, Administration, and MaintenanceFramework for MPLS-Based Transport Networks. RFC 6371, Inter-net Engineering Task Force (IETF), 2011. http://tools.ietf.org/html/rfc6371.

[58] B. Niven-Jenkins, D. Brungard, M. Betts, N. Sprecher, and S. Ueno.Requirements of an MPLS Transport Profile. RFC 5654, NetworkWorking Group, 2009. http://tools.ietf.org/html/rfc5654.

[59] M. Vigoureux, D. Ward, and M. Betts. Requirements for Opera-tions, Administration, and Maintenance (OAM) in MPLS TransportNetworks. RFC 5860, Internet Engineering Task Force (IETF), May2010. http://tools.ietf.org/html/rfc5860.

[60] L. Andersson, L. Berger, L. Fang, N. Bitar, and E. Gray. MPLS Trans-port Profile (MPLS-TP) Control Plane Framework. RFC 6373, In-ternet Engineering Task Force (IETF), 2011. http://tools.ietf.org/html/rfc6373.

[61] S. Hubbard. MPLS-TP in Next-Generation Transport Networks. Whitepaper, Light Reading/Heavy Reading MPLS-TP Initiative, July 2011.

[62] Ethernet vs. MPLS-TP in the Access, 2012. http://www.rad.com/12/Ethernet-MPLS-TP-in-the-Access/23979/.

[63] J. Yang and S.J. B. Yoo. An integrated network management systemwith online traffic engineering for optical label switching networks.

In Global Telecommunications Conference, 2004. GLOBECOM ’04.IEEE, volume 3, December 2004.

[64] L. Raptis, G. Hatzilias, F. Karayannis, K. Vaxevanakis, and E. Grampin.An integrated network management approach for managing hybrid IPand WDM networks. Network, IEEE, 17(3):37 – 43, May-June 2003.

[65] WDM and IP Network MANagement (WINMAN). http://winman.upc.es/, 2001-2003.

[66] N. Vardalachos, E. Grampin, A. Galis, and J. Serrat. Policy Manage-ment Approach for IP over WDM Networks: A Synthesis Study. InAnnual London Conference on Communications, 2001.

[67] Recommendation ITU-T G.800: Unified functional architecture oftransport networks. ITU-T G.800, ITU-T, February 2012. http://www.itu.int/rec/T-REC-G.800-201202-I/es.

[68] R. Cafini, W. Cerroni, C. Raffaelli, and M. Savi. Standard-based ap-proach to programmable hybrid networks. Communications Magazine,IEEE, 49(5):148–155, May 2011.

[69] H. Takenouchi, R. Urata, T. Nakahara, T. Segawa, H. Ishikawa, andR. Takahashi. First demonstration of a prototype hybrid optoelectronicrouter. In Optical Communication, 2009. ECOC ’09. 35th EuropeanConference on, volume 2009-Supplement, pages 1–2, 2009.

[70] E. Oki, T. Takeda, JL. Le Roux, and A. Farrel. Framework for PCE-Based Inter-Layer MPLS and GMPLS Traffic Engineering. RFC 5623,IETF, September 2009. http://tools.ietf.org/rfc/rfc5623.txt.

[71] JP. Vasseur and J.L. Le Roux A. Path Computation Element Commu-nication Protocol. RFC 5440, IETF, March 2009. http://tools.ietf.org/rfc/rfc5440.txt.

[72] IETF PCE Working Group Web Page. http://datatracker.ietf.org/wg/pce/charter/.

[73] V. Lopez, B. Huiszoon, J. Fernandez-Palacios, O. Gonzalez de Dios,and J. Aracil. Path computation element in telecom networks: Recentdevelopments and standardization activities. In Optical Network Designand Modeling (ONDM), 2010 14th Conference on, pages 1–6, 2010.

[74] C. Margaria, O. Gonzalez de Dios, and F. Zhang. PCEP extensions forGMPLS. IETF Internet Draft, March 2012. http://datatracker.ietf.org/doc/draft-ietf-pce-gmpls-pcep-extensions/.

[75] Y. Lee, G. Bernstein, J. Martensson, T. Takeda, T. Tsuritani, andO. Gonzalez de Dios. PCEP Requirements for WSON Routing andWavelength Assignment. IETF Internet Draft, April 2012. http://datatracker.ietf.org/doc/draft-ietf-pce-wson-routing-wavelength/.

[76] JP. Vasseur and et al. A Backward-Recursive PCE-Based Computa-tion (BRPC) Procedure to Compute Shortest Constrained Inter-DomainTraffic Engineering Label Switched Paths. RFC 5441, IETF, April2009. http://tools.ietf.org/rfc/rfc5441.txt.

[77] F. Cugini, A. Giorgetti, N. Andriolli, I. Paolucci, L. Valcarenghi, andP. Castoldi. Multiple Path Computation Element (PCE) Cooperation forMulti-layer Traffic Engineering. In Optical Fiber Communication andthe National Fiber Optic Engineers Conference, 2007. OFC/NFOEC2007. Conference on, pages 1–3, 2007.

[78] M. Chamania, O. Gonzalez de Dios, V. Lopez, M. DrogonM. Cuaresma, A. Jukan, X. Masip-Bruin, and M. Yannuzzi. Coor-dinated Computation of Multi-layer Paths via Inter-layer PCE Com-munication: Standards, Interoperability and Deployment. In IEEE ICC2012 WS - From Research to Standards, 2012.

[79] A. Bukva, R. Casellas, R. Martınez, and R. Munoz. A Dynamic Path-Computation Algorithm for a GMPLS-Enabled Multi-layer Network.4:436–448, June 2012.

[80] B. Jabbari, S. Gong, and E. Oki. On Constraints for Path Compu-tation in Multi-layer Switched Networks. In IEICE Transactions onCommunications, volume E90-B, pages 1922–1927, August 2007.

[81] S. Gong and B. Jabbari. Optimal and Efficient End-to-End Path Com-putation in Multi-Layer Networks. In Communications, 2008. ICC ’08.IEEE International Conference on, pages 5767–5771, 2008.

[82] X. Yang, T. Lehman, K. Ogaki, and T. Otani. A study on cross-layermulti-constraint path computation for IP-over-optical networks. In Pro-ceedings of the 2009 IEEE international conference on Communica-tions, ICC’09, pages 2247–2252, Piscataway, NJ, USA, 2009. IEEEPress.

[83] P. Fodor, G. Enyedi, G. Retvari, and T. Cinkler. Layer-preference poli-cies in multi-layer GMPLS networks. In Photonic Network Commun.,volume 18, pages 300–313, February 2009.

[84] F. Dijkstra, J.V.D. Ham, P. Grosso, and C. de Laat. A path findingimplementation for multi-layer networks. Future Generation ComputerSystems, 25(2):142 – 146, 2009.

[85] F. Kuipers and F. Dijkstra. Path selection in multi-layer networks.Comput. Commun., 32(1):78–85, January 2009.


[86] A. Rahat Rubuyat. Path Computation Element in GMPLS EnabledMulti-layer Networks. Masters’ Degree Project, KTH Electrical Engi-neering, 2006.

[87] D. Eppstein. Finding the k shortest paths. SIAM J. Comput., 28(2):652–673, February 1999.

[88] V. M. J. Pelayo and A. M. Varo. Computing the k shortest paths: Anew algorithm and an experimental comparison. In Proceedings 3rdInternational Workshop Algorithm Engineering, number 1668, pages15–29, 1999.

[89] E. Q. V. Martins and M. M. B. Pascoal. A new implementation ofYEN’s ranking loopless paths algorithm. 1:121–134, January 2003.

[90] H. Zhu, H. Zhang, K. Zhu, and B. Mukherjee. A novel genericgraph model for traffic grooming in heterogeneous WDM networks. InIEEE/ACM Transactions on Networking, volume 11, pages 258–299,2003.

[91] W. Yao and B. Ramamurthy. A link budled auxiliary graph modelfor constrained dynamic traffic grooming in WDM mesh networks. InIEEE Journal on Selected Areas in Communications, volume E90-B,pages 1922–1927, August 2007.

[92] JL. Le Roux, JP. Vasseur, and Y. Lee. Encoding of Objective Functionsin the Path Computation Element Communication Protocol (PCEP).RFC 5541, Network Working Group, June 2009.

[93] F. Zhang and A. Farrel. Conveying Vendor-Specific Constraints in thePath Computation Element Protocol. Internet draft, Network WorkingGroup, April 2013.

[94] D. Papadimitriou and et al. Generalized Multi-Protocol Label Switch-ing (GMPLS) Protocol Extensions for Multi-layer and Multi-RegionNetworks (MLN/MRN). RFC 6001, IETF, October 2010.

[95] J. Lang. Link Management Protocol (LMP). RFC 4204, NetworkWorking Group, October 2005.

[96] A. Farrel and I. Bryskin. GMPLS - Architecture and Applications.Morgan Kaufmann, 2006.

[97] EU Project ONE. http://www.ict-one.eu, 2010-2013.[98] J. Schoenwaelder. Overview of the 2002 IAB Network Management

Workshop. RFC 3535, IETF, May 2003. http://tools.ietf.org/search/rfc3535#page-10.

[99] NETCONF Data Modeling Language (netmod) active internet drafts.http://datatracker.ietf.org/wg/netmod/, 2014.

[100] Framework DDP BA TMF518 FMW Version 1.2. Document DeliveryPackage, TeleManagement Forum, September 2011.

[101] P.C.G. Costa, C. d’Amato, N. Fanizzi, K.B. Laskey, K.J. Laskey,M. Nickles, and M. Pool. Towards Machine Learning on the Seman-tic Web. In Uncertainty Reasoning for the Semantic Web I - ISWCInternational Workshop, volume 5327 of Lecture Notes in ComputerScience, pages 282–314, Piscataway, NJ, USA, 2008. URSW.

[102] A. Ka Yiu Wong, P. Ray, N. Parameswaran, and J. Strassner. OntologyMapping for the Interoperability Problem in Network Management.IEEE Journal on Selected Areas in Communications, 23(10):2058–2068, October 2005.

[103] H. Xu and D. Xiao. Applying Semantic Web Services to AutomateNetwork Management. In 2nd IEEE Conference on Industrial Elec-tronics and Applications, pages 461–466, May 2007.

[104] H. Xu and D. Xiao. A Common Ontology-Based Intelligent Config-uration Management Model for IP Network Devices. In InnovativeComputing, Information and Control, 2006. ICICIC ’06. First Interna-tional Conference on, volume 1, pages 385–388, 2006.

[105] J. Lopez de Vergara, V. Villagri, J. Asensio, and J. Berrocal. Ontologies:Giving Semantics to Network Management Models. IEEE Network,17(3):15–21, May 2003.

[106] I. Horrocks, P. F. Patel-Schneider, H. Boley, S. Tabet, B. Grosof, andM. Dean. SWRL: A Semantic Web Rule Language Combining OWLand RuleML. http://www.w3.org/Submission/SWRL/.

[107] D. Martin, M. Burstein, J. Hobbs, O. Lassila, D. McDermott, and et al.OWL-S: Semantic markup for web services, 2004. http://www.w3.org/Submission/OWL-S/.

[108] P. Pan and T. Nadeau. Software-Defined Network (SDN) ProblemStatement and Use Cases for Data Center Applications. draft-pan-sdn-dc-problem-statement-and-use-cases, IETF, March 2012.

[109] Network Functions Virtualisation Introductory White Paper. Whitepaper, October 2012.

[110] PTX Series Packet Transport Switch Datasheet. http://www.juniper.net/us/en/local/pdf/datasheets/1000364-en.pdf, December 2011.

[111] S. Das, G. Parulkar, and N. McKeown. Unifying Packet and CircuitSwitched Networks. In Below IP Networking workshop in conjunctionwith Globecom ’09, 2009.

[112] A. Ford, C. Raiciu, M. Handley, S. Barre, and J. Iyengar. Architec-tural Guidelines for Multipath TCP Development. RFC 6182, InternetEngineering Task Force, March 2011.

[113] Olivier Bonaventure, Mark Handley, and Costin Raiciu. An overviewof Multipath TCP. USENIX login;, October 2012.

[114] R. van der Pol, M. Bredely, A Barczyky, B. Overeinderz, N. vanAdrichemx, and F. Kuipersx. Experiences with MPTCP in an intercon-tinental OpenFlow network. In The 29th TERENA Networking Confer-ence, pages 1617–1624, June 2013.

[115] R. van der Pol, S. Boele, F. Dijkstra, A. Barczyk, G. van Malenstein,J.H. Chen, and J. Mambretti. Multipathing with MPTCP and Open-Flow. In High Performance Computing, Networking, Storage and Anal-ysis (SCC), 2012 SC Companion:, pages 1617–1624, November 2012.

[116] Felician Nemeth, Balazs Sonkoly, Levente Csikor, and Andras Gulyas.A Large-scale Multipath Playground for Experimenters and EarlyAdopters. In Proceedings of the ACM SIGCOMM 2013 Conference onSIGCOMM, SIGCOMM ’13, pages 481–482, New York, USA, 2013.ACM.

[117] M. Coudron, S. Secci, G. Pujolle, P. Raad, and P. Gallard. Cross-layercooperation to boost multipath TCP performance in cloud networks.In Cloud Networking (CloudNet), 2013 IEEE 2nd International Con-ference on, pages 58–66, November 2013.

[118] P. Gorja and R. Kurapati. Extending open vSwitch to L4-L7 serviceaware OpenFlow switch. In Advance Computing Conference (IACC),2014 IEEE International, pages 343–347, February 2014.

[119] S. Gringeri, N. Bitar, and T.J. Xia. Extending software defined net-work principles to include optical transport. Communications Maga-zine, IEEE, 51(3):32–40, March 2013.

[120] B. Nunes, M. Mendonca, X. Nguyen, K. Obraczka, and T. Turletti. ASurvey of Software-Defined Networking: Past, Present, and Future ofProgrammable Networks. Communications Surveys Tutorials, IEEE,PP(99):1–18, 2014.

[121] A. Sadasivarao, S. Syed, P. Pan, C. Liou, A. Lake, C. Guok, andI. Monga. Open Transport Switch: A Software Defined NetworkingArchitecture for Transport Networks. In Proceedings of the SecondACM SIGCOMM Workshop on Hot Topics in Software Defined Net-working, HotSDN ’13, pages 115–120, New York, USA, 2013. ACM.

[122] K. Idoudi and H. Elbiaze. Enhancing OpenFlow to enable multilayernetworking. In Transparent Optical Networks (ICTON), 2013 15thInternational Conference on, pages 1–5, June 2013.

[123] M. Channegowda, R. Nejabati, and D. Simeonidou. Software-definedoptical networks technology and infrastructure: Enabling software-defined optical network operations. Optical Communications and Net-working, IEEE/OSA Journal of, 5(10):A274–A282, October 2013.

network management challenges and trends in multi-layer and

Documents