pnda the open platform for network data analytics...1hwzrun gdwd lv ehfrplqj d elj gdwd sureohp«...

$: PNDA The open platform for network data analytics...1hwzrun gdwd lv ehfrplqj d elj gdwd sureohp« irog lqfuhdvh lq wrwdo ,3 7udiilf ! lqfuhdvh lq ghylfhv dqg frqqhfwlrqv 7hohphwu\$
PNDAThe open platform for network data analytics

Network data is becoming abig data problem …

3-fold increase in total IP

Traffic

3-fold increase in total IP

Traffic

>60% increase in devices and connections

>60% increase in devices and connections

Telemetry data streamed in

near real-time

Telemetry data streamed in

near real-time

Computer science is driving bifurcation of the OSS stack…

OSSOSS

BSSBSS

Network and Services

Network and Services

CustomerCustomerOrderOrder

Order MgmtOrder Mgmt

Provisioning & ActivationProvisioning & Activation

ServiceService DataData

Monitoring and analysisMonitoring

and analysis

Billing and ReportingBilling and Reporting

Bills and ReportsBills and Reports

Orchestration is responsible for service provisioning and pushes state to the infrastructure

The “C” in FCAPS

Analytics is responsible for collecting data from the infrastructure, monitoring and analysisThe “F_APS” in FCAPS

AnalyticsAnalyticsOrchestrationOrchestration

• Tight coupling of data aggregation/store/analysis

• Multiple analytics pipelines implemented from open source components

• Common design patterns ~75% of effort wasted / duplicated

• Siloes limit the potential of big data analytics and lead to industry divergence

Today’s siloed analytics pipelines

TelemetryTelemetry

MetricsMetrics

Data sources

HDFS

Data store

Spark Streaming

Spark Streaming

MapRMapR

Data analysis

Hbase

StormStorm

KafkaKafka

StreamsetsStreamsets

Data aggregation

KafkaKafka

ImpalaImpala

QueryQuery

Outputs

Dashboard & Reporting

Dashboard & ReportingNiFiNiFi

LogsLogs

What is PNDA?PNDA brings together a number of open source technologies to provide a simple, scalable open big data analytics Platform for Network Data Analytics

Linux Foundation Collaborative Project based on the Apache ecosystem

• Scalable data platform

• Provides a common set of services for developing analytics applications

• 100% open source

• Open APIs

• Application packaging and management

• Multi-vendor, multi-data

• Infra and distribution agnostic

PNDA

PNDAPlugins

ODLODL

LogstashLogstash

OpenBPMOpenBPM

pmacctpmacct

XR TelemetryXR Telemetry

Real-timeReal-time

Data D

istributio

nD

ata Distribu

tion

FileStoreFile

Store

Platform Services: Installation, Mgmt, Security, Data Privacy


App Packaging and Mgmt


StreamStream

BatchBatch

Processing

SQL QuerySQL

Query

OLAP CubeOLAP Cube

Search/LuceneSearch/Lucene

NoSQLNoSQL TimeSeriesTime

Series

DataExploration

DataExploration

Metric Visualisation


Event Visualisation

Event Visualisation PNDA

Mnged AppPNDA

Mnged App

PNDA Mnged App

PNDA Mnged App

UnmngedApp

UnmngedApp

UnmngedApp

UnmngedApp

Query Visualisationand Exploration

PNDA Applications

PNDAProducer API

PNDAConsumer API

• PNDA decouples data aggregation from data analysis

• Horizontally scalable platform for analytics and data processing applications

• Support for near-real-time stream processing and in-depth batch analysis on massive datasets

• Consuming applications can be either platform apps developed for PNDA or client apps integrated with PNDA

• Client apps can use one of several structured query interfaces or consume streams directly.

• Leverages best practise in big data analytics

PNDA

PNDAPlugins

ODLODL

LogstashLogstash

OpenBPMOpenBPM

pmacctpmacct

XR TelemetryXR Telemetry

Real-timeReal-time

Data D

istributio

nD

ata Distribu

tion

FileStoreFile

Store





StreamStream

BatchBatch

Processing

SQL QuerySQL

Query

OLAP CubeOLAP Cube

Search/LuceneSearch/Lucene

NoSQLNoSQL TimeSeriesTime

Series

DataExploration

DataExploration



Event Visualisation

Event Visualisation PNDA

Mnged AppPNDA

Mnged App

PNDA Mnged App

PNDA Mnged App

UnmngedApp

UnmngedApp

UnmngedApp

UnmngedApp

Query Visualisationand Exploration

PNDA Applications

PNDAProducer API

PNDAConsumer API

There are a bewildering number of big data technologies out there, so how do you decide what to use?

Combining multiple technologies into an end-to-end solution can be extremely complex and time-consuming

PNDA solves these problems –streamlining the process and minimsing the TCO of developing data processing applications

Why PNDA?

PNDA Software Components

• The PNDA console provides a dashboard across all components in a cluster

• Inbuilt platform test agents verify the operation of all components

• Active platform testing verifies the end-to-end data pipeline

PNDA Console

• Platform for data aggregation, distribution, processing and storage

• Automated installation, creation, and configuration• Openstack, AWS and baremetal

• Ubunto and RHEL

• Typical install ~1hr

• Online and offline install; modular install

• Open producer and consumer APIs• Avro platform schema

• Plugins for Logstash, pmacct, OpenBMP, OpenDaylight, Cisco XR-telemetry, bulk ingest …

• Data distribution – Apache Kafka

• Data store:• Automated data partitioning and storage

(HDFS)

• OpenTSDB – time series analysis

• Hbase - NoSQL

• Support for batch and stream processing:• Apache Spark and Spark Streaming

• Jupyter notebook server for app prototyping and data exploration

• Impala-based SQL query support

• Grafana for time series visualisation

• PNDA application packaging

• PNDA management and dashboard

PNDA 3.5 Capabilities

• Ingested data should be encapsulated in PNDA Avro schema and published on a pre-defined Kafka topic or set of topics

Publishing Data to PNDA

PNDA PluginsData Type Data Aggregator Data Aggregator Reference PNDA Producer Plugin Reference

BGP (inc. BGP LS) OpenBMP http://www.openbmp.org/#!index.md#Using_Kafka_for_Collector_Integration

http://pnda.io/pnda-guide/producer/openbmp.html

BGP PMACCT (BGP listener) http://www.pmacct.net/ http://pnda.io/pnda-guide/producer/pmacct.html

Bulk Ingest PNDA Bulk Ingest Tool http://pnda.io/pnda-guide/bulkingest/

ISIS PMACCT (ISIS listener) http://www.pmacct.net/ http://pnda.io/pnda-guide/producer/pmacct.html

Cisco XR streaming telemetry Pipeline https://github.com/cisco/bigmuddy-network-telemetry-collector

CollectD (CollectD supports multiple plugins as listed here https://collectd.org/wiki/index.php/Table_of_Plugins)

Logstash https://www.elastic.co/guide/en/logstash/current/plugins-codecs-collectd.html

http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/

IoT sensor via HTTP Node-RED https://nodered.org

Logstash (Logstash supports multiple plugins as listed here https://www.elastic.co/guide/en/logstash/current/input-plugins.html)

Logstash http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/

NETCONF Notifications ODL http://www.opendaylight.org/ http://pnda.io/pnda-guide/producer/opendl.html

Netflow / IPFIX Logstash https://www.elastic.co/guide/en/logstash/current/plugins-codecs-netflow.html


Netflow / IPFIX / sFlow pmacct http://www.pmacct.net/ http://pnda.io/pnda-guide/producer/pmacct.html

Openstack Work in progress

sFlow Logstash https://github.com/ashangit/logstash-codec-sflow


SNMP Metrics and Traps ODL https://wiki.opendaylight.org/view/SNMP_Plugin:Getting_Started

http://pnda.io/pnda-guide/producer/opendl.html

SNMP Traps Logstash https://www.elastic.co/guide/en/logstash/current/plugins-inputs-snmptrap.html


Syslog Logstash https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html


Syslog (RFC3164 or RFC5424 - needed for newer IOS/IOS XR/ NX OS etc.)

Logstash https://gist.github.com/donaldh/89b7304981f96497c94fe4d98bb03d71


• Smaller, simpler subset of PNDA designed for development, demonstration and education

• Can run it on your laptop

• Consistent technologies, including:• PNDA data-ingest (Kakfa/AVRO)

• Data-exploration tools: Jupyter, OpenTSDB and Grafana

• Apache Spark and Hbase

• Doesn’t include HDFS and other Hadoop infrastructure for distributed processing.

Red PNDA

PNDA Flavors span the Data Science Lifecycle

pico

standard Red

PNDA Applied to NFVIn

fras

truc

tur

eIn

fras

truc

tur

e

OSS Analytics

OSS Analytics

Data AggregatorsData Aggregators

Open Data Platform (PNDA)Open Data Platform (PNDA)

OSS Analytics ApplicationsOSS Analytics Applications

OSSFunctions

OSSFunctions

BSSFunctions

BSSFunctions

Ops and BusIntelligenceOps and BusIntelligence

AlertsAlerts

Metrics

Metrics

Telemetry

Telemetry

LogsLogs

Data Sources

InventoryInventory

Orc

hest

ratio

nO

rche

stra

tion

NFVONFVO

VNFMVNFM

VIMVIM

NFVINFVI

VNFVNF

Data CenterCoreUser

StateState D

ata

Dat

a

Access Aggregation

Loosely coupledsystems

ContextContext

NetworkControlNetworkControl

Orchestration service models

provide necessary context for analytics

Output from analytics can close the control loop to realize reactive networking

Enabling LSO to realise the vision of reactive networking MEF and PNDA Collaboration

• Analytics Based Service assurance

• ML-based Security Analytics with Apache SPOT on PNDA

• Path Anomaly detection in PNDA using in-band OAM

• Openstack Analytics with PNDA and Calipso

• Smart Transport – Connected Car Cloud Analytics with Machine Learning using PNDA

• BGP analytics with SNAS.io and PNDA.io

• ETSI NFV and Big Data Analytics with PNDA

• PNDA and Paris IOT Smart Cities Pilot

• Cable Plant Anomaly Detection with PNDA

Increasing List of PNDA Use Cases

Convergence of network data analytics

OperationalIntelligence

PlanningIntelligence

SecurityIntelligence

• Bare metal install

• HDP / Apache Ambari

• Apache Flink

• OPNFV

• ONAP (Release 2)

What’s coming?

pnda the open platform for network data analytics...1hwzrun gdwd lv ehfrplqj d elj gdwd sureohp«...

Documents