network management & monitoring

1 v1.2

2 v1.2

Network Management & MonitoringModern Monitoring Architecture & Model Driven Programmability

bdNOG13Fakrul Alam | APNIC CT

3 v1.2

Table of Content• Basics of Network Monitoring

• Network Monitoring and Management– Monitoring – Why, What and How– Management – Why, What and How

• Network Monitoring and Management best practices

• Techniques and Tools

• Modern Monitoring Architecture

• Model Driven Programmability

4 v1.2

Module 1: Network Management & Monitoring

5 v1.2

Basics of Network Monitoring• Network have evolved from being a flat to a complex network with

lot more technologies:– Cloud– Wireless– Remote Users and VPN– Mobile Devices– IoT– VOIP

• Simple networks don’t meet the requirements of modern infrastructure anymore

6 v1.2

Basics of Network Monitoring• In spite of all the evolution that has occurred, one factor that has

been constant is the need for network monitoring

• For effective monitoring solution it’s critical to understand the– Major network components → Router, Switch, Firewall, Load Balancer, Server &

Services– Protocols → SNMP, ICMP, gRPC, Netflow– Monitoring Tools → LibreNMS, Nagios, Cacti etc

7 v1.2

Network Monitoring and Management• Monitoring

– Constantly checks/scans the status of a network– Collect statistics and performance metrics– Checking for error conditions notifies the network administrator for slow or failing

components

• Network Monitoring involves– What we should be looking for → Throughput, Latency, Packet loss, Bandwidth– How to find it → Ping, SNMP Trap, SNMP Poll, API– Where and how to store the values → Cloud, On-prem, RRD, Time Series Data– What thresholds indicate a problem situation → Performance metrics– How to notify → Email, SMS, Webhook

8 v1.2

Network Monitoring and Management• Management

– The processes, tools and applications used to administer, operate and maintain a network infrastructure

• Network Management involves– Administration → Tracking network resources– Operation → Network functions well– Maintenance → Upgrades and fixes to network resources– Provisioning → Network resource configuration

9 v1.2

Every Network is different

And

No single system will solve all your problems

Or

Meet all your requirements

10 v1.2

WHY do we Monitor• Track resource utilization and get historical data• Establish a baseline (what’s normal for the network)• Understand the performance matrices and do capacity planning• Helps network and systems administrators identify possible issues

before they affect business continuity• Find the root cause of problems when something goes wrong in

the network• Track configuration changes• Identify security threats

11 v1.2

WHAT do we Monitor• All the resources vs Critical resources

– Hardware → Network Devices & Servers– Services → DNS, DHCP, HTTP/HTTPS, SMTP– Application → On-prem and Cloud

• Underlay vs Overlay– IGP (OSPF, ISIS)– OMP – EVPN & VXLAN

Availability Reliability Capacity Performance

Uptime Jitter, Latency, RTT Traffic, Port Utilization CPU, Memory, Disk, Processes

12 v1.2

HOW to Monitor• Commonly used technologies:

– Ping– SNMP

• SNMP Trap• SNMP Poll

– Syslog– CDP/LLDP– Netflow– API

• There are tools which leverages these features to monitor network

13 v1.2

WHY do we Manage• Maximum efficiency and improve IT productivity

• Security and Threat detection

• Network upgrade and visibility

14 v1.2

WHAT do we Manage• Network resources (routers, switches, Firewall, Load Balancer)

• Systems (Servers, Applications)

• Power system devices

• Customer Premises Equipment (CPE)

• Storage

15 v1.2

HOW to Manage• Agent based – reside on mange network/system element

• Most common protocol is SNMP SET

• Few new protocols include NETCONF and RESTCONFSNMP NETCONF SOAP RESTCONF

Standard IETF IETF W3C IETFResources OIDs Paths URLsData Models Defined in MIBs YANG YANGManagement Operations

SNMP NETCONF XML HTTP Operations

Encoding BER XML XML XML, JSONTransport Stack UDP SSH

TCPSSLHTTPTCP

SSLHTTPTCP

16 v1.2

Network Monitoring and Management best practices

• Baseline network behaviour

• Escalation matrix

• Layered breakdowns

• Implement High Availability with failover options

• Capacity planning and growth

17 v1.2

Techniques and Tools• Agent-based and Agentless Monitoring

• Internal and External Monitoring

• Centralized vs Decentralized Network Management

18 v1.2

SaaS based Monitoring and Management• SaaS based monitoring and management tools

• Scalability, accessibility, upgradability, resilience and pay-as-you-go pricing options

• Work for both on-prem and cloud infrastructure• Few SaaS based Monitoring

tools• LogicMonitor• New Relic• Auvik• StatusCake

19 v1.2

Tools• Few Network Monitoring & Management toolsTool Function LinkIcinga Availability, Performance, Monitoring https://icinga.com/Nagios Availability, Performance, Monitoring https://www.nagios.org/LibreNMS Availability, Capacity, Discovery, Performance, Monitoring https://www.librenms.org/Zabbix Availability, Capacity, Discovery, Performance, Monitoring https://www.zabbix.com/Smokeping Availability, Latency, Monitoring https://oss.oetiker.ch/smokeping/Nfsen/Nfdump Traffic Analysis, Monitoring, Flow Collection http://nfsen.sourceforge.net/

https://github.com/phaag/nfdumpAS-Stats Traffic Analysis, Monitoring, Flow Collection https://github.com/manuelkasper/AS-StatsRancid Backup, Monitoring, Management https://shrubbery.net/rancid/Oxidized Backup, Monitoring, Management https://github.com/ytti/oxidizedRT/OSTicket Ticketing System https://osticket.com/NetDisco Discovery, Inventory, IPAM http://netdisco.org/Syslog-ng Log Management https://github.com/syslog-ng/syslog-ngGraylog Log Management https://www.graylog.org/products/open-sourceNetdot Documentation https://github.com/cvicente/Netdot/Nipap IPAM https://spritelink.github.io/NIPAP/

20 v1.2

Network Monitoring and Management – What Next?Legacy Way Modern Way

Network Monitoring SNMP GetSNMP TrapRRD

API (Webhook)Model Driven Telemetry (gRPC)Time Series Database (TSD)

Network Management SNMP Set NetConfRestConfOpenConfig

21 v1.2

Module 2: Modern Monitoring Architecture

22 v1.2

Time Series Database (TSDB)• Time series data is a set of data indexed by time

• A time series database (TSDB) is a database optimized for time-stamped or time series data

• This type of data describes the measurements of a measured subject at each time point of a time range

23 v1.2

Time Series Database (TSDB)• Time series data can be useful for:

– Tracking daily, hourly, or weekly weather data– Tracking changes in application performance– Medical devices to visualize vitals in real time– Tracking network logs

24 v1.2

Time Series Data Models• Modeling of time series data includes three important parts:

– Subject: The subject to be measured. Example: Taking server status monitoring. The measured subject is a server, and its attributes may include the cluster name, host name, and so on

– Time point: A subject may have one or more measurements, each corresponding to a specific metric. In the case of the server status monitoring, the measured metrics may include CPU usage and IOPS.

– Measurements: The measurement report is always attached with a timestamp attribute to indicate the time

timestamp cluster hostname cpu iops202-12-20T 17:50:00Z Cluster-A Host-a 10 10202-12-20T 17:50:10Z Cluster-A Host-b 20 30202-12-20T 17:50:20Z Cluster-A Host-a 5 9

Timestamp Subject Dimension Metrics and measurements

25 v1.2

Time Series Databases• Few popular Time-Series Databases:

– InfluxDB– Promethues– RRDtool– TimeScale– OpenTSDB– Graphite

26 v1.2

Monitoring Stacks• Popular open source solution stacks:

– Elastic Stack– TICK (Telegraf, InfluxDB, Chronograf and Kapacitor) – TIG (Telegraf, InfluxDB, and Grafana)

27 v1.2

A Modern Monitoring Architecture - TIG Stack• Open source tools built to make collection, storage, graphing, and

alerting on time series data easy• TIG Stack stand for Telegraf, InfluxDB, and Grafana

– InfluxDB is an open-source time series database written in Go– Telegraf is an agent for collecting, processing, aggregating, and writing metrics– Grafana is an open source data visualization and monitoring suite

A Modern Monitoring Architecture

Collects time-series data from different source

Store time-series data Visualizes and graphs the time series data stored in InfluxDB

CPU MEMORY

DISK PROCESS

LOAD NET

APPLICATIONS

Servers

28 v1.2

TIG Architecture

https://www.influxdata.com/products/influxdb/

Telegraf:• Telegraf is a data collection agent

that captures data from a growing list of sources and translates it into InfluxDB line protocol format for storage in InfluxDB

• Telegraf’s extensible architecture makes it easy to create plugins that both pull data (input plugins) and push data (output plugins) to and from different sources and endpoints

InfluxDB:• InfluxDB stores data for any use case

involving large amounts of timestamped data, including DevOps monitoring, log data, application metrics, IoT sensor data, and real-time analytics

Grafana:• Grafana is the user interface for the

TIG stack that provides customizable dashboards, data visualizations, and data exploration

29 v1.2

Telegraf• Telegraf is a plugin-driven server agent for

collecting and sending metrics and events from databases, systems, and IoT sensors

• It can collect metrics from a wide array of inputs and write them into a wide array of outputs

• With 200+ plugins already written by subject matter experts on the data in the community, it is easy to start collecting metrics from your end-points

• For all the plugins and integration– https://www.influxdata.com/products/integra

tionsInput Plugins Process

(transform, filter)Aggregate

(sum, count) Output Plugins

30 v1.2

InfluxDB• InfluxDB is an open-source time series database (TSDB)

developed by InfluxData

• Optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics

• Provides an SQL-like language, listening on port 8086

31 v1.2

Grafana• The open-source platform for monitoring and observability.• Grafana allows us to query, visualize, alert on and understand metric• Features:

– Visualize– Dynamic Dashboards– Explore Metrics– Explore Logs– Alerting– Mixed Data Sources

• Check Grafana in action– https://play.grafana.org/

• Import dashboard– https://grafana.com/grafana/dashboards

32 v1.2

Prometheus• Prometheus is an open-source monitoring tool and time-series

database

• Prometheus provides powerful query language, storage, and visualization features for its users

• InfluxDB vs Prometheus:

The most notable difference is between the scopes of these platforms. Both systems could be used for monitoring and time-series data storing. However, InfluxDB is more known as a time-series database, while Prometheus has a broader scope of monitoring purposes

Prometheus Demo:https://github.com/prometheus/demo-site

33 v1.2

Prometheus Architecture

source: https://prometheus.io/assets/architecture.png

34 v1.2

Monitoring StacksTime Series Database

Agent / Data Collector

Visualize Log Aggregators / Shippers

Logging Alerts Scripting/Query

Promethues PushgatewayExporters

Grafana, Promethues web UI

Promtail Loki AlertManager PromQLLogQ

Elasticsearch Beats → MetricbeatHeartbeat

Kibana Beats → FilebeatWinlogbeat

Logstash

InfluxDB Telegraf Chronograf Kapacitor FluxInfluxQL

OpenTSDB StasDCollectd

Graphite (Whisper & Carbon)

Graphite-web

35 v1.2

Module 3: Explore TICK Stack

36 v1.2

Explore TICK Stack – InfluxDB Cloud• To explore TICK stack we need to install following components

– Telegraf– InfluxDB– Capacitor– Kapacitor

• In this tutorial we will use InfluxDB Cloud. This will allow us to skip downloading and installing InfluxDB and jump into exploring InfluxDB 2.0 technology

• The Free Plan is designed for getting started with InfluxDB

https://www.influxdata.com/products/influxdb-cloud/

37 v1.2

InfluxDB CloudGot to https://www.influxdata.com/get-influxdb/ and signup

“Use it for free” Create account with valid email and verify

Choose where you like to store your data

Keep free plan

38 v1.2

Create a new bucket

Go to Data > Buckets and + Create Bucket

Name the bucket with your group name (example group10)

39 v1.2

Configure Telegraf Agent

Click on “Configure Telegraf Agent”

Choose “System” Name the config “system-monitor”

40 v1.2

Configure Telegraf Agent

Save the “API Token” & “Start Telegraf” command in notepad

41 v1.2

Install TelegrafLogin to you server and run the following commandswget https://dl.influxdata.com/telegraf/releases/telegraf_1.17.0-1_amd64.debsudo dpkg -i telegraf_1.17.0-1_amd64.deb

Run the “export INFLUX……” & “telegraf –config……” command; the one you have copied in notepad

Source: https://portal.influxdata.com/downloads/

42 v1.2

Verify the connection

“Listen for Data”

“Connection Found!”

43 v1.2

Explore Graph!1. Go to Explore and choose the

right bucket (group10 as an example)

2. Choose measurement (for example cpu)

3. Click Submit

44 v1.2

Module 4: Explore Grafana Cloud

45 v1.2

Grafana Cloud• Go to https://grafana.com/products/cloud/ and subscribe

• After login you will see services available in Grafana Cloud

46 v1.2

Grafana Cloud

47 v1.2

Grafana Cloud

Import dashboard ID 6126

48 v1.2

Grafana Cloud

49 v1.2

Module 5: Model Driven Programmability

50 v1.2

Table of Content• Device configurations - CLI & SNMP

• Model-driven programmability

• Data Model - YANG

• Protocols - NETCONF & RESTCONF

• Data Encoding - JSON & XML

• API Tools & Resources

51 v1.2

Device Configuration - Command-line Interface• There are many different ways to connect to and manage a

network

• The most commonly used method for the past 30 years has been by using the command-line interface (CLI)

• And one of the most glaring and biggest flaws with using CLI to manage a network is misconfiguration

• A majority of network outages are caused by human errors

52 v1.2

Device Configuration - Command-line Interface• CLI Pros and Cons

PROs CONsWell know and documented Difficult to scaleCommonly used method Large number of commandsCommand can be scripted Must known IOS command syntaxSyntax help available on each command Executing command can be slow

Can execute only one command at a timeCLI and command can change between software versions and platformsUsing CLI can pose a security threat if using Telnet (plaintext)Vendor-specific commandsComplex in parsing

53 v1.2

Device Configuration - SNMP• SNMP is widely used for fault handling and monitoring• In practice, SNMP is rarely used for configuration• Major problems of the traditional SNMP mode

– Insufficient Performance: Data configuration and reading are low, especially in the development of large-scale networks

– Difficult to deliver configurations: Only a few MIB objects support the write operation

– No support for the transaction mechanism: SNMP operations are stateless. Therefore, the operations cannot be interrupted in the case of a configuration timeout

– Poor programmability: Lack of composite data structures, few RPC interfaces and time-consuming commissioning

54 v1.2

A new API is needed?• In June, 2002 the IETF Internet Architecture Board (IAB) held a

Network Management Workshop to assess the state of network management and develop requirements for next generation network management protocol

• As a result, RFC 3535 was published that identified the need for a new NETwork CONfiguration protocol

• They outline some key requirements for this protocol– Easy to use for the operator– Provides Clear separation between Configuration state and operational state of the

device– Backup and restore capability– Human and Machine friendly– Provides Error checking

An API (Application Programming Interface) in its simplest form can be considered a Contract or Agreement between two end points, regarding what to send and what to expect in return.

It is used with Machines to request data and or to change data on the device

55 v1.2

The Rise of NETCONF• In 2006 NETCONF was born

– RFC 4741: NETCONF protocol v1.0– RFC 6241: NETCONF protocol v1.1

• The NETCONF standard define a communication channel between a Server (network Node) and client (NMS, SDN or Application)

• The end result has been a programmable device interface ideally suited for use in SDN and NFV

56 v1.2

Model-driven Programmability

source: https://blogs.cisco.com/getyourbuildon/model-driven-programmability

App1 App2 App3

Model-Driven APIsYANG Development Kit (YDK)

NETCONF RESTCONF gNMI

XML JSON GPB

SSH HTTP(S)

Data ModelsYANG (native, open)

Apps

Bindings

Protocol

Encoding

Transport

Models

Model-driven programmability inherits the power of models, making it easier to configure routers. It overcomes drawbacks posed by traditional router management

techniques

Model-Driven Configuration

Model-Driven Telemetry

57 v1.2

Model-driven Programmability - Benefits• Model based, structured, computer friendly

• Multiple model types (native, OpenConfig, IETF)

• Models decoupled from transport, protocol end encoding

• Choice of transport, protocol and encoding

• Model-driven APIs for abstraction and simplification

• Wide standard support while leveraging open source

58 v1.2

Data Models• Data models are used to describe

– whatever can be configured on a device, – everything that can be monitored on a device, – and all the administrative actions that can be executed on a device (example:

resetting counters or rebooting the device)

• The list of supported models includes – Native (Vendor)– OpenConfig and – IETF models

• YANG (Yet Another Next Generation) provides a modeling language optimized for network devices and with a growing number of tools and utilities

59 v1.2

Protocols• The separation of models from the choice of protocol provides a

high degree of flexibility

• We can control the network device using – NETCONF– RESTCONF or– gNMI (gRPC Network Management Interface)

• Your protocol choice will ultimately be influenced by your networking, programming and automation background, plus tooling available

60 v1.2

Encodings• The separation of encodings from the choice of model and

protocol provides additional flexibility

• Data can be encoded in – JSON– XML or – Google protocol buffers (GPB) format

• While some transports are currently tied to specific encodings (e.g. NETCONF and XML), the programmability infrastructure is designed to support different encodings of the same data model if the transport protocol supports it

61 v1.2

Model Driven Programmability - Summary

62 v1.2

Data ModelsModel Driven Programmability

63 v1.2

YANG• “Yet Another Next Generation” – A Data Modeling Language for NETCONF• YANG is a data modeling language used to

– model configuration– state data and – administrative actions manipulated by the NETCONF protocol

• Key YANG Capabilities– Human readable, easy to learn representation– Hierarchical configuration data models– Reusable types and groupings (structured types)– Extensibility through augmentation mechanisms– Supports the definition of operations (RPCs)– Formal constraints for configuration validation– Data modularity through modules and submodules– Versioning rules and development support

RFC 6020RFC 7950

64 v1.2

YANG vs Other Modeling Language• Other Modeling languages are already existed

– SMI (SNMP)– UML– XML Schema

• None of these languages were specifically targeted to the needs of configuration management

• The Network Modeling (NETMOD) working group is responsible for the YANG data modeling language

65 v1.2

YANG Structure• Yang files are called Modules• Each module is uniquely identified by a

namespace URI• YANG models data using a hierarchical,

tree-based structure with nodes• Main node types:

– Container : A grouping of other statements (which have no value)

– List : Multiple records containing at least one Leaf “Key” and an arbitrary hierarchy of other statements within the Module

– Leaf : A single Key/Value Pair– Leaf-List : Multiple Key/Value pairs of the same

type or commonality

• Each leaf is associated against a data type

66 v1.2

YANG: A Familiar ComparisonYANG Element Python ClassContainer DictionaryContainer name Dictionary keyLeaf name Dictionary keyLeaf Dictionary valueList ListString StringBool BoolInteger IntegerEmpty None

67 v1.2

YANG Model Types• Currently there are three main publishers for the YANG Modules

Vendor (native) IETF OpenConfig

Vendor specific

Example:“BGP” extensions on IOS-XE

Reference:https://github.com/YangModels/yang/tree/master/vendor/cisco

Standard definition from IETF, ITU etc)

Example:ietf-diffserv-policy.yangietf-diffserv-target.yang

Reference:https://github.com/YangModels/yang

Vendor-neutral data models

Example:openconfig-bgp-common-multiprotocol.yangopenconfig-mpls-rsvp.yang

Reference:https://www.openconfig.net/

OpenConfig and IETF models are mapped to native models

68 v1.2

YANG Tools - pyang• pyang is a YANG validator, transformation and code generator,

written in python

• It can be used to validate YANG modules for correctness, to transform YANG modules into other formats, and to generate code from the modules

• https://pypi.org/project/pyang/

$ pyang -f tree ietf-interfaces@2018-02-20.yangmodule: ietf-interfaces+--rw interfaces| +--rw interface* [name]| +--rw name string| +--rw description? string| +--rw type identityref| +--rw enabled? boolean| +--rw link-up-down-trap-enable? enumeration {if-mib}?| +--ro admin-status enumeration {if-mib}?| +--ro oper-status enumeration| +--ro last-change? yang:date-and-time| +--ro if-index int32 {if-mib}?| +--ro phys-address? yang:phys-address| +--ro higher-layer-if* interface-ref| +--ro lower-layer-if* interface-ref| +--ro speed? yang:gauge64

69 v1.2

YANG Tools - YANG Explorer• A GUI driven tool to test

NETCONF and RESTCONF interfaces defined by YANG models

• Features– Load YANG models from device– Browse YANG models– Execute NETCONF or RESTCONF

Operations– Generate self-contained Python scripts– Open Source https://github.com/CiscoDevNet/yang-explorer

70 v1.2

ProtocolsModel Driven Programmability

71 v1.2

Protocols• Set of rules that determine how data is transmitted between

different devices in the same network

• Common protocols for model-driven programmability– NETCONF– RESTCONF

72 v1.2

NETCONF• NETCONF is an IETF network management protocol designed

specifically for configuration management

• Features:– Makes a distinction between configuration and state data – Utilizes multiple configuration data stores (candidate, running, startup) – Configuration change transactions – Provides client-side configuration validation– Uses filtering mechanisms for selective data retrieval– Uses a client-server model and SSH as transport protocol

73 v1.2

NETCONF Communications• The NETCONF standard define a communication channel

between a Server (network Node) and client (NMS, SDN or Application)

74 v1.2

NETCONF Protocol Stack• The following are the key characteristics of NETCONF

– It uses SSH as Transport protocol– It is Connection oriented and Session Oriented– It uses XML for encoding and representing data– It defines multiple operations to interact with the network device– It uses RPC methods to invoke these operations on the remote network device

(4) Content Configuration / Operational Data

(3) Operations

(2) Messages

(1) Transport

Actions to Take

Remote Procedure Call (RPC)

TCP/IP

<data>

<get>, <get-config>, <edit-config>

<rpc>, <rpc-reply>

SSH (830)

XML

75 v1.2

NETCONF OperationsMain Operations Cisco Command Description<get> show ? Retrieve running configuration and device state

information<get-config> show run Retrieve all or part of specified configuration datastore<edit-config> con t Loads all or part of a configuration to the specified

configuration datastoreOther Operations Description<copy-config> Replace an entire configuration datastore with another <delete-config> Delete a configuration datastore<commit> Copy candidate datastore to running datastore (ex: XR)<lock> / <unlock> Lock or unlock the entire configuration datastore

system<close-session> Graceful termination of NETCONF session<kill-session> Forced termination of NETCONF session

76 v1.2

NETCONF Data Stores• The NETCONF standard defines an important concept Called

Data Stores to interact with the network devices, these Data Stores are:

– Running Data Store: Include the current active configuration on the device– Candidate Data Store: Holds that configuration changes before

committing/copying to the running configuration – Startup Data Stores: This is the configuration that the device use during boot up

• Not all data stores are supported by all devices

• Running config is the only required data store

77 v1.2

RESTCONF• It’s an implementation of a REST API

• Model-driven API

• Functional sub-set of NETCONF

• Exposes YANG models via a REST API (URL)

• Uses HTTP(S) as transport

• Uses XML or JSON for encoding

• Developed to use HTTP tools and programming libraries

78 v1.2

RESTCONF - Background on REST• Same HTTP Request Methods and Response Codes are used

Client Server

HTTP GET

HTML Response

API Client Web ServerOn a Network Device

HTTP GET

JSON/XML Response

79 v1.2

REST - HTTP Verbs

GET Retrieve / Read a resource show commandPOST Creates a new resource create logical interfacePUT Update/Replace a resource replace full interface config with what’s in

the body of requestPATCH Update/Modify a resource update (append) interface config with

what’s in the body of requestDELETE Removes a resource remove logical interface

• HTTP Verbs in the context of network devices

80 v1.2

YANG models over RESTCONF

81 v1.2

REST - HTTP Response Code• Common HTTP Response CodeSuccess (2xx) Description200 Request Succeeded201 The request has been fulfilled; new resource created204 The server fulfilled request but does not return a body

Client Error (4xx) Description400 Bad Request. Malformed syntax401 Unauthorized403 Forbidden. Server understood request, but refuses to full fill it.404 Not found

Server Error (5xx) Description500 Internal Server Error501 Not implemented

82 v1.2

REST API message formats - GETcurl -v https://ipwhois.app/json/8.8.8.8\?objects\=country,city,timezone

> GET /json/8.8.8.8?objects=country,city,timezone HTTP/1.1> Host: ipwhois.app> User-Agent: curl/7.58.0> Accept: */*>< HTTP/1.1 200 OK< Server: nginx/1.14.0< Date: Mon, 04 Jan 2021 12:43:10 GMT< Content-Type: application/json; charset=utf-8< Transfer-Encoding: chunked< Connection: keep-alive< X-Powered-By: PHP/7.4.6< Access-Control-Allow-Origin: *< Access-Control-Allow-Headers: *< X-Robots-Tag: noindex<* Connection #0 to host ipwhois.app left intact{"country":"UnitedStates","city":"Ashburn","timezone":"America\/New_York"}%

Request Headers

Response Headers

Response Body (Payload)

Status Code

Verb

83 v1.2

NETCONF vs RESTCONFNETCONF RESTCONF

Standard IETF IETFResources Paths URLsData Modeling Language YANG -Management Operation NETCONF HTTP OperationsEncoding XML XML, JSONTransport Stack SSH

TCPSSLHTTPTCP

84 v1.2

Data EncodingModel Driven Programmability

85 v1.2

Data Encoding• There needs to be structure behind what is communicated

between systemscore-router# show run interface GigabitEthernet 1 Building configuration...

Current configuration : 146 bytes ! interface GigabitEthernet1

vrf forwarding MANAGEMENT ip address 10.0.0.151 255.255.255.0 negotiation auto

end

• This is formatted text, not structured data• Standard data formats

• XML• JSON

86 v1.2

XML• XML stands for eXtensible Markup

Language

• Designed to store and transport data• In XML we have tags, elements, and

attributes• A tag is the text between <>, such as

<ip>

• An element is the starting tag, ending tag, and everything in between. Example from line 7-14

• An attribute is a key/value pair inside the starting tag of an element

<?xml version="1.0" encoding="UTF-8"?><GigabitEthernet>

<name>1</name><vrf>

<forwarding>MANAGEMENT</forwarding></vrf><ip>

<address><primary>

<address>10.0.0.151</address><mask>255.255.255.0</mask>

</primary></address>

</ip><negotiation xmlns="http://cisco.com/ns/yang/Cisco-

IOS-XE-ethernet" ><auto>true</auto>

</negotiation></GigabitEthernet>

123456789101112131415

161718

To start a comment, you use <!-- and then you end it with --> such as <!--This is a comment -->

87 v1.2

XML• Run the following command:curl -i -k -X "GET" "https://ios-xe-mgmt.cisco.com:9443/restconf/data/Cisco-IOS-XE-native:native/interface" -H "Accept: application/yang-data+xml" -u "developer:C1sco12345" -H "Content-Type: application/yang-data+xml"

<interface xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native" xmlns:ios="http://cisco.com/ns/yang/Cisco-IOS-XE-native">

<GigabitEthernet><name>1</name><description>MANAGEMENT INTERFACE - DON'T TOUCH ME</description><ip><address><primary>

<address>10.10.20.48</address><mask>255.255.255.0</mask>

</primary></address>

</ip>

Output:

88 v1.2

JSON• JSON is based on a subset of the JavaScript programming language, as the name

implies• JSON is a syntax for storing and exchanging data• JSON has no requirement for indentation or white space. It is ideal to use white

space and spaces, most likely either two or four to make it human readable– Strings MUST use double quotes– Object literal names MUST be lowercase (null, false, true etc)– Special characters need to be escaped– { says “begin object”– } says “end object”– [ says “begin array”– ] says “end array”– : separates key and value in key/value pair– , separates key/value pair in an object or separates values in an array, think of it as “expect another one”

89 v1.2

JSON• JSON supports the following data

types:– Object– String– Number– Boolean– Null– Array

• It’s useful to use JSON formatter and validator

– https://jsonformatter.curiousconcept.com/

{"Cisco-IOS-XE-native:GigabitEthernet":{

"name":"1","vrf":{

"forwarding":"MANAGEMENT"},"ip":{

"address":{"primary":{

"address":"10.0.0.151","mask":"255.255.255.0"

}}

},"Cisco-IOS-XE-ethernet:negotiation":{

"auto":true}

}}

12345678910111213141516171819

90 v1.2

JSON• Run the following command:curl -i -k -X "GET" "https://ios-xe-mgmt.cisco.com:9443/restconf/data/Cisco-IOS-XE-native:native/interface" -H "Accept: application/yang-data+json" -u "developer:C1sco12345" -H "Content-Type: application/yang-data+json"

{"Cisco-IOS-XE-native:interface": {"GigabitEthernet": [{"name": "1","description": "MANAGEMENT INTERFACE - DON'T TOUCH ME","ip": {"address": {"primary": {"address": "10.10.20.48","mask": "255.255.255.0"

}}

},"mop": {"enabled": false,"sysid": false

},"Cisco-IOS-XE-ethernet:negotiation": {"auto": true

}},

Output:

91 v1.2

API Tools and Resources

92 v1.2

How to consume API• API documentation : Normally API contains documentation and examples, so

the first step is to read the doc to see auth and methods are supported, what format is expected, etc

– https://ipwhois.io/documentation– https://developers.facebook.com/docs/graph-api/overview/– https://sandbox-sdwan-1.cisco.com/apidocs– https://developer.cisco.com/meraki/api-v1/#!get-device

• Know API endpoint which consists of:– Server URL : Webserver. For example: NGNIX or Flask– Resource : Path like /device/interface/id or /client/payments/

• Client– Browser– CURL– Postman– Programming language

93 v1.2

CURL• Curl is a command-line tool for transferring data specified with URL syntax

• With curl, we can download or upload data using one of the supported protocols including HTTP, HTTPS, SCP, SFTP and FTP

• Install curl$ sudo apt install curl –y

• CURL CLI arguments-X --request - Custom request method-d --data - Sends the specified data

-H --header - Sends headers-i --include - Display response headers

94 v1.2

CURL - REST API• CURL verbose modecurl -v https://ios-xe-mgmt.cisco.com:9443

• CURL GET commandcurl -i -k -X "GET" "https://ios-xe-mgmt.cisco.com:9443/restconf/data/Cisco-IOS-XE-native:native/interface" -H "Accept: application/yang-data+xml" -u "developer:C1sco12345" -H "Content-Type: application/yang-data+xml"

95 v1.2

Postman• The Collaboration Platform for API Development

• Features– Create, send and save REST, SOAP or GraphQL requests– Edit URL– Select method or create and save custom method– Edit request headers and– Save preset headers– Manage cookies associated with various domains– Send multipart/form-data, url encoded, binary, or raw data in request body

96 v1.2

Postman• Download Postman

– https://www.postman.com/downloads/

• Postman Chrome Extension– https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdd

domop?hl=en

97 v1.2

LabsModel Driven Programmability

98 v1.2

NETCONF• Model Driven Programmability NETCONF Labs

– Exercise 1: NETCONF get-config using SSH– Exercise 2 : NETCONF get-config using ncclient

99 v1.2

RESTCONF• Model Driven Programmability RESTCONF Labs

– Exercise 1: RESTCONF using Curl– Exercise 2 : RESTCONF using Postman

100 v1.2

References• YANG (https://tools.ietf.org/html/rfc6020)• NETCONF (https://tools.ietf.org/html/rfc6241)• RESTCONF (https://tools.ietf.org/html/rfc8040)• OpenConfig (https://github.com/openconfig/public)• Postman-for-AlwaysOn-Cisco-SD-WAN

(https://github.com/CiscoDevNet/Postman-for-AlwaysOn-Cisco-SD-WAN)

• YANG Catalog (https://yangcatalog.org/)• IOS XR Telemetry Collection Stack

(https://xrdocs.io/telemetry/tutorials/2018-06-04-ios-xr-telemetry-collection-stack-intro/)

101 v1.2

Thank You!