WORKSHOP:I Wouldn't Start From Here

Anna Wilson

Brian McArdle

Eoin Kenny

14/11/2019

Goals of this Workshop

• Raise awareness of the possibilities of automation• Introduce some of the tools we use at HEAnet• Describe some HEAnet use cases • Share the lessons we have learnt• Facilitate discussion • Possibly collaborate on common shared areas of interest

HEAnet RMAN Project 2016-2018

Install and Commission:4 x Juniper MX960

14 x Juniper MX480 41 x Juniper MX104171 x Juniper ACX2200

Create:168 IP services247 Eline services

Decommission and Uninstall:2 x Cisco CRS-1619 x Cisco 7600185 x ME3400/3750115 x CPE routers

Delete:150 IP services520 Eline services

“Replace the existing HEAnet IP and layer 2 network without any major incident”

Tender Requirements vs Actual

NMS Requirements Implemented solution SHIBA Service provisioning system Juniper CSDEquipment config management Juniper SpaceAutomated equipment provisioning Ansible, Docker, GITLAB CI, CheckMeAutomated external peering Ansible, Docker, GITLAB CIFault management & alerting Existing Icinga

Graphing and statistics Existing Cacti

DNS integration (IPAM) Existing DNS (6Connect)

Self service for clients Not implemented Client reports ExistingClient Reports

HEAnet & Network Automation Today

• Islands of automation• Goal is to automate as much as we can• Service provisioning (Eline, L3VPN, IP transit) – Juniper CSD• New device commissioning – Git, Docker, Ansible, Juniper Space• Base configuration – Git, Docker, Ansible Gitlab CI/CD• External peers – Git, Docker, Ansible and Gitlab CI/CD• Data Centre Switches – Ansible, GIT

• No Orchestration• No SDN• No NFV• No VNF

HEAnet Server Automation Today

• Islands of automation• Goal is to automate as much as we can• Heavy use of Git and some Gitlab CI/CD• Server lifecycle management via Foreman and Puppet• Starting to use Ansible/Git Lab CI for server iptables configuration

• No Orchestration • Small use of Cloudformation• No Terraform but interested

Config Management vs Orchestration

• Ansible, Chef, Puppet, and SaltStack are all “configuration management” tools are designed to install and manage software on existing servers.

• CloudFormation and Terraform are “orchestration tools” are designed to provision the servers themselves, leaving the job of configuring those servers to other tools.

• These two categories are not mutually exclusive, as most configuration management tools can do some degree of provisioning and most orchestration tools can do some degree of configuration management.

• Git with Ansible can roll out an application, the servers, the network infrastructure, but should you…

• Many ways to achieve the same result. It is all about the workflow you chose...

Software/Application/Network CI/CD

https://www.slideshare.net/dgarros/infrastructure-as-code-for-network

Example - Infrastructure as code

https://www.slideshare.net/dgarros/infrastructure-as-code-for-network

Current General Automation Approach

• We like the concept of Infrastructure As Code

• Same tools and workflow for server, network equipment and application provisioning

• Ansible, Docker and Gitlab CI/CD

• Will continue to develop this approach

• Very happy to share ideas, workflows etc

• Integration of network testing/validation into Gitlab CI/CD workflow.• GÉANT have done considerable work on this already – similar to Juniper NITA

product.

• Working with a GÉANT task group on examples of best practices guides for:• Orchestration• Automation• Virtualisation• Plan is to document and package up(code/tools) and share with our

community.

• Investigating SD-WAN and running HEAnet VNFs on telco CPE

Future Developments

New provisioning tool

• Space network mgmt.• CSD service provisioning

• Alerting

• Graphing

• Config logging

• DNS

• CRM

27/11/2019 1212

Previous processes

• Space network mgmt.• CSD service provisioning

27/11/2019

• L2 automatedL3 manual

• manual

• manual

• manual

• non-existent

1313

DNS (bind)

SHIBA

• Space HEAnet Integrated Brokerage Application (SHIBA)• Reads all network/service data from Space/CSD APIs• Stores it in relevant format• Export data to each tool via separate modules

27/11/2019 14

Alerting - Icinga

• API? Yes• But…• Generate config files

• E.g. like this:

• Push to Git repo• Use Puppet on the Icinga server

• pull repo and reload Icinga

27/11/2019 15

Graphing - Smokeping

• API? No• Generate config files

• E.g. like this:

• Push to Git repo• Use CI to check syntax• Use Puppet on the Smokeping server

• pull repo and reload Smokeping

27/11/2019 16

Graphing - Cacti

• API? No• Network scanning of IP range twice daily

• Use SHIBA to replace IP range with IP of new device -> instant graphs!• But IP range wasn’t always replaced…

27/11/2019 17

Graphing - Cacti

• Devices had to be manually refreshed• Click this button:

• On update to device, SHIBA will…• Query Cacti DB for ID of device• Log into non-SSO password-protected IP-restricted vhost• Go to URL using ID• Click button

• Automation!

27/11/2019 18

Config logging - RANCID

• API? No• Generate config files

• E.g. like this:

• Push to Git repo• Use Puppet on the RANCID server

• Pull repo• New list used on next run

27/11/2019 19

DNS - ProVision

• API? Yes• And it works!• Load issues

• Dedicated DB cluster• Throttled requests

27/11/2019 20

DNS - ProVision

• New application for DNS• Do we trust it to edit files directly on our main DNS server?

• Hell no!

• Created a second DNS server• Hidden master for particular zones

• Push to main DNS server• Public slave for particular zones

27/11/2019 21

DNS - ProVision

CRM - ClientDB

• API? Yes• And it works!

27/11/2019 23

New processes

27/11/2019 24

SHIBA

Lessons?

• Use APIs (assuming they work)• Practicality > idealism

• Whatever works for you, works!

• Trust• Didn’t trust automation not to break our DNS• Still don’t trust automation not to delete everything

− Removed services/devices are manually confirmed

27/11/2019 25

Managing our systems

Problem: Firewalling

Central firewall?

Every machine for itself?

Problem: Firewalling

Central control, distributed implementation

Problem: Firewalling

Ansible?Puppet?Chef?

Server?Continuous Integration?

Proprietary?Open Source?

Git?*Just kidding.

It’s always git.

Problem: Firewalling

Design choices

Git – version controlGitlab/Docker – change control, deploymentAnsible – implementationPuppet – initial deployment

Problem: Firewalling

Result

Problem: Drop junk traffic

Problem: Drop junk traffic

Design choices

Git – version controlGitlab – change control, deploymentAnsible – implementationDocker – runtime environment

Problem: Drop junk traffic

Result

Thank you