workshop: i wouldn't start from here · config management vs orchestration • ansible, chef,...
Post on 16-Apr-2020
7 Views
Preview:
TRANSCRIPT
WORKSHOP:I Wouldn't Start From Here
Anna Wilson
Brian McArdle
Eoin Kenny
14/11/2019
Goals of this Workshop
• Raise awareness of the possibilities of automation• Introduce some of the tools we use at HEAnet• Describe some HEAnet use cases • Share the lessons we have learnt• Facilitate discussion • Possibly collaborate on common shared areas of interest
HEAnet RMAN Project 2016-2018
Install and Commission:4 x Juniper MX960
14 x Juniper MX480 41 x Juniper MX104171 x Juniper ACX2200
Create:168 IP services247 Eline services
Decommission and Uninstall:2 x Cisco CRS-1619 x Cisco 7600185 x ME3400/3750115 x CPE routers
Delete:150 IP services520 Eline services
“Replace the existing HEAnet IP and layer 2 network without any major incident”
Tender Requirements vs Actual
NMS Requirements Implemented solution SHIBA Service provisioning system Juniper CSDEquipment config management Juniper SpaceAutomated equipment provisioning Ansible, Docker, GITLAB CI, CheckMeAutomated external peering Ansible, Docker, GITLAB CIFault management & alerting Existing Icinga
Graphing and statistics Existing Cacti
DNS integration (IPAM) Existing DNS (6Connect)
Self service for clients Not implemented Client reports ExistingClient Reports
HEAnet & Network Automation Today
• Islands of automation• Goal is to automate as much as we can• Service provisioning (Eline, L3VPN, IP transit) – Juniper CSD• New device commissioning – Git, Docker, Ansible, Juniper Space• Base configuration – Git, Docker, Ansible Gitlab CI/CD• External peers – Git, Docker, Ansible and Gitlab CI/CD• Data Centre Switches – Ansible, GIT
• No Orchestration• No SDN• No NFV• No VNF
HEAnet Server Automation Today
• Islands of automation• Goal is to automate as much as we can• Heavy use of Git and some Gitlab CI/CD• Server lifecycle management via Foreman and Puppet• Starting to use Ansible/Git Lab CI for server iptables configuration
• No Orchestration • Small use of Cloudformation• No Terraform but interested
Config Management vs Orchestration
• Ansible, Chef, Puppet, and SaltStack are all “configuration management” tools are designed to install and manage software on existing servers.
• CloudFormation and Terraform are “orchestration tools” are designed to provision the servers themselves, leaving the job of configuring those servers to other tools.
• These two categories are not mutually exclusive, as most configuration management tools can do some degree of provisioning and most orchestration tools can do some degree of configuration management.
• Git with Ansible can roll out an application, the servers, the network infrastructure, but should you…
• Many ways to achieve the same result. It is all about the workflow you chose...
Software/Application/Network CI/CD
https://www.slideshare.net/dgarros/infrastructure-as-code-for-network
Example - Infrastructure as code
https://www.slideshare.net/dgarros/infrastructure-as-code-for-network
Current General Automation Approach
• We like the concept of Infrastructure As Code
• Same tools and workflow for server, network equipment and application provisioning
• Ansible, Docker and Gitlab CI/CD
• Will continue to develop this approach
• Very happy to share ideas, workflows etc
• Integration of network testing/validation into Gitlab CI/CD workflow.• GÉANT have done considerable work on this already – similar to Juniper NITA
product.
• Working with a GÉANT task group on examples of best practices guides for:• Orchestration• Automation• Virtualisation• Plan is to document and package up(code/tools) and share with our
community.
• Investigating SD-WAN and running HEAnet VNFs on telco CPE
Future Developments
New provisioning tool
• Space network mgmt.• CSD service provisioning
• Alerting
• Graphing
• Config logging
• DNS
• CRM
27/11/2019 1212
Previous processes
• Space network mgmt.• CSD service provisioning
27/11/2019
• L2 automatedL3 manual
• manual
• manual
• manual
• non-existent
1313
DNS (bind)
SHIBA
• Space HEAnet Integrated Brokerage Application (SHIBA)• Reads all network/service data from Space/CSD APIs• Stores it in relevant format• Export data to each tool via separate modules
27/11/2019 14
Alerting - Icinga
• API? Yes• But…• Generate config files
• E.g. like this:
• Push to Git repo• Use Puppet on the Icinga server
• pull repo and reload Icinga
27/11/2019 15
Graphing - Smokeping
• API? No• Generate config files
• E.g. like this:
• Push to Git repo• Use CI to check syntax• Use Puppet on the Smokeping server
• pull repo and reload Smokeping
27/11/2019 16
Graphing - Cacti
• API? No• Network scanning of IP range twice daily
• Use SHIBA to replace IP range with IP of new device -> instant graphs!• But IP range wasn’t always replaced…
27/11/2019 17
Graphing - Cacti
• Devices had to be manually refreshed• Click this button:
• On update to device, SHIBA will…• Query Cacti DB for ID of device• Log into non-SSO password-protected IP-restricted vhost• Go to URL using ID• Click button
• Automation!
27/11/2019 18
Config logging - RANCID
• API? No• Generate config files
• E.g. like this:
• Push to Git repo• Use Puppet on the RANCID server
• Pull repo• New list used on next run
27/11/2019 19
DNS - ProVision
• API? Yes• And it works!• Load issues
• Dedicated DB cluster• Throttled requests
27/11/2019 20
DNS - ProVision
• New application for DNS• Do we trust it to edit files directly on our main DNS server?
• Hell no!
• Created a second DNS server• Hidden master for particular zones
• Push to main DNS server• Public slave for particular zones
27/11/2019 21
DNS - ProVision
CRM - ClientDB
• API? Yes• And it works!
27/11/2019 23
New processes
27/11/2019 24
SHIBA
Lessons?
• Use APIs (assuming they work)• Practicality > idealism
• Whatever works for you, works!
• Trust• Didn’t trust automation not to break our DNS• Still don’t trust automation not to delete everything
− Removed services/devices are manually confirmed
27/11/2019 25
Managing our systems
Problem: Firewalling
Central firewall?
Every machine for itself?
Problem: Firewalling
Central control, distributed implementation
Problem: Firewalling
Ansible?Puppet?Chef?
Server?Continuous Integration?
Proprietary?Open Source?
Git?*Just kidding.
It’s always git.
Problem: Firewalling
Design choices
Git – version controlGitlab/Docker – change control, deploymentAnsible – implementationPuppet – initial deployment
Problem: Firewalling
Result
Problem: Drop junk traffic
Problem: Drop junk traffic
Design choices
Git – version controlGitlab – change control, deploymentAnsible – implementationDocker – runtime environment
Problem: Drop junk traffic
Result
Thank you
top related