symantec sdn deployment
TRANSCRIPT
Class of Service
• Dev– For developers to get familiar with OpenStack cloud– Each developer has a project
• Production– For teams to onboard their members– Each team has a project– Manage user roles– Manage production workloads
Copyright © 2015 Symantec Corporation3
Self-Service User Onboarding
• Zero tickets for user onboarding– Provide sign up capabilities on Horizon
• Provide easy networking on Dev CoS– Hide all complexities– Automatically create network– Allocate routable subnets by using Contrail VNC APIs– Create security group with proper rules– Create unique domain names for instances by using Designate for routable
IPs
Copyright © 2015 Symantec Corporation5
Load Balancer as a Service
•Out of the box
– Icehouse, v1 APIs
– Launch HA Proxy service instances on a single AZ
– SSL Support: Wildcard cert
• Symantec fixes
–Multiple AZ, SSL Passthrough, Stats and Metrics
• Performance:
–~6.5 Gbps throughput with 10K parallel connections, VIP with 2 members
–20K HTTPS requests/sec for 10K parallel connections with 1 million requests, 1K response size
• Tuning - haproxy.cfg: maxconn 50K, nbproc 4, ulimit-n 200K, Cipher
• Pain points
–No control over ha proxy cfg
–No control over resource allocations (cpu, etc)Copyright © 2015 Symantec Corporation
7
Baremetal on Overlay
•Applications that run on baremetal but needs to be on the overlay– Example: swift proxy and data nodes– Launch them inside network namespaces– Plug them to the vRouter– East-West Traffic
• Manual Setup via scripts– Nova is not aware but Contrail is.– Multiple nics sitting on multiple networks– Static IPs
Copyright © 2015 Symantec Corporation9
Control Plane Availability
• Goal - 99.95% Availability
• 5 SDN controller VMs distributed over 3 racks
• 5 Cassandra database baremetal nodes distributed over 3 racks– RF of 3 for analytics– RF of 5 for config– Compaction throughput 256 Mbps
• Deployment Automation: Puppet
• Issues seen: DB Timeouts, Version mismatch, admin token
Copyright © 2015 Symantec Corporation11
Failed Customer Interactions
• Measure the control plane availability
• Use Symantec’s Logging-Monitoring-Metering as a Service to parse Neutron logs
• Compare response codes: 5XX counted as failures
• Dashboards!
Copyright © 2015 Symantec Corporation13
Data Plane Availability
• Work in progress..
–FIP Availability
–vDNS
–Link Local
–Private Network
Copyright © 2015 Symantec Corporation15
Upgrade 1.20 to 2.0.1
• Goal - Zero Downtime
• Controller upgrades– No in-place upgrades– Build a parallel control plane with new release– Add them to the VIP pool and gradually decommission old controllers
• Database upgrades– Add new DB nodes one by one to the existing cluster– Repair the DB– Decommission old DB node one by one
• Compute upgrades– Automate unloading and loading of kernel module in all computes
Copyright © 2015 Symantec Corporation17
Health Monitoring
• Volta –Logging •Logstash•Elasticsearch
–Metrics•InfluxDB•Statsd•Collectd)
–RESTful APIs make it easy:•Response Codes, Bytes Transfered, Time, Verb, etc.
• OpsView / Zabbix
Copyright © 2015 Symantec Corporation19
Troubleshooting
• Most incidents are trivial– Known issues– Trivial fixes/workarounds
• Some incidents are complex– RCA is very involved– Might have to wait for next code release for a fix– Quick and dirty solution – use auto healing scripts for workarounds
•Periodically check system health (Synthetic Transactions)•Remediate known bugs•Fix problems as they are detected, Save pagers, run 24x7!(MX Encapsulation, Dead processes, etc.)
Copyright © 2015 Symantec Corporation21
Thank you!
Copyright © 2014 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners.
This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.