Self-service virtual environments at DeutscheTelekom based on OpenStack

Alexander StellwagDeutsche Telekom AGProducts & Innovation

Agenda

● Introduction● Short History Of Wolke-7● Planning and Setup● Virtual Private Environments● Version 2: havana and neutron● Lessons Learned● Q&A

Introduction

● Alex Stellwag● 43 Years● Started at Deutsche Telekom

in 2001● Cloud Architect since 2011● OpenStack since „Diablo“

Short History of Wolke-7● Started back in late 2011 after major reorganization to gain knowledge of

OpenStack● Dec 2012: initiated two basic use cases

– Plain IaaS, configurable via EC2-API

– Fully managed environments (PaaS)

● Both available from one cloud environment● Internal customers only, no public services● Core principles

– Stay open source

– Build over partnering

Intranet Infrastructure

● Supermicro servers (2UTwin² and 2U 12 Disk)● Ubuntu 12.04 LTS (Kernel 3.2 and 3.11)● OpenStack packages from UCA● Ceph packages from Inktank● I/S automation via FAI1 and puppet● Monitoring via nagios / check_mk

1 http://fai-project.org

Some Figures

● 12 compute nodes with 2x Xeon X5650 (6 cores, 2.67 GHz, Intel HT enabled), 96 GB Ram, 2x 250 GB SSD each

● 3 Ceph nodes with 2x Xeon E5-2630, 64 GB Ram, 2x 240 GB SSD + 10x 2TB S-ATA (OSDs) each. ==> ~60 TB net

● 2x Cisco Nexus 5548 for tenant VLANs and floating IPs● 2x Cisco 48 port 1 GE for administration

Planning and Setup

● Setting up OpenStack is quite painless thanks to the official docs● We needed a complete DC

– Location– Connectivity

– Basic I/S services (DNS, Mail, Proxy)– Automation services (FAI, puppet)

● Complex Setup due to internal security requirements● Adapting the puppet modules for our needs took a while

Virtual Private Environments I

● Simple applications (apache, mysql, tomcat)– Provided by user-data

– Published on project website

– User has to copy & paste them into dashboard

– Designed to fulfill security requirements

– Full self-service via OpenStack dashboard

– Users have full control and responsibility

#cloud-config

# this cloud config code installs apache2-worker webserver# and changes several configuration directives to already meet# some GIS requirements# after customization of the webserver the requirements have to be# checked agein

# supported OS: Ubuntu 12.04

# addtional packages to be installedpackages: - apache2 - libapache-mod-security

# commands to modify the default installationruncmd: - a2dismod cgid # req 7 - sed -i 's/\(.*Alias \/\(cgi-bin\|doc\)\)/#\1/; //{s/^/#/}; //,/Options /{s/\(.*\n\t\t\tdeny from all\n\t\t \ <\/LimitExcept>/}' /etc/apache2/sites-available/default # req 9,11,12,13,16,18 - sed -i 's/^/#/' /etc/apache2/mods-available/alias.conf # req 16 - sed -i 's/^\s*ServerTokens .*/ServerTokens Prod/' /etc/apache2/conf.d/security # req 19 - service apache2 restart # restart webserver to apply changes

Virtual Private Environments II

Virtual Private Environments III

● Complex environments (currently CI/CD for Java only)– Jenkins, git, tomcat

– Set up via bash and python scripts

– Not yet integrated in dashboard, ordered via e-mail

– Fully managed by Test & Development team

Version 2: havana & neutron

● Started in 10/2013● New location, so again we needed to setup a complete DC● Ready for production in May 2014● Currently looking for dedicated operations team● Heat installed and enabled but Virtual Environments not ported

over yet.● Users may use their own stack templates

Infrastructure Changes

● No dedicated network node, all agents distributed amongst compute nodes

● Multiple L3 and DHCP agents per L3 network● Custom python script to check availability of agents and

reschedule if necessary● Works, but we have no long-time experience● Currently looking into Icehouse, but migration has issues

(mainly OS-wise)

New Hardware

● 11 compute nodes with 2x Xeon E5-2660 v2 (10 cores, 2.2 GHz, Intel HT enabled), 128 GB Ram, 2x 400 GB SSD each

● 8 Ceph nodes with 2x Xeon E5-2620 v2 (6 cores, 2.1 GHz, Intel HT enabled), 32 GB Ram, 2x 100 GB SSD + 10x 4TB S-ATA (OSDs) each. ==> ~312 TB net

● 2x Alcatel Lucent OS 6900-T40 for VxLAN tunnels and floating IPs

● 2x Alcatel Lucent OS 6850 for administration

Lessons Learned

● I/S deployment is hard – even if you start on a green field● Deploying OpenStack without the ability to fix the code is

difficult (and sometimes frustrating)● HA for neutron is a complex beast● Deploying multi-tier applications without heat is no fun

Lessons Learned II

● Manage internal stakeholder for proper funding● Many customers don't know the difference between

virtualization and cloud computing● Change management program needed because cloud impacts

95% of operations roles● For large and inhomogeneous teams SCRUM is not the right

project management method

Q & A

Questions ?