data center automation with suse manager · • it- staff ~2,100 internal and external admins and...

Data Center Automation with SUSE® Manager Federal Deployment Agency “Bundesagentur für Arbeit”Data Center Automation Project

Ralf Bueker DSE Bundesagentur für Arbeitrbueker@suse.com

2

What is a DSE?

• DSE means Designated Support Engineer

• Designated to one customer

• Onside 4 days a week

• Working embedded in the customers IT team

• Takes care of and escalates support issues

• Interface for the customer to SUSE

• Assist the customer in ongoing project

3

The Customer: Federal Employment Agency • The Federal Employment Agency (Bundesagentur für

Arbeit: BA) is the largest provider of labor market services in Germany, with more than 800 branch offices nationwide.

• BA is a governmental organization based in Nuremberg, Germany

• Employs ~100,000 people

• IT- staff ~2,100 internal and external admins and engineers

• BA drives ~100 different tailored applications (VAM, ALG2, ZELOS. ALEGRO … ) for their customers

4

The Task

• The BA software systems needs to be changed and upgraded according to social laws

• Obtaining the laws requires a lot of development and testing

• A test server should match perfectly in existing infrastructure. → No solution from scratch

• The allocation of test HW had been identified as a bottleneck (~6 weeks waiting period for appropriate HW)

5

DCA Challenges

• No common API / Authorization / CMDB

• New data added in every step of the process

• No consistent server classification (templates)

• Complex software stack needs to be applied

• Solution should match for Linux and Windows

• Servers should seamlessly match in the current admin/ update/ monitoring infrastructure

• For Linux: ARS (Remedy) tool (end user GUI) → UC4 (task management) → Soap Service → SUSE® Manager / Python script → VMware ESX

6

Workflow

7

SUSE® Manager and the Python script

• SUSE Manager Configuration

• XML and Schema

• AutoYaST customization

• Cobbler to create profiles and boot ISO

• ESX Server communication

• Monitoring installation

8

SUSE® Manager Configuration

• Create Distribution

• Create profile(s) • Create configuration channel

• Create software channel

• Create activation key(s)

• /etc/cobbler/settings and set: redhat_management_permissive: 1 // allows login in cobbler api

• Install DCA Python scripts

SUSE® Manager Profile / AutoYaST

10

SUSE® Manager Configuration

• Create Distribution

• Create profile(s)

• Create configuration channel• Create software channel

• Create activation key(s)

• /etc/cobbler/settings and set: redhat_management_permissive: 1 // allows login in cobbler api

• Install DCA Python scripts

11

Configuration Channel

12

Configuration Files

13

SUSE® Manager Configuration

• Create Distribution

• Create profile(s)

• Create configuration channel

• Create software channel• Create activation key(s)

• /etc/cobbler/settings and set: redhat_management_permissive: 1 // allows login in cobbler api

• Install DCA Python scripts

14

Software Channel

15

SUSE® Manager Configuration

• Create Distribution

• Create profile(s)

• Create configuration channel

• Create software channel

• Create activation key(s)• /etc/cobbler/settings and set: redhat_management_permissive: 1 // allows login in cobbler api

• Install DCA Python scripts

16

Activation Key

17

SUSE® Manager Configuration

• Create Distribution

• Create profile(s)

• Create configuration channel

• Create software channel

• Create activation key(s)

• /etc/cobbler/settings and set: redhat_management_permissive: 1 // allows login in cobbler api

• Install DCA Python scripts

18

Python Script: Class CreateServer

• Main program

• Installation flow control

• Reports back to soap service (back channel)

• Other helper functions

• Error Handler

• Logger Function

19

XML and Schema

• Parameters and settings can easily be checked and restricted by an appropriate XML schema.

• All parameters used during the installation from different programs can be collected and mapped in the schema

• The schema rules deliver a readable and evaluable description of data and objects for all different integrated applications

• Python delivers with lxml a very good library to handle xml and schema

20

Schema

21

Python Script Class XmlInputHandler

• XML_input• XmlParseInput• XmlValidateInput• XmlGetEntry

‒ apply ".//" to search string to enforce search from root

• XmlGetSubTree• FormatXML

‒ Used to create index for “multivalued Attributes” needed in more than one output xml (nic, disc)

22

Input file

23

Python Script Class PrepareIsoInstallation• MountEsx

‒ Mount ESX Drive for iso Images

• XmlGetKopts‒ Collects information for Installation Kernel Options

• GetDistro‒ Gets the SUSE® Manager distro from server class provided

• CopyToEsx‒ Obvious …

24

Python Script Class EsxHandler 1 / 2

VMware vCenter provides an API for its services (https://www.vmware.com/support/developer/vc-sdk/index.html).

• For DCA this API has been published and partly customized as a webservice. Sending an xml to the webservice triggers requested function. ‒ Install server‒ Clone install server‒ Delete server‒ PowerOn‒ esx_data (publish returnvalues from ESX)

25

Python Class EsxHandler 2 / 2

• SendToEsx: used to send xml to ESX Server• InputStringDelete, InputStringVmCreate,

InputStringCloneCreate, InputStringPowerOn, InputStringQueryTask: creates xml input for ESX Tasks

• FormatXml: parses answers from ESX (not necessary xml … )

• check_esx_output: processes ESX output• ExtractTaskId: Gets ESX task ID for status query • getMac: gets the randomly created MAC address(es)

for the new virtual machine from ESX output

26

ESX Input

27

Python Class SumaHandler 1/2

• OpenSumaConnection (key = client.auth.login(username, password, timeout)

• GetServerSumaId ((client.system.getId(key, servername)))

• DeleteServerFromSuma (client.system.deleteSystem(systemid))

• RebootServer (client.system.scheduleReboot(key, sumaid, now)

• RunRemoteBashScript client.system.scheduleScriptRun(key, sumaid, "root", "root", 5, script, now)

28

Python Class SumaHandler 2/2

• OsaRefresh subprocess.call ([curl, -Sks, "https://" + self.__suma_server + "/rhn/systems/details/Overview.do?sid=" + str(sumaid) + "&ping=1"])

• list_system_events (client.system.listSystemEvents(key, msumaid))

• get_server_dmi (client.system.getDmi(key, sumaid))• CloseSumaConnection (client.auth.logout(key))

29

Python Class AutoYaSTHandler

• AutoYaSTGetAutoYaSTfile: gets appropriate autoyast file template from /var/lib/rhn/kickstarts/upload/

• AutoYaSTGetParameterList: get the parameter list from the AutoYaST template

• AutoYaSTSetMetaData: creates a dictionary from AutoYaST parameter and values from XML input file

• AutoYaSTCreateNewAyfile : creates a specific AutoYaST file for the new server

30

Class CobblerHandler 1/2

Cobbler creates the iso image for the installation.To allow automatic boot boot.template has to be rewritten. This is implemented in a class with following functions: • CobblerBuildProfile: creates profile

‒ profile_id = server.new_profile(token)‒ server.modify_profile(profile_id, 'name', bcpname, token)‒ server.modify_profile(profile_id, 'distro', bcpdistro, token)‒ server.modify_profile(profile_id, 'kopts', bcpkopts, token)‒ server.modify_profile(profile_id, 'kickstart', bcpkickstart, token)‒ server.save_profile(profile_id, token)

31

Class CobblerHandler 2 / 2

• cobbler_build_iso_template: customizes the /etc/cobbler/iso/boot.template, so that the new iso is booted automaticallyBuild_iso_template

• cobbler_build_iso: build iso image and copies to ESX server (currently nfs share)

‒ Could be done by xmlrpc (see: pydoc cobbler.api on SuMa Server), currently uses cmdline cobbler buildiso commandline on Suma Server

• cobbler_delete_profile_if_exist: obivous …

32

Boot.template

33

Class BootstrapMonitor 1 / 3

• FirstPing: waits till the new server is pingable for the first time

• SumaReg: queries SUSE Manager if the new Server is already registered SumaHandler.GetServerSumaId

• WaitForBootstrapEnd: SumaHandler.RunRemoteBashScript

• ConfigDone: uses the same as above to see if all configuration from the channel is completedSumaHandler.RunRemoteBashScript

34

Class BootstrapMonitor 2 / 3

• CheckConfigLog: analyzes configuration on the new server (return value remote script “cat /var/adm/autoinstall/logs/bootstrap.sh.log” had any errors

• LastBoot: SumaHandler.RebootServer

• FinalUp: SumaHandler.get_server_dmi

• RemoteCleanUp: cleans all the remains from installation from the serverSumaHandler.RunRemoteBashScript

35

Your Server is ready!

• Sending a success message to the SOAP Webservice

• SOAP webservice forwards message to UC4

• Next Step in workflow can be triggered (I.E. DB installation)

• Clean up the SUSE® Manager and DCA environment (Deleting the profiles of the new Server, iso files … )

36

Operating Experience

• DCA has been introduced to the internal customer

• The SUSE® Manager implementation won against the VMware cloning approach

• Several hundred DCA installations have already performed successfully

• Stress tests have been performed (we delivered 20 Server in 30 minutes instead of 6 weeks …)

• The customer wants to develop DCA further

• A cloud software like SUSE® Cloud should replace the homegrown SUSE Manager Python implementation

37

Lessons Learned 1 / 2

• Local caches and infrastructure play an important role when doing live provisioning in a productive environment

• Error handling is vital part of process management and should be considered carefully

• A schema is a very useful to control and parameters from participating applications

• Automatic logging of every single step in the automatic process is vital!

38

Lessons Learned 2 / 2

We showed the customer:

• complex installation and configuration can be described flexible in SUSE® Manager

• server classification can be mapped in SUSE Manager

• An existing administration concept can be represented in SUSE Manager

• Thanks to the API SUSE Manager can flexibly used in a complex environment for several tasks

Corporate HeadquartersMaxfeldstrasse 590409 NurembergGermany

+49 911 740 53 0 (Worldwide)www.suse.com

Join us on:www.opensuse.org

39

Unpublished Work of SUSE LLC. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE LLC. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.