Open Standards andOpen Source in

Datacenter Management

蔡鎮宇 Chen-Yu Tsai <wens@csie.org>

2014/4/11 OSDC 2014 1

Who am I?• Software Engineer @ CloudMosa, Inc.

• System Administrator for 10+ years starting in college

• Skills: breaking and fixing things

2014/4/11 OSDC 2014 2

Overview

•Monitoring

•Management

•Provisioning

2014/4/11 OSDC 2014 3

- Monitoring -

2014/4/11 OSDC 2014 4

Log Everything!

2014/4/11 OSDC 2014 5

Where to start?

2014/4/11 OSDC 2014 6

MRTG

2014/4/11 OSDC 2014 7

Based on SNMPSupported by most network devices

2014/4/11 OSDC 2014 8

Exports data and metrics

2014/4/11 OSDC 2014 9

Network traffic counters– used by MRTG

2014/4/11 OSDC 2014 10

Known MAC addresses- Map the network

2014/4/11 OSDC 2014 11

2014/4/11 OSDC 2014 12

2014/4/11 OSDC 2014 13

Whatever the device supports

Look up vendor specific MIBs

2014/4/11 OSDC 2014 14

RRDToolTime Series Database

2014/4/11 OSDC 2014 15

MRTG uses it

2014/4/11 OSDC 2014 16

Munin uses it

2014/4/11 OSDC 2014 17

… uses it

2014/4/11 OSDC 2014 18

Write your own!

2014/4/11 OSDC 2014 19

2014/4/11 OSDC 2014 20

2014/4/11 OSDC 2014 21

Munin –Resource Monitoring

2014/4/11 OSDC 2014 22

System is slow…

2014/4/11 OSDC 2014 23

CPU usage?

2014/4/11 OSDC 2014 24

2014/4/11 OSDC 2014 25

Memory usage?

2014/4/11 OSDC 2014 26

2014/4/11 OSDC 2014 27

Disk I/O?

2014/4/11 OSDC 2014 28

2014/4/11 OSDC 2014 29

Web requests?

2014/4/11 OSDC 2014 30

2014/4/11 OSDC 2014 31

Use plugins from standard set

2014/4/11 OSDC 2014 32

Or write Your Own!

2014/4/11 OSDC 2014 33

2014/4/11 OSDC 2014 34

Aggregate DataManual configuration for now

2014/4/11 OSDC 2014 35

2014/4/11 OSDC 2014 36

Others• Monitoring

• Xymon (Hobbit)

• Nagios

• Cacti

• Data collection / Graphing• Graphite

• ZipKin (Twitter)

• Log collection• Scribe (Facebook)

2014/4/11 OSDC 2014 37

Management

2014/4/11 OSDC 2014 38

IPMIIntelligent Platform Management Interface

2014/4/11 OSDC 2014 39

2014/4/11 OSDC 2014 40

Image from Wikipedia

Built into most BMCs

2014/4/11 OSDC 2014 41

Out-of-Bandvs

Side-band

2014/4/11 OSDC 2014 42

Power ControlOn, Off, Reset

2014/4/11 OSDC 2014 43

Serial over LANConsole Access

2014/4/11 OSDC 2014 44

Boot OrderForce PXE boot?

2014/4/11 OSDC 2014 45

SSHSecure Shell

2014/4/11 OSDC 2014 46

SSH Public Key Authentication

Don’t need to input password every time.

2014/4/11 OSDC 2014 47

OmniTTYConsole-based interactive SSH multiplexer

2014/4/11 OSDC 2014 48

Parallel-SSH (pssh)Parallel versions of OpenSSH

2014/4/11 OSDC 2014 49

FabricScriptable, Parallel SSH

2014/4/11 OSDC 2014 50

Provisioning

2014/4/11 OSDC 2014 51

DHCPNetwork Provisioning

2014/4/11 OSDC 2014 52

PXE BootBoot over Network

2014/4/11 OSDC 2014 53

Auto-configurationvia DHCP

Network Switches

2014/4/11 OSDC 2014 54

Kickstart/PreseedAutomatic Install

2014/4/11 OSDC 2014 55

ChefPuppet

Disclaimer: We don’t use them.

2014/4/11 OSDC 2014 56

Custom PackagesPut programs/services/settings

into native packages.

2014/4/11 OSDC 2014 57

Apt-cacher-ngWeb cache for package files

2014/4/11 OSDC 2014 58

Put It All Together

2014/4/11 OSDC 2014 59

2014/4/11 OSDC 2014 60

With the proper hardware/software

2014/4/11 OSDC 2014 61

Datacenters Become Manageable

2014/4/11 OSDC 2014 62

2~3 People2k+ Nodes in4 Datacenters

2014/4/11 OSDC 2014 63

Hands free afterracking and cabling

2014/4/11 OSDC 2014 64

2014/4/11 OSDC 2014 65

10k nodes?

2014/4/11 OSDC 2014 66

100k nodes?

2014/4/11 OSDC 2014 67

Evolve!

2014/4/11 OSDC 2014 68

We are Hiring!

2014/4/11 OSDC 2014 69

Thank You

2014/4/11 OSDC 2014 70