open stack in sina

OpenStack in Sina

@程辉freedomhui@gmail.com

Agenda

OpenStack Overview Architecture Analysis Integration Changes Sina Contributions

AWS模式的巨大成功 构建了完整的云计算生态系统 通过Web Service(API)管理一切服务 * 完全面向服务架构 SOA(Service-Oriented

Architecture)* 事实上的 IaaS 标准　 成功的商业模式

*https://plus.google.com/112678702228711889851/posts/eVeouesvaVX*http://coolshell.cn/articles/5701.html

More Detail: http://aws.amazon.com/products/

OpenStack横空出世 目标： AWS开源实现 Rackspace & NASA联合成立

Swift Nova

OpenStack Companies

More detail: http://www.openstack.org/community/companies/

来源：《 OpenStack, OpenNebula， Eucalyptus， CloudStack社区活跃度比较》 http://www.qyjohn.net/?p=1856

Open Source Apache 2.0 license, NO ‘enterprise’ version

Open Design Open Design Summit

Open Development Anyone can involve development process Open development management via Launchpad & Github

Open Community OpenStack Foundation in 2012

OpenStack Mission

"To produce the ubiquitous Open Source cloud computing platform that will meet the needs of public and private cloud providers regardless of size, by being simple to implement and massively scalable."

OpenStack Projects

Core Projects OpenStack Compute(Nova) OpenStack Object Storage(Swift) Image Service (Glance) Identity (Keystone) Dashboard (Horizon) Network Connectivity (Quantum)

Community Projects Melange Altas-LB Crowbar Juju RedDwarf Burrow

AWS OpenStack

EC2 nova

S3 swift

EBS nova-volume

ELB Atlas-LB

SQS Burrow

Console Dashboard

IAM Keystone

VPC Quantum

RDS RedDwarf

Architecture Overview

Detail Overview

Where to Get Started?

Ubuntu 12.04 server 集成 OpenStack

Trystack.org申请测试账号

devstack.sh一键安装

OpenStack Development

Bug tracking

Authorization(group membership)

Feature planning(Blueprints)

Hosting code &formal docs

Mailing lists

User support(Answers)

Wiki

Continuousintegration

Informal docs

Nova Key Features

ReST-base API Asynchronous communication Horizontally scalable Shared nothing architecture* Distribute everything Test everything 100% Python Based

* http://en.wikipedia.org/wiki/Shared_nothing_architecture* http://wiki.openstack.org/BasicDesignTenets

OpenStack Compute: Nova

nova-apiCompute API Server

OpenStack API, EC2 compatibility API

nova-computeCompute worker

Manage compute host and VMs

Libvirt(QEMU,KVM,LXR), XenServer and XCP, ESX(i)*

nova-networkNetwork controller

Manage network resources: IPAM, VLAN, NAT

*http://wiki.openstack.org/HypervisorSupportMatrix

OpenStack Compute: Nova(cont.)

nova-schedulerDetermines the placement of new resources

nova-volumeBlock storage, remote attach a LVM volume using iSCISI protocol

Like Amazon EBS, but far way from mature

RabbitMQMessage Queue

Cast and RPC Call for services

Keystone: Concept

User/Tenant

Authentication/Authorization

Token

Service/Endpoint

Role

Keystone: User Case

Nova Network

L2FLAT, FLATDHCP, VLAN

L3IPAM(IP Address Management)

Fixed IP, Floating IP

Gateway, NAT, VPN

Quantum

Quantum BasicsNova: virtual server

Quantum: virtual network

Quantum : Expose a API for creating virtual networks and attaching

instances(e.g.,novaservers) to those networks Manage switches(virtual or physical) in the data center to

implement connectivity described via API Provide a“plugin” architecture to leverage support using

different back-end technologies

Quantum: available plugins

Open vSwitch - Builds isolated networks with OVS and L2-in-L3 tunnelCisco UCS - Isolation based on VLAN and net-profiles applied to Cisco UCS converged network adaptersLinux Bridge - Build isolated networks with VLAN interfaces and linux bridges

- Works with every Linux DistroNTT-Data Ryu - Acts as a proxy for the NTT Ryu platformNicira NVP - Acts as a proxy for the Nicira NVP platform

Swift: Storage Types

Types Protocol Application

Block Storage SATA, SCISI, iSCISI SAN, NAS, EBS

File Storage Ext3/4, XFS, NTFS PC, Servers, NFS

Object Storage HTTP, REST Amazon S3, Google Cloud Storage, Rackspace Cloud Files

Specific Storage Specific protocol based on tcp

MySQL, MongoDB, HDFS

We want a Object Storage like Amazon S3.

Swift vs

Amazon S3

Features Swift Amazon S3

object/bucket CRUD √ √

account/bucket/object ACL √ √

object metadate √ √

large object √ √

rate limit √ √

expiring object √ √

static web √ √

REST API √ √

Account support √ X

Account metadata √ X

Bucket metadata √ X

Bucket sync across cluster √ X

Object versioning X √

Log to bucket X √

Notification X √

Reduced Redundancy Storage X √

SOAP API X √

Server Side Encryption X √

BitTorrent protocol X √

Swift Evaluation

Extremely Durable and Highly Available Superior Scalability Linear Growth of Performance Symmetric Architecture No Single-failure Simple & Reliable

Swift Components

The Ring: Mapping of names to entities (accounts,containers, objects) on disk.

Stores data based on zones, devices, partitions, and replicas Weights can be used to balance the distribution of partitions Used by the Proxy Server for many background processes

Proxy Server: Request routing, exposes the public API Replication: Keep the system consistent, handle failures Updaters: Process failed or queued updates Auditors: Verify integrity of objects, containers, and account

Swift Architecture

Load Balancer

Proxy Server

Object Server

Container Server

Account Server

Zone1

Proxy Server

Object Server

Container Server

Account Server

Zone2

Proxy Server

Object Server

Container Server

Account Server

Zone3

Proxy Server

Object Server

Container Server

Account Server

Zone4

Proxy Server

Object Server

Container Server

Account Server

Zone5

PUT abc.pngGET abc.png

• 1 Zone = 1 Physical Server with 12x2T disk• Write/Read applies quorum protocol

31

Swift Installation

Physical Deployment

disk1 disk2 disk3 disk4

sda

……

sdb sdc

disk5

sdd

disk12

sdk

Storage NodesOS installation

Swift packagesProxy Server

Account ServerContainer Server

Object Server

raid 1

Conclusion 核心功能基本可用，但稳定性需要加强 云服务 (web service)比较丰富 起步虽晚，但发展飞快， OpenStack生态系统正在形成

逻辑结构清晰、文档丰富、源码规范易懂，便于二次开发

Open [Source | Desgin | Development | Community]

Integration Challenges

Best Network Topology Security Enhancement Load Balancer CDN Services Metering & Billing

Infrastructure & Platform

Physical ServersTraditional Operation

Virtualization Platform(IaaS)●VM Management System(VMMS) → Sina Web Service(SWS)●VMMS is private solution developed in-house●SWS is based on OpenStack

Application Platform(PaaS)●Virtual Host → Sina App Engine(SAE)●SAE provides both Public and Private Service.●Proved to be Efficient and Robust

新浪云计算

Nova Network

Networking is the biggest challenges for IaaS

Network Topology:

• VLAN

• FlatDHCP

• FlatDHCP & Multihost

Network Topology (VLAN)

Drawback:• Pre-allocate network for future projects• Hard-limit of vlan 4096• Traffic bottleneck in the gateway/NAT

Capability:• Accessibility of VMs within one tenant• Isolation of VMs from different tenants• VM is able to access public network• VM can be accessible from public network• Isolation between virtual network and

internal network

Network Topology(Flat)

Capability:• Accessibility of all VMs in the fixed IP range• VM is able to access public network• VM can be accessible from public network• Full isolation between virtual network and

internal network

Bonus:• Do not need pre-allocate for new projects• Eliminating bottleneck between tenants

Drawback:• Tenant isolation has gone• Traffic bottleneck still exists in NAT

Network Topology(Flat & Multihost)Capability:• Accessibility of all VMs in the fixed IP range• VM is able to access public network• VM can be accessible from public network

Bonus:• Totally distributed architecture avoid single-

point failure.• Multiple gateway eliminates NAT bottleneck• High speed between OS regions

Drawback:• Tenant isolation lessens• Need security facility(SWS-filter) to protect

intranet

If security problems were solved, this would be our best choice!

Security in OpenStack

Static filters --- L2 Filter

MAC, IP, and ARP spoofing protection Not configurable Defined in /etc/libvirt/nwfilter/*.xmlImplemented by ebtables ebtables -t nat --list

Security Group --- L3 Filter

Role-based firewall One security group is a RoleIngress filtering Target is the instance Source can be CIDR or another groupImplemented by iptables See details: iptables -t filter -n -L Whitelist mechanism(ACCEPT rules)

Security Enhancement

SWS Filter

Prevent Intranet Penetration• Intranet is the internal network outside of

OpenStackEgress filtering• Target is internal network• Source is instances in OpenStackImplementation• Whitelist mechanism(ACCEPT rules)• On the top of nova-filter-top Forward

Chain

Rational• SWS filter is managed by cloud manager • Only explicit authorized packets can reach Internal network C • Packet should be controlled within Compute Node

Security Enhancement

Security Group VS SWS Filter

Load Balancer

Goals

Load Balance • Dispatch request• Support multiple routing algorithm• Health check

Acceleration• Reality: narrow bandwidth between ISPs• Building fiber channels from ISPs to pivot• Given the same endpoint within user’s ISP

IPv4 Shortage• Reality: dozens of public IPs support

hundreds of VMs• IPv4 has been exhausted• IPv6 is not realistic yet in China

Unicom Others ISPMobileTelecom

Pivot

Smart DNS

DNS Acceleration Design

High speed fiber channel

Public Network

L7 Load Balancer

Layer 7 Load Balancer

Consideration:1. dispatch request by Host header2. nginx module

L4 Load Balancer

Layer 4 Load Balancer

Consideration:1. dispatch request by TCP port2. lvs + haproxy

ssh –p 2000 root@socket.abc.com

46

Kanyun: Monitoring system

Aggregator

API daemon

Compute

Network

Storage

Worker

Worker

Responds to client requestCalculates/

stores metrics

Retrieve usage info

DashboardRDBMS

Billing

NoSQLhttp://github.com/lzyeval/kanyun

47

Dough:Billing system

Farmer API daemon

Compute

Network

Storage

Monitoring(Metering)

Collector

CollectorSubscribe orunsubscribeproducts /Query info

Dispatch jobs

Check status /Retrieve usage /

Create purchases

Dashboard

RPCDatabase

Client

RDBMS

http://github.com/lzyeval/dough

Sina Contributions

48

Top 10 Bugfix Company

Sina Contributions

49

Sina creating open source project “Dough” to contribute metering & billing capability

Present in OpenStack Design Summit & Conference

Q & A

@程辉Technical Manager @ Sina

Twitter：@freedomhuiEmail： chenghui@staff.sina.com.cnGtalk: freedomhui@gmail.com