deploy on cloud - where are we? qiming teng [email protected]
TRANSCRIPT
DEPLOY ON CLOUD - WHERE ARE WE?
Qiming [email protected]
AGENDA
Heat Basic
Heat SoftwareConfig
Heat BootConfig
Heat DockerCompose
HeatKubelet
Heat Docker Plugin
Heat Ansible
Senlin
Convergence
FROM TEMPLATE TO STACKversion: xxxparameters: key: mykeyresources: server: OS::Nova::Server key: {get: key} image: gold flavor: m1.small network: {get: network} volume: {get: volume} network: OS::Neutron::Network ... volume: OS::Cinder::Volume ...
Heat
Nova
Neutron
Cinder Volume
Instance
• orchestrator not bandmaster• deployment tool
NOT JUST A DEPLOYMENT TOOL, PLEASE!!!
version: xxxparameters: key: mykey
resources: server: OS::Heat::ServerGroup count: 5
volume: OS::Cinder::Volume ...
Heat
S0 S1 S2 S3 S4
(1) Parallelized Operation
(2) listen
Observed States
S0 S1 S2 S3 S4
Desired States
(3) converge
S2 S3
InstanceOS
App
SOFTWARE-CONFIG/SOFTWARE-DEPLOYMENT
version: xxxparameters: key: mykeyresources: config: OS::Heat::SoftConfig group: script config: # your script server: OS::Nova::Server key: {get: key} image: gold flavor: m1.small network: {get: network} volume: {get: volume} user_data: {get: config}
Heat
Nova
ChefPuppe
t
• OS::Heat::CloudConfig• OS::Heat::SoftwareConfig• OS::Heat::StructuredConfig• OS::Heat::SoftwareDeployment• OS::Heat::SoftwareDeployments• OS::Heat::SoftwareComponent
SOFTWARE-CONFIG/SOFTWARE-DEPLOYMENTversion: xxx
resources: config-1: OS::Heat::SoftConfig deploy-1: config: config-1 server: server-1 server-1: OS::Nova::Server
config-2: OS::Heat::SoftConfig deploy-2: depends_on: deploy-1 config: config-2 server: server-2 server-2: OS::Nova::Server
server-1 server-2
config-1 config-2
depends on
DECLARATIVEserver-1 server-2
config-1 config-2
depends on
InstanceOS
SOFTWARE-CONFIG/SOFTWARE-DEPLOYMENT
App
ChefPuppe
tQ: What is it?
[1] http://git.openstack.org/cgit/openstack/os-collect-config/[2] http://git.openstack.org/cgit/openstack/os-refresh-config/ [3] http://git.openstack.org/cgit/openstack/os-apply-config/ [4] http://git.openstack.org/cgit/openstack/heat-templates/tree/hot/software-config/elements
A: THEY are collection of agents including
• os-collect-config [1]• os-refresh-config [2]• os-apply-config [3]• heat-config-script [4]• heat-config-puppet [4]• heat-config-docker-compose [4]• heat-config-kubelet [4]• ...
InstanceOS
SOFTWARE-CONFIG/SOFTWARE-DEPLOYMENT
App
ChefPuppe
tQ: How does the agent authenticate?
NOTE: There is a side path of generating EC2 tokens
A: Heat does secret job in the background
• heat domain• created during setup
• stack_domain_project• name stack id
• stack_domain_user• name resource name
• password ?• uuid.uuid4().hex
InstanceOS
SOFTWARE-CONFIG/SOFTWARE-DEPLOYMENT
App
ChefPuppe
tQ: How are the agents injected/installed?
[1] http://git.openstack.org/cgit/openstack/diskimage-builder/[2] http://git.openstack.org/cgit/openstack/heat-templates/tree/hot/software-config/boot-config
A: There are two ways
• disk-image-builder (dib) • a TripleO project [1]• prebuilt images for use
• Heat boot-config [2]• install these agents on the fly
when VM boots up
goal install agents required to use certain software deployments in templates [1]
how it's used define an env yaml file with a Heat::InstallConfigAgent resource refer to this resource in your server.properties.user_data
HEAT BOOT-CONFIG
[1] http://git.openstack.org/cgit/openstack/heat-templates/tree/hot/software-config/boot-config/
env yaml
inst-config
config-config
start-config
#!/bin/shyum install ..
#!/bin/shcat << EOF ...mkdir ...#!/bin/shsystemctl enable ...systemctl start ...
heat stack-create -f template -e environment mystack
template
server
user_data
environmentInstAgent
config
MIME
goal prepare guest environment for container deployment with docker-compose
HEAT CONTAINER AGENT
install_container_agent
write_image_pull_script
install_container_agent
#cloud-configwrite_files: # write a script that will # grab specified image via # 1. curl + docker load, or # 2. docker pull#!/bin/sh# 1. create a service:# heat-container-agent # 2. enable/start docker svc# 3. enable/start agent svc
template
server
user_data
environmentInstAgent
config
# execute the image pull script to get the <image>
# docker run --name heat-container-agent ... \ <image>
heat-container-agent service
sample image: http://git.openstack.org/cgit/openstack/heat-templates/tree/hot/software-config/heat-container-agent/
goal a 'hook' that uses 'docker-compose' to deploy containers [1] an element that you will build into your guest image
HEAT DOCKER COMPOSE
[1] http://git.openstack.org/cgit/openstack/heat-templates/tree/hot/software-config/elements/heat-config-docker-compose
template
server
user_data
config group:docker-compose inputs: [env_files] config: db: image: redis web: image: nginx
Instance
OS
os-collect-config docker-composeup -d --no-build
heat-config
heat-docker-compose
container
container
metadata
env files
yml file
template
server
user_data
config group: kubelet inputs: [env_files] config: containers: - name: doecho image: busybox command: ...
Instance
OS
os-collect-configmetadat
a
heat-config
config
hook-kubelet
goal a 'hook' that uses 'kubelet' agent from kubernetes to deploy containers
an element that you will build into your guest image
HEAT KUBELET
[1] http://git.openstack.org/cgit/openstack/heat-templates/tree/hot/software-config/elements/heat-config-docker-compose
container
container
poll
kubelet.service
/opt/heat-docker/images.tar
preinstalled
preinstalled
poll
template
server
user_data
config group: ansible inputs: [...] config: # your ansible book # here
Instance
OS
os-collect-configmetadat
a
heat-config
playbook
goal a 'hook' that uses 'ansible' to configure A instance
HEAT CONFIG ANSIBLE
[1] http://git.openstack.org/cgit/openstack/heat-templates/tree/hot/software-config/elements/heat-config-ansible
ansible-playbook -i localhost <file>
hook-ansible.pycontaine
rapplicati
on
A Resource Type (Contrib[1]) for Heat DockerInc::Docker::Container built on docker-py [2]
HEAT DOCKER PLUGIN
[1] http://git.openstack.org/cgit/openstack/heat/tree/contrib/heat_docker/[2] https://github.com/docker/docker-py
image: stringcommand: listhostname: stringuser: stringstdin_open: booleantty: booleanmem_limit: integerports: listenvironment: listdns: listvolumes: mapcpu_shares: integercpuset: string
CREATE
privileged: booleanbinds: map (volumes)volumes_from: listport_bindings: maplinks: maprestart_policy: mapcap_add: listcap_drop: listread_only: Booleandevices: list
START
infonetwork_infonetwork_ipnetwork_gatewaynetwork_tcp_portsnetwork_udp_portslogslogs_headlogs_tail
SHOW
SERVICES ROADMAP ON SUPERVESSEL
17
SuperVessel Cloud Infrastructure
SuperVessel Cloud Service
SuperVessel Big Data and
HPCService
Super Class
Service
OpenPOWER Enablement
Service
Super Project Team
Service
1.VM and container service
2.Storage service
3.Network service
4.Accelerator as service
5.Image service
1.Big Data: MapReduce (Symphony), SPARK
2.Performance tuning service
1.X-to-P migration: AutoPort tool
2.OpenPOWER new system test service
1.On-line video courses
2.Teacher course management
3.User contribution management
1.Project management service
2.DevOps automation
Storage IBM POWER serversOpenPOWER server FPGA/GPU
Docker
(Online) (Online) (Preparing)(Online)
SUPERVESSEL
HEAT -- TRIED NOT TO JUST A DEPLOYER Supports to High-Availability
OS::Heat::HARestarter recreates a resource when failure detected
Supports to Auto-Scaling OS::Heat::InstanceGroup OS::Heat::ResourceGroup OS::Heat::AutoScalingResourceGroup OS::Heat::ScalingPolicy AWS::AutoScaling::AutoScalingGroup AWS::AutoScaling::ScalingPolicy AWS::AutoScaling::LaunchConfiguration
AUTOSCALING REORG
BLUEPRINTS ON REWORKING HEAT AUTOSCALING
BP Priority Description
autoscaling-api-resources high Heat resources invoking AS APIs
as-api-group-resource high ScalingGroup resource wrapping AS API's group functionality
as-api-policy-resource high ScalingPolicy resource wrapping AS API's policy functionality
as-api-webhook-resource high Webhook resource wrapping AS API's execution of webhooks
autoscaling-api-client high A python client for Heat to interact with AS API
autoscaling-api - A separate service for the implementation of autoscaling w/ Heat
as-engine - A separate engine/service for autoscaling support AS API
as-engine-db - A DB dedicated to autoscaling, using schema created in as-lib-db
as-lib - A separate module to be used by the AS service
as-lib-db - A DB for autoscaling bookkeeping
DEPENDENCIES AMONG BPS
autoscaling-api-resources
as-api-group-resourcesas-api-policy-resources as-api-webhook-resources
autoscaling-api-client
autoscaling-api
as-engine
as-engine-db
as-lib
as-lib-db
OVERVIEW OF AUTOSCALING
A STRUGGLE BEFORE SENLIN STARTS Should we do this within Heat or outside Heat?
Within Heat pros
smooth transition; strict reviews better quality cons
long (maybe forever) code churn; eventually, a dedicated service is needed, thus the pain to switch over
Outside Heat pros
quick development; less code churn to Heat cons
high requirements of skills and cycles; eventual switch over, i.e. another animal to feed in the OpenStack zoo
We choose OUTSIDE HEAT There are Heat core team supports to this approach We see a lot potentials in a standalone clustering service We don't have to do everything from scratch we "borrow" and "steal" code
whenever license permits
WHAT DO WE REALLY NEED?
Scalable
Load-Balanced
Highly-Available
Manageable
......
of any (OpenStack) objects
-- What is missing from OpenStack
Senlin[Chinese Pinyin for Forest]
SENLIN ARCHITECTURE
Senlin Engine
Senlin API
Senlin Database
Senlin Client
REST RPC
Profiles
Policies
ER DIAGRAM
cluster
nameuuiduserprojectparentprofile_idstatus
profile
nameuuidtypespec
node
nameuuidcluster_idprofile_idindexstatuscreated_timeupdated_time
policy
nameuuidtypelevelspec
placement_policy
update_policy
deletion_policy
scaling_policy
health_policy create()delete()update()add()remove()
lb_policy
«policy_type»
os.nova.server
os.cinder.volume
os.keystone.user
os.heat.stack
cluster_policy
cluster_idpolicy_idenabledlevelcooldownpriority
«profile_type»action
contextactioninputsoutputs
webhook
targetactionuser
APIplugins
plugins
SENLIN OPERATIONS (ACTIONS)
Cluster• CREATE• DELETE• UPDATE• LIST• SHOW• ADD_NODES• DEL_NODES• SCALE_OUT• SCALE_IN• POLICY_ATTACH• POLICY_DETACH• POLICY_UPDATE
Node• CREATE• DELETE• UPDATE• LIST• SHOW• JOIN• LEAVE• MIGRATE
Policy• CREATE• UPDATE• DELETE• LIST• SHOW
Profile• CREATE• UPDATE• DELETE• LIST• SHOW
Action• LIST• SHOW
Event• LIST• SHOW
Webhook• CREATE• DELETE• LIST• SHOW
RELATION TO OTHER PROJECTS Senlin provides the array data type for cloud programming
Senlin CeilometerHeat
Nova Cinder Neutron Swift Keystone
Horizon
Primitive Data Types
Complex Data Types
struct person { int age; char name[0];}
person team[10]; // Senlin cluster of Heat stacks
// Senlin cluster of nova servers
// Heat stack containing senlin clusters
CURRENT STATUS
Code Base http://git.openstack.org/cgit/stackforge/senlin
including API design under doc subdirectory http://git.openstack.org/cgit/stackforge/python-senlinclient
IRC channel: #senlin
Date Milestone
2014-12-10 Initial Git Repository inside CRL
2014-12-25 Migration to github.com
2014-01-14 Introduction to IBM Heat Community
2015-01-19 Weekly conference call started
2015-02-06 Announcement on IBM openstack-dev mailinglist
2015-02-13 Email to OpenStack Heat Core team
2015-03-16 Senlin project accepted to OpenStack StackForge
2015-03-21 Senlin client project accepted to OpenStack StackForge
2015-03-26 Project announcement in community (link)
NEXT STEP Complete AutoScaling support
Cross-Region AutoScaling
Features Pipeline (draft)
High Priority Middle Priority Low Priority
Item W Item W Item W
Event Listening ** Horizon Plug-in * Metrics Collection *
Scavenger Process * User Defined Actions/Ansible *** AWS Compatible API ***
Multi-Engine Support * Quota Enforcement * Integration with Mistral **
Test Case Coverage ** Event Notification * Cluster suspend/resume **
Babican Support * Scheduled actions *
VPNaaS support ** Interaction with Congress *
Nova ServerGroup API * Integration with Tooz **
THANK YOU!