mpack based operations model
TRANSCRIPT
Jayush Luniya
Apache Ambari PMC
Principal Software Engineer @ Hortonworks
MPACK BASED OPERATIONS MODEL
Agenda
Agenda
Agenda
Stack Based Operations Model
Management Packs – V1
Limitations
Mpacks Based Operations Model
Summary
Q&A
Stack Based Operations Model
Agenda
Stack Based Operations Model
Stack Based Operations Model
Ambari built on a standardized stack based operations model
Pluggable architecture; no vendor lock-in
Stacks wrap services of all shapes and sizes
Consistent definition & lifecycle control layer
Consistent approach for managing & monitoring services
Stacks embedded in Ambari
Management Packs – V1
Agenda
Management Packs – V1
Introduced in 2.4
V1 release based on existing stack based operations model
First take on decoupling stack releases
Bundles stacks, add-on services
Final stack definition is overlay of multiple mpacks
Stacks don’t need to be embedded in Ambari
Management Packs – V1
Service Level Extension Points
Extension Apache JIRA Apache Wiki
Stack Advisor Extensions AMBARI-15226 Ambari Wiki
Role Command Order Extensions AMBARI-9363 Ambari Wiki
Quick Links Extensions AMBARI-11268 Ambari Wiki
Service Repos Extensions AMBARI-15538 Ambari Wiki
Upgrade Pack Extensions AMBARI- 15388 Ambari Wiki
Limitations
Agenda
Limitations
Cluster attached to single stack Cannot deploy HDFS from HDP, NiFi from HDF in same cluster
Entire cluster upgrade
Stacks are monolithic
Ambari upgrade for refreshing stacks
No clear upgrade story for add-on services
Ownership of service deployment scripts
Mpack Based Operations Model
Agenda
Goals
Replacement for monolithic stacks
Cluster to multiple mpacks
Scenario: Deploy HDP +HDF cluster
Decouple Ambari core release
Independent release cycles
First class support for add-on services
Support dynamic installation
Modular upgrades
New Concepts
Deployment artifact
Individual service ownership
Independent release cycles
Examples
HDFS-3.0.0-packlet
SPARK-2.0.0-packlet
Distribution artifact
Composition of packlets
Modular release & upgrades
Examples
ODPi-2.0.0-mpack
HDP-3.0.0-mpack
HDF-3.1.0-mpack
Packlets Management Packs (Mpacks V2)
New Concepts
Catalog of management packs
Compatibility between mpacks
Search and discover mpacks
Support multiple mpack repos
Examples
ODPi Mpack Repository
Hortonworks Mpack Repository
Mpack Repository
Mpack Based Operations Model
Mpack Processing Engine
Mpack Repository
DistroX3.0.0
DistroY3.2.0
Release Management
Completely decouple Ambari releases
Service ownership with domain experts
Independent release cadence
Mpack Based Upgrades
Modular upgrades
Upgrade subset of services in a cluster that belong to mpack
Upgrade packs defined at mpack level
Download new version of mpack and kickoff upgrade
Upgrade Pack
“upgrade-from” instead of “upgrade-to”
Add-on Mpacks
First class support for 3rd party services
Mpacks provide a 3rd party onboarding platform
Add-on services not overlaid into stack
Independent release management
Independent and modular upgrades
Summary
Agenda
Summary
Modular build & release
Independent release mgmt
Modular upgrades
3rd party onboarding platform
Multi-everything vision
Service level refactoring
Resource management library
No network connectivity
Ownership
Testing
Key Takeaways Challenges
Summary
Future YARN Services
Umbrella Apache Epic
AMBARI-14714 – Multi Everything Architecture
Apache Epic
AMBARI-19621 – Mpack Based Operations Model
Dataworks Summit Talk
Future of Apache Ambari – Jaimin, Jayush
Q&A
Agenda
Backup Slides
Agenda
