hortonworks technical workshop: apache ambari
TRANSCRIPT
Apache Ambari WorkshopStacks, Blueprints, Views
We Do Hadoop
Ajay Singh, Ali Bajwa, Sean RobertsPartner Solution Engineering
March-2015
Who
Ajay Singh- Director- [email protected]
Ali Bajwa- Engineer- [email protected]
Sean Roberts- Engineer, London & EMEA- @seano
Page 4 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
How do you Operate a Hadoop Cluster?
Apache Ambari is a platform to provision, manage and monitor
Hadoop clusters
Page 5 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Apache Ambari Themes
Operate Hadoop at Scale
Deliver the core operational capabilities to provision, manage and monitor Hadoop clusters at scale.
Integrate with the Enterprise
Robust API for integration with existing enterprise systems, such as Teradata Viewpoint and Microsoft SCOM.
Extend for the Ecosystem
Provide extensible platform for Customers, Partners and the Community to, such as Stacks and Views.
Page 6 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Hadoop Operations Platform
Apache Ambari is a 100% open source platform for provisioning, managing and monitoring Apache Hadoop clusters
AMBARI WEB
Others
compute&
storage. . .
. . .
. . compute&
storage
.
. EXTEND
AMBARI REST API
AMBARI SERVERPROVISION | MANAGE | MONITOR
Integration with Existing Tools
OPERATE
AMBARI STACKS
Page 7 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Innovation at the Speed of CommunityAn active 100% open source Community:
• 90+ Contributors / 45+ Committers
• 270+ Ambari User Group Members
2014
Dec
Graduates to Apache Top Level Project2013
Dec
Apache Ambari 1.7 ReleasedAdds support for Hortonworks Data Platform 2.2Adds new operational and extensibility capabilities
Page 8 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Recent Ambari Releases
Ambari 1.6.0May 2014
Ambari 1.5.1Apr 2014
Ambari 1.5.0Apr 2014
Ambari 1.6.1Jul 2014
IntroducedAmbari Stacks
IntroducedAmbari Blueprints
HDP2.1 GA
Ambari 1.7.0Dec 2014
HDP2.2 GA
IntroducedAmbari Views
Page 9 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Extensibility Features
• To add new Services (ISV or otherwise) beyond HDP Stack• To customize a Stack for customer specific environments
• To use Ambari for automating cluster installations• To share best practices on layout and cluster configuration
• To extend and customize the Ambari Web UI• Add new capabilities, customize existing capabilities
Stacks
Blueprints
Views
Goal: Extend Ambari without hard-coding in Ambari
1.5.0
1.7.0
1.6.0
Page 10 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Anatomy of Ambari Extension Points
AmbariServer
AmbariAgentAmbari
AgentAmbariAgent
AmbariWeb
StacksStacks
Stacks
javajs python
Ambari Views
Ambari Stacks
Ambari Blueprints
Ambari API
Ambari Community
Apache Ambari: http://ambari.apache.org ● wiki: https://cwiki.apache.org/confluence/display/AMBARI/Ambari ● mail lists: https://ambari.apache.org/mail-lists.html ● issues: https://issues.apache.org/jira/browse/AMBARI
Hortonworks:● forum: http://hortonworks.com/community/forums/forum/ambari/● social media: twitter, facebook, slideshare, linkedin, ...
Partner Resources
This Workshop:● github: http://git.io/jmz4 (https://github.com/abajwa-hw/ambari-workshops)● blog: http://hortonworks.com/blog/apache-ambari-technical-workshop/
Past & Future Workshops:● http://hortonworks.com/partners/learn/
Upcoming meetups and events:● http://hortonworks.com/events/
Page 15 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Stack TerminologyTerm Definition Examples
STACK Defines a set of Services, where to obtain the software packages and how to manage the lifecycle. HDP-2.0, HDP-1.3.3
SERVICE Defines the Components that make-up the Service. HDFS, NAGIOS, YARN
COMPONENT The building-blocks of a Service, that adhere to a certain lifecycle.
NAMENODE, DATANODE, OOZIE_SERVER
CATEGORY The category of Component. MASTER, SLAVE, CLIENT
Page 16 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Stack MechanicsStacks define Services + ReposWhat is in the Stack, and where to get the bits
Each Service has a definitionWhat Components are part of the Service
Each Service has defined lifecycle commandsstart, stop, status, install, configure
Lifecycle is controlled via command scriptsAbility to define “custom” commands
AMBARISERVER
Stack
Command Scripts
Service Definitions
AMBARIAGENT/S
AMBARIAGENT/S
AMBARIAGENT/S
pythonxml
Repos
Page 17 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
Stacks Support Inheritance
HDP 2.0 Stack
HDP 2.1 Stack
• Defines a set of Service definitions• Default service configurations and command scripts
• Overrides any Service definitions, commands and configurations
• Adds new Services specific to this Stack
Before Ambari
● Manual installation● Manual configuration● Manual distribution of configuration● XML● XML● XML ...
Automation with Ambari Blueprints
Stack DefinitionComponent
Layout& Configuration
BLUEPRINT
BLUEPRINT INSTANTIATE CLUSTER
2 HTTP calls for a Deployed Cluster
{ "Blueprints": { "stack_name": "HDP", "stack_version": "2.2" }, "host_groups": [ { "name": "workers", "components": [ { "name": "DATANODE" }, { "name": "JOURNALNODE" }, { … … } }, { "name": "master-1", "components": [ { "name": "NAMENODE" }, { … … } }]}
{ "Blueprints": { "blueprint": "myblueprint", "default_password": "awesome", "configurations": [ { "hive-site": { "properties": { "hive.execution.engine": "tez" }}}], "host_groups": [ { "name": "master-1", "hosts": [ { "fqdn": "hdp00" } ]}, { "name": "workers" "hosts": [ { "fqdn": "hdp01" }, { "fqdn": "hdp01" }, { "fqdn": "hdp.." }, { "fqdn": "hdp99" }]}]}
1) POST /api/v1/blueprints/myblueprint 2) POST /api/v1/clusters/mycluster
Export Cluster to BlueprintGET /api/v1/clusters/my_cluster?format=blueprint
{ "configurations" : [ { "cluster-env" : { "cluster_name" : "sandbox", ... }}, { "hive-site" : { "hive.execution.engine" : "tez", ... }} ], "host_groups" : [ { "name" : "host_group_1", "configurations" : [ ], "components" : [ { "name" : "NAMENODE" }, { "name" : "HIVE_SERVER"}, ... ], "cardinality" : "1" }], "Blueprints" : { "stack_name" : "HDP", "stack_version" : "2.2" }}
Live Demo
Command snippets: http://nbviewer.ipython.org/github/seanorama/ambari-bootstrap/blob/master/api-examples/ambari-workshop-blueprints.ipynb
For API examples including Python versions in an ipython notebook:https://github.com/seanorama/ambari-bootstrap/tree/master/api-examples
Page 26 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Files needed to create an Ambari service
• Service definition file (metainfo.xml)
• Service configuration file (myservice-config.xml)
• Service configuration parameters (params.py)
• Service master python script (master.py)
Not always needed:
• Service status configuration file (myservice-env.xml)
• Service status configuration parameters (status_params.py)
Page 27 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Service Definition (metainfo.xml)● name – the internal name of the service (must be unique)● displayName – the public display name of the service● version – the version of the service● component category – is the component a MASTER or CLIENT (one or more)● script – the script that will be executed during service install, start, stop● package – packages to be installed via yum during install
<service> <name>MAVEN</name> <displayName>Maven</displayName> <comment>Apache Maven Project Build Tool</comment> <version>1.0.0</version> <components> <component> <name>MAVEN_CLIENT</name> <displayName>Maven</displayName> <category>CLIENT</category> <commandScript> <script>scripts/client.py</script> <scriptType>PYTHON</scriptType>
Page 28 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Service Configuration (myservice-config.xml)
● name – the internal name of property (must be unique)● value – default value for property● property-type – e.g. whether property is password (optional)● description – tool tip help text for property
<configuration> <property require-input="true"> <name>vnc.password</name> <value></value> <property-type>PASSWORD</property-type> <description>VNC password</description> </property> <property> <name>vnc.geometry</name> <value>1400x860</value> <description>VNC screen geometry/resolution</description> </property>
Page 29 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Service Status Configuration (myservice-env.xml)
● status related parameters (e.g. location of file containing pid of service)● name – the internal name of property (must be unique)● value – default value for property● description – tooltip help text for property
<configuration> <property> <name>pid_file</name> <value>/var/run/ipython-notebook/notebook.pid</value> <description>Process ID File</description> </property> </configuration>
Page 30 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Service Configuration Params (params.py)
Python script to read configuration properties from myservice–config.xml
from resource_management import *
# server configurationsconfig = Script.get_config()
user = config['configurations']['ipython-config']['ipython.user']group = config['configurations']['ipython-config']['ipython.group']
Page 31 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Service Status Config Params (status_params.py)
Python script to read status related configuration properties from myservice–env.xml
from resource_management import *
config = Script.get_config()pid_file = config['configurations']['ipython-env']['pid_file']
Page 32 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Service master python script (master.py) - simpleIf the Ambari service is a just wrapper around a Linux service,
the start/stop/status methods as as easy as…
def start(self, env): Execute("service myservice start")
def status(self, env): Execute("service myservice status")
def stop(self, env): Execute("service myservice stop")
Page 33 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Service master python script (master.py)● def install – called to install service● def start – called to start service● def stop – called to stop the service● def status – called to get status of service● def configure – called to configure service
def install(self, env): #Install packages from metainfo.xml self.install_packages(env) self.configure(env) #code to install service def start(self, env): import params import status_params
#code to start service
def configure(self, env): import params env.set_params(params) #code to re-configure service
def stop(self, env): import status_params env.set_params(status_params) self.configure(env) Execute (format('kill `cat {stack_pidfile}` >/dev/null 2>&1')) Execute (format("rm -f {stack_pidfile}"))
def status(self, env): import status_params env.set_params(status_params) check_process_status(status_params.stack_pidfile)
Page 35 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Demo• Build a MASTER service
• Ntpd
• Sample CLIENT service walk through• Maven
• Use VNC service to setup a developer sandbox • Installs Eclipse, IntelliJ, Maven, Spark
• Real world scenarios• Data Science: iPython service to easily manage platform for data science on Hadoop• Security: OpenLDAP, KDC, NSLCD services to easily setup security with minimal CLI work
Page 37 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Ambari Views Framework
Goal: enable the delivery of custom UI experiences in Ambari Web
Developers can extend the Ambari Web interface• Views expose custom UI features for Hadoop Services
Ambari Admins can entitle Views to Ambari Web users• Entitlements framework for controlling access to Views
Page 38 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Views Framework
Views Framework vs. Views
Page 38
Views
Core to Ambari
Built by Hortonworks, Community,
Partners
Page 39 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Views Framework
Views Framework vs. Views
Page 39
Views
Core to Ambari
Built by Hortonworks, Community,
Partners
Page 40 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Components
• Serve client-side assets (such as HTML + JavaScript)
• Expose server-side resources (such as REST endpoints)
VIEWClient-side
assets(.js, html)
AMBARI WEB
VIEWServer-side resources
(java)
AMBARI SERVER
{rest}
Hadoop and
other systems
Page 41 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Delivery
1. Develop the View (just like you would for a Web App)
2. Package as a View (basically a WAR)
3. Deploy the View into Ambari
4. Ambari Admins create + configuration view instance(s) and give access to users + groups
Develop DeployPackageCreate
Instance(s)
Page 42 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Versions and Instances
• Deploy multiple versions and create multiple instances of a view
• Manage accessibility and usage
Page 43 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Choice of Deployment Model
• For Hadoop Operators:Deploy Views in an Ambari Server that is managing a Hadoop
cluster
• For Data Workers:Run Views in a “standalone” Ambari Server
AmbariServer
HADOOPStore & Process
AmbariServer
Operators manage the cluster, may have Views deployed
Data Workers use the cluster and use a “standalone” Ambari Server for Views
Page 44 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Views REST API
Resource Description
GET /api/v1/views List of available views
GET /api/v1/views/{viewName}/versions List of available view versions
GET /api/v1/views/{viewName}/versions/{version}/instances
List of view instances in a given version
GET /api/v1/views/{viewName}/versions/{version}/instances/{instanceName} Details about a specific view instance.
Page 45 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Deployment
Place view-package.jar on Ambari Server/var/lib/ambari-server/resources/views
Restart Ambari Server to start deployment
ambari-server restart
Deployment registers the view with Ambari and extracts the view/var/lib/ambari-server/resources/views/work
Once complete, the view is available to create instances
Dev Tip: UI assets in extracted view can be modified w/o restart
Page 46 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Views Framework Terminology
Term Description
View Definition Describes the view resources and core view properties such as name, version and any necessary configuration properties.
View Package Packages the view client and server assets (and dependencies)
View Deployment Deploying a view into Ambari
View Version A specific version of a view
View Instance An instantiation of a specific view version
Framework Services View context, instance data, configuration properties and events
Page 47 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Definition (view.xml)name – the internal name of the view (must be unique)label – the public display name of the viewversion – the version of the viewparameter – (optional) metadata about view instance propertiesresource – (optional) the names of the classes required to support a view sub-resourceinstance – (optional) property sets which define static view instances
<view> <name>MYVIEW</name> <label>My View</label> <version>1.0.0</version> <parameter> <name>scope</name> <description>The scope of the instance.</description> </parameter></view>
Dev Tip: Use Static instances for development
Page 48 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Maven build file (pom.xml)
artifactId – determines name of jar
ambari-views 1.7.0.0 dependency ensures jar not required locally<modelVersion>4.0.0</modelVersion> <artifactId> hello-world-view</artifactId> <packaging>jar</packaging> <name>Hello World View</name> <groupId>com.github.hortonworks</groupId> <version>1.0-SNAPSHOT</version>
<dependencies> <dependency> <groupId>org.apache.ambari</groupId> <artifactId> ambari-views</artifactId> <version> 1.7.0.0</version> </dependency>
Page 49 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Package
View Definition: view.xml
Resource / Service classes : JAX-RS annotated
UI classes : html, Servlets deployed as web app (WEB-INF/web.xml).
Application logic : Supporting classes
Dependencies : 3rd party jars or classes├── WEB-INF│ └── web.xml├── org│ └── apache│ └── ambari│ └── view│ └── weather│ ├── CityResource.class│ ├── CityResourceProvider.class│ ├── CityService.class│ └── WeatherServlet.class└── view.xml
view-package.jar
Page 50 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
View Context
• Available to the view server-side through injection.• Provides access to instance configuration.• Provides access to the view and instance attributes.• Provides access to run time information about the current
execution context, including authenticated user principal.public interface ViewContext { public String getUsername(); public String getViewName(); public String getInstanceName(); public Map<String, String> getProperties(); public void putInstanceData(...); public String getInstanceData(...);}
Page 52 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Demo• Build example view with only HTML
• Hello World• iFrame view
• Sample views with HTML and javascript w/o frameworks• REST API explorer • Google map
• Sample views using javascript framework• HDFS Visualizer
• Sample servlet view (uses java) • Hive query view
The End & Questions
This Workshop:● github: https://github.com/abajwa-hw/ambari-workshops● blog: http://hortonworks.com/blog/apache-ambari-technical-workshop/
Apache Ambari: http://ambari.apache.org/ ● wiki: https://cwiki.apache.org/confluence/display/AMBARI/Ambari ● mail lists: https://ambari.apache.org/mail-lists.html ● issues: https://issues.apache.org/jira/browse/AMBARI
Hortonworks:● forum: http://hortonworks.com/community/forums/forum/ambari/● social media: twitter, facebook, slideshare, linkedin, ...
Page 56 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Ambari Views: Where Can I Learn More?
https://cwiki.apache.org/confluence/display/AMBARI/Views
https://github.com/apache/ambari/blob/trunk/ambari-views/docs/index.md
https://github.com/apache/ambari/tree/trunk/ambari-views/examples
https://github.com/apache/ambari/tree/trunk/contrib/views
Page 57 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Ambari Views: Developer resources
https://nodesource.com/blog/your-first-nodejs-package
https://docs.angularjs.org/tutorial/
http://www.jeromecukier.net/blog/2012/09/04/getting-to-hello-world-with-d3/
http://jsbin.com (e.g. http://jsbin.com/hirirazu/3/edit?html,output)
More sample views:
https://github.com/apache/ambari/tree/trunk/ambari-views/examples