past, present and future of apache ambari

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache Ambari Past, Present and Future September 2016


Agenda

Ã  Background, Release History and Demo

Ã  Ambari Alerts and Metrics

Ã  Ambari Blueprints

Ã  Security Setup (Kerberos, RBAC)

Ã  Log Search (Technical Preview)

Ã  Automated Cluster Upgrade

Ã  Extensibility: Stacks and Views

Ã  What’s New and Next


What is Apache Ambari?

A completely open source management plaWorm for provisioning, managing, monitoring and securing Apache Hadoop clusters. Apache Ambari takes the guesswork out of operaYng Hadoop.


What Ambari Does

Simplified InstallaYon, ConfiguraYon and Management

Centralized Security Setup

Full Visibility into Cluster Health

Highly Extensible and Customizable

•  Wizard-‐driven and automated cluster provisioning •  Smart Configura>ons and Cluster Recommenda>ons •  Automated Rolling and Express cluster upgrades

•  Reduce complexity to administer security across the plaWorm •  Automate setup Kerberos •  Simplify the configuraYon of Apache Ranger

•  Predefined alerts based on operaYonal best pracYces •  Advanced metrics visualizaYon with Grafana •  Integrated with SmartSense for proacYve issues prevenYon

•  Seamlessly fit into your enterprise environment •  Bring custom Services under management via Ambari Stacks •  Customize the UI with Ambari Views


Ambari Recent Releases

Ambari 2.2.0 Dec 2015

Ambari 2.2.2 Apr 2016

Ambari 2.4.0.0 GA Aug 2016


Ambari Alerts


Full Visibility into Cluster Health

•  Centralized management of Health Alerts

•  Pre-‐defined alerts, configured by default on cluster install


Customizing Alerts

•  Control thresholds, check intervals and response text


Alert No>fica>ons

•  What: Create and manage mul>ple no>fica>on targets

•  Control who gets noYfied when

•  Why: Filter by severity •  Send only certain noYficaYons to certain

targets based on severity

•  How: Control dispatch method •  Support for EMAIL + SNMP


Alert Groups

•  Create and manage groups of alerts •  Group alerts further controls what alerts

are dispatched which noYficaYons

•  Assign group to no>fica>ons •  Only dispatch to interested parYes


Alert Check Counts

Ã  Customize the number of Ymes an alert is checked before dispatching a noYficaYon

Ã  Avoid dispatching an alert noYficaYon (email, snmp) in case of transient issues


Configuring the Check Count

Ã  Set globally for all alerts, or override for a specific alert

Global Sefng

Alert Override


State Change Types

Ã  SOFT state changes do not perform a dispatch

Ã  HARD state changes (to non-‐OK) perform dispatch

Ã  Regardless of change: –  The Ambari Web UI will show the current state (OK/WARN/CRIT) –  The state change is wrihen to ambari-alerts.log

2016-05-31 13:20:52,294 [CRITICAL] [SOFT] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>) 2016-05-31 13:22:52,290 [CRITICAL] [HARD] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>)

Note: check counts are not configurable for AGGREGATE alert types. All state changes are considered HARD.


Example: Check Count = 3

Check 1/3 State: OK

Change: n/a

Check 1/3 State: OK

Change: n/a

Check 1/3 State: CRIT

Change: SOFT

Check 2/3 State: CRIT Change: n/a

Check 3/3 State: CRIT

Change: HARD

Check 1/3 State: OK

Change: HARD

DISPATCH

Check Interval Check Interval Check Interval Check Interval Check Interval

no state change state changes to CRIT

performing mulYple checks

back to OK


Other Items

Ã  Alerts Log (AMBARI-‐10249)

•  Alert state changes are wrihen to /var/log/ambari-server/ambari-alerts.log

Ã  Script-‐based Alert NoYficaYons (AMBARI-‐9919)

•  Define a custom script-‐based noYficaYon dispatcher

•  Executed on alert state changes

2015-07-13 14:58:03,744 [OK] [ZOOKEEPER] [zookeeper_server_process] (ZooKeeper Server Process) TCP OK - 0.000s response on port 21812015-07-13 14:58:03,768 [OK] [HDFS] [datanode_process_percent] (Percent DataNodes Available) affected: [0], total: [1]


Ambari Aler>ng System 1.  User creates or modifies cluster 2.  Ambari reads alert definiYons from

Stack 3.  Ambari sends alert definiYons to

Agents and Agent schedules instance checks

4.  Agents reports alert instance status in the heartbeat

5.  Ambari responds to alert instance status changes and dispatches noYficaYons (if applicable)

Ambari Server

1

2

4

Stack definition alerts.json

5

Ambari Agent(s)

3

emailsnmp


Alert REST APIs

REST Endpoint Descrip>on /api/v1/clusters/:clusterName/alert_definitions The list of alert definiYons for the cluster.

/api/v1/clusters/:clusterName/alerts The list of alert instances for the cluster. Use parYal response syntax to filter.

Example: find all alert instances that are CRITITAL /api/v1/clusters/c1/alerts?Alert/state.in(CRITICAL) Example: find all alert instances for “ZooKeeper Process” alert def /api/v1/clusters/c1/alerts?Alert/definition_name=zookeeper_server_process

/api/v1/clusters/:clusterName/alert_groups The list of alert groups.

/api/v1/clusters/:clusterName/alert_history The list of alert instance status changes.

/api/v1/alert_targets/ The list of configured alert noYficaYon targets for Ambari.


Ambari Metrics


Ambari Metrics

Ã  Built using Hadoop technologies

Ã  Three components: Metrics Collector, Metrics Monitors, Grafana

Local Filesystem or HDFS

HBase ATS

Phoenix


Metrics Collec>on

1.  Metric Monitors send system-‐level metrics to Collector

2.  Sinks send Hadoop-‐level metrics to Collector

3.  Metrics Collector service stores and aggregates metrics

4.  Ambari exposes REST API for metrics retrieval

Ambari Server

Metrics Monitor

Metrics Collector

Host1

Sink(s)

3

Metrics Monitor

Host1

Sink(s) Metrics Monitor

Hosts

Sink(s)

1 2

4


Ambari Metrics + Grafana: Introduced with Ambari 2.2

Ã  Including Grafana as a “NaYve UI” for Ambari Metrics

Ã  Including pre-‐build Dashboards

Ã  Supports configuring for HTTPS

Ã  System Home, Servers

Ã  HDFS Home, NameNodes, DataNodes

Ã  YARN Home, ApplicaYons, Job History Server

Ã  HBase Home, Performance, Misc

Highlights List of Dashboards


Grafana includes pre-‐built dashboards for visualizing the most important cluster metrics.


Ambari Blueprints


Ambari Blueprints

Ã  Two primary goals of Ambari Blueprints:

•  Provide API-‐driven deployments based on self-‐contained cluster descripYon

•  Ability to export a complete cluster descripYon of a running cluster

Ã  Blueprints contain cluster topology and configuraYon informaYon

Ã  Enables interesYng use cases between physical and virtual environments


Ambari Stacks + Blueprints Together

Stack Defini>on Component Layout &

Configura>on BLUEPRINT

BLUEPRINT INSTANTIATE CLUSTER HOSTS


Blueprints API

BLUEPRINT POST /blueprints/my-blueprint

POST /clusters/MyCluster

1

2

CLUSTERCREATIONTEMPLATE


Example: Single-‐Node Cluster

{ "configurations" : [ { ”hdfs-site" : {

"dfs.namenode.name.dir" : ”/hadoop/nn" } } ], "host_groups" : [ { "name" : ”uber-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "SECONDARY_NAMENODE” }, { "name" : "DATANODE” }, { "name" : "HDFS_CLIENT” }, { "name" : "RESOURCEMANAGER” }, { "name" : "NODEMANAGER” }, { "name" : "YARN_CLIENT” }, { "name" : "HISTORYSERVER” }, { "name" : "MAPREDUCE2_CLIENT” } ], "cardinality" : "1" } ], "Blueprints" : { "blueprint_name" : "single-node-hdfs-yarn", "stack_name" : "HDP", "stack_version" : "2.2" }}

{ "blueprint" : "single-node-hdfs-yarn", "host_groups" :[ { "name" : ”uber-host", "hosts" : [ { "fqdn" : "c6401.ambari.apache.org”

} ] } ]}

BLUEPRINT

CLUSTER CREATION TEMPLATE

Descrip>on •  Single-‐node cluster •  Use HDP 2.2 Stack •  HDFS + YARN + MR2 •  Everything on c6401


Blueprints Add Host

•  Add hosts to a cluster based on a host group from a Blueprint

•  Add one or more hosts with a single call

POST /api/v1/clusters/MyCluster/hosts{ "blueprint" : "myblueprint", "host_group" : "workers", "host_name" : "c6403.ambari.apache.org"}

POST /api/v1/clusters/MyCluster/hosts[ { "blueprint" : "myblueprint", "host_group" : "workers", "host_name" : "c6403.ambari.apache.org" }, { "blueprint" : "myblueprint", "host_group" : ”edge", "host_name" : "c6404.ambari.apache.org" }]

hhps://issues.apache.org/jira/browse/AMBARI-‐8458


Blueprints Host Discovery (AMBARI-‐10750)

Ambari

POST /api/v1/clusters/MyCluster/hosts

[ { "blueprint" : "single-node-hdfs-test2", "host_groups" :[ { "host_group" : "slave", "host_count" : 3, "host_predicate" : "Hosts/cpu_count>1” } ] }]


Where Can I Learn More About Blueprints?

Ã  hhps://cwiki.apache.org/confluence/display/AMBARI/Blueprints


Kerberos Setup


Background: Hadoop + Kerberos

Ã  Strongly authenYcaYng and establishing a user’s idenYty is the basis for secure access in Hadoop. Users need to be able to reliably “idenYfy” themselves and then have that idenYty propagated throughout the Hadoop cluster.

Ã  Once this is done, those users can access resources (such as files or directories) or interact with the cluster (like running MapReduce jobs).

Ã  Besides users, Hadoop cluster resources themselves (such as Hosts and Services) need to authenYcate with each other to avoid potenYal malicious systems or daemon’s “posing as” trusted components of the cluster to gain access to data.


Background: Hadoop + Kerberos

Service Component

A

Service Component

B

Hadoop Cluster

KDC

keytab keytab

Service Component

C

keytab

Service Component

D

keytab

Service Component

X

Service Component

X

keytab keytab

Service Component

X

keytab

Service Component

X

keytab

Kerberos is used to secure the

Components in the cluster. Kerberos idenYYes are managed via

“keytabs” on the Component hosts.

Principals for the

cluster are managed in the KDC.


Automated Kerberos Security Setup

Wizard Driven Simplify the experience of enabling and managing Kerberos security

Automated and Manual Op>ons CriYcal for today’s enterprise

Works with Exis>ng Kerberos Infrastructure Such as AcYve Directory and MIT KDC


Automated vs. Manual Kerberos Op>ons

Automated Manual

KDC Infrastructure MIT, AcYve Directory MIT, AcYve Directory, FreeIPA

Requires KDC administraYve credenYals Yes No

InstallaYon of Kerberos clients Yes, opYonal No

Management of Kerberos client krb5.conf Yes, opYonal No

CreaYon of principals Yes No

CreaYon of keytabs Yes No

DistribuYon of keytabs Yes No

Cluster configuraYon Yes Yes


Automated: Principals and Keytabs 1.  User provides KDC Admin Account

credenYals to Ambari 2.  Ambari connects to KDC, creates

principals (Service and Ambari) needed for cluster

3.  Ambari generates keytabs for the principals

4.  Ambari distributes keytabs to Ambari Server and cluster hosts

5.  Ambari discards the KDC Admin Account credenYals

Ambari Server KDC

1 2

4

3

5

Cluster


Automated: Customizable Principal Afributes

•  Customize the directory principal ahributes for your environment

•  ParYcularly useful when using AcYve Directory for Kerberos


Automated: Kerberos Clients

•  OpYon to manage krb5.conf

•  OpYon to install clients

OS Client RHEL/CentOS/OEL krb5-‐workstaYon SLES 11 krb5-‐client Ubuntu 12 krb5-‐user, krb5-‐

config


Automated: Kerberos Opera>ons

Ã  Once enabled, Ambari works directly with the Kerberos infrastructure to automate these common tasks, removing the burden from the operator:

•  Add/Delete Host

•  Add Service

•  Add/Delete Component

•  Regenerate Keytabs

•  Disable Kerberos


Manual: Specify Realm, Client U>li>es Path


Manual: Principals and Keytabs

Ã  Configure IdenYYes

Ã  Download CSV


Role-‐Based Access Control


Role-‐Based Access Control

Ã  Use “roles” for more granular division of control for cluster operaYons

Old Permission New Role Notable Permissions

Operator Cluster Administrator Full operaYonal control, including upgrades. Ambari Admins are implicitly granted this Role.

Cluster Operator Adding and removing hosts.

Service Administrator Manage configuraYons, move components.

Service Operator Service stop and start and service-‐specific operaYons such as HDFS Rebalance.

Read-‐Only Cluster User View cluster service and host informaYon.

Note: Users flagged as “Ambari Admins” are implicitly granted Cluster Administrator permission.


Managing Cluster Roles Assign roles to users or groups

Manage roles in Block or List View

layouts


Managing Cluster Roles View users or groups

Change current role assignment


Log Search

TECH PREVIEW


Log Search

Solr

AMBAR I Log

Search

Search cluster Component Logs from within Ambari!

Goal: When issues arise, be able to quickly find issues across all cluster components ⬢  Capabili>es

–  Rapid Search of all cluster component logs –  Search across Yme ranges, log levels, and for keywords

⬢  Core Technologies: –  Apache Ambari –  Apache Solr –  Apache Ambari Log Search


Log Search Architecture

AMBAR I

L O G F E E D E R

L O G F E E D E R

L O G F E E D E R

L O G F E E D E R

L O G F E E D E R

L O G F E E D E R

WOR K E R NOD E

WORK E R NOD E

WORK E R NOD E

WORK E R NOD E

WORK E R NOD E

WORK E R NOD E

Solr

L OG S E A R CH

U I


Log Search Details

WORK E R NOD E

L O G F E E D E R

Solr

L OG S E A R CH

U I

Solr

Solr

AMBAR I

Java Process MulY-‐output Support Grok

Solr Cloud Local Disk Storage TTL


Automated Cluster Upgrade


Automated Cluster Upgrade Terminology

Rolling Upgrade Orchestrates the cluster upgrade in an order that is meant to preserve cluster operaYon and minimize service impact. The tradeoff is more stringent prerequisites and a longer upgrade Yme.

Express Upgrade Orchestrates the cluster upgrade in an order that will incur cluster downYme. This method has less stringent prerequisites but completes in a faster upgrade Yme.


Automated Upgrade: Rolling or Express Choice

Check Prerequisites Prepare Register +

Install Perform Upgrade

Perform the upgrade. The steps depend on upgrade method: Rolling or Express.

Finalize

Finalize the upgrade, making the target version the current

version.

Perform the preparaYon steps,

which include making backups of cri>cal cluster metadata.

Review the prerequisites to

confirm your cluster configuraYon is ready

for upgrade

Register the sosware repository and install the target version on

the cluster.


Automated Upgrade Choice: Rolling or Express


Automated Upgrade: High-‐Level Process

Register Install Perform Upgrade Finalize


Register New Version

•  Register new version with Ambari

•  Provide the Repository Base URLs for new version



Install Packages

Ã  Click “Install Packages” to start the install of the version across hosts



Upgrade Choice

Ã  Upgrade OpYons dialog gives you a choice of method: Rolling or Express



Upgrade Choice: Rolling or Express

Checks indicate if an opYon is available



Upgrade Choice: Rolling or Express

New opYons to handle failures



Upgrade Tolerance Op>ons

Tolerance Op>on Descrip>on

Skip all Service Check failures Ambari will automa>cally skip any Service Check failures and complete the task without requiring user intervenYon to conYnue. Aser all the Service Checks have run in a task, you will be presented with summary of the failures and an opYon to conYnue the upgrade or pause.

Skip all Slave Component failures Ambari will automa>cally skip any Slave Component failures and complete the task of upgrading Slave components without requiring user intervenYon to conYnue. Aser all Slave Components have been upgraded, you will be presented with a summary of the failures and an opYon to conYnue the upgrade or pause.



Wizard Driven Experience

With verificaYon and validaYon



Rolling Upgrade – Success! Register Install Perform

Upgrade Finalize


Ambari Extensibility


Extensibility Features

•  To add new Services (ISV or otherwise) beyond Stack •  To customize a Stack for customer specific environments

•  To extend and customize the Ambari Web UI •  Add new capabilities, customize existing capabilities

Stacks

Views


Anatomy of Ambari Extension Points

Ambari Server

Ambari Agent Ambari

Agent Ambari Agent

Ambari Web

Stacks Stacks

Stacks

javajs python

Ambari Views Ambari Stacks


Ambari Stacks


Ambari Stacks

Ã  Defines a consistent Stack lifecycle interface that can be extended

Ã  Encapsulates Stack Versions, Services, Components, Dependencies, Cardinality, ConfiguraYons, Commands

Ã  Dynamically add Stack + Service definiYons

AMBARI {rest}

<ambari-web>

Stacks

HDFS YARN MR2

Hive

Pig

Oozie HBase

Storm Falcon


Stack Terminology Term Defini>on Examples

STACK Defines a set of Services, where to obtain the sosware packages and how to manage the lifecycle. HDP-‐2.4, HDP-‐2.5

SERVICE Defines the Components that make-‐up the Service. HDFS, YARN, HIVE, SPARK

COMPONENT The building-‐blocks of a Service, that adhere to a certain lifecycle. NAMENODE, DATANODE, OOZIE_SERVER

CATEGORY The category of Component. MASTER, SLAVE, CLIENT


Stack Mechanics

Ã  Stacks define Services + Repos –  What is in the Stack, and where to get the bits

Ã  Each Service has a definiYon –  What Components are part of the Service

Ã  Each Service has defined lifecycle commands –  start, stop, status, install, configure

Ã  Lifecycle is controlled via command scripts

Ã  Ability to define “custom” commands

AMBARI SERVER

Stack

Command Scripts

Service Defini>ons

AMBARI AGENT/S AMBARI AGENT/S AMBARI AGENT/S

pythonxml

Repos


Stacks Support Inheritance

HDP 2.0 Stack

HDP 2.4 Stack

•  Defines a set of Service definiYons •  Default service configuraYons and command scripts

•  Overrides any Service definiYons, commands and configuraYons

•  Adds new Services specific to this Stack


Where Can I Learn More About Stacks?

Ã  hhps://cwiki.apache.org/confluence/pages/viewpage.acYon?pageId=38571133

Ã  hhps://github.com/apache/ambari/tree/trunk/ambari-‐server/src/main/resources/stacks/HDP/


Guided Configs


Guided Configura>ons

Ã  Take the guesswork out of cluster configuraYon

Ã  IntuiYve layout and grouping of configuraYons

Ã  Smart UI controls to make it easier to set values

Ã  RecommendaYons and cross-‐service dependency checks

Ã  Take the guesswork out of cluster configuraYon

Ã  Driven by Stack definiYon


Layout and Grouping

Subtabs


UI Controls

Recommended Bounds

Recommended Value

Set Recommended

Escape Hatch


Control “Escape Hatch”


Dependency Checking


Driven By Stack Defini>on: Themes


Tabs

Tab1 Tab2


Sec>ons

Sec>on1 Sec>on2


SubSec>ons

SubSec>on1 SubSec>on2 SubSec>on1


UI Controls and Placement

Control + Placement

Control + Placement

Control + Placement Control + Placement

Control + Placement


Ambari Views


Ambari Views Framework

Ã  Developers can extend the Ambari Web interface •  Views expose custom UI features for Hadoop Services

Ã  Ambari Admins can enYtle Views to Ambari Web users •  EnYtlements framework for controlling access to Views


Views Framework

Views Framework vs. Views

Views

Core to Ambari

Built by Hortonworks, Community, Partners


Example Views

Capacity Scheduler “Queue Manager” View

Tez “Jobs” View


View Components

Ã  Serve client-‐side assets (such as HTML + JavaScript)

Ã  Expose server-‐side resources (such as REST endpoints)

VIEW Client-‐side assets

(.js, html)

AMBARI WEB

VIEW Server-‐side resources (java)

AMBARI SERVER

{rest}

Hadoop and other

systems


View Delivery

1.  Develop the View (just like you would for a Web App)

2.  Package as a View (basically a WAR)

3.  Deploy the View into Ambari

4.  Ambari Admins create + configuraYon view instance(s) and give access to users + groups

Develop Deploy Package Create

Instance(s)


Versions and Instances

Ã  Deploy mulYple versions and create mulYple instances of a view

Ã  Manage accessibility and usage


Choice of Deployment Model

Ã  For Hadoop Operators: –  Deploy Views in an Ambari Server that is managing a Hadoop cluster

Ã  For Data Workers: –  Run Views in a “standalone” Ambari Server

Ambari Server

HADOOP Store & Process

Ambari Server

Operators manage the cluster, may have Views deployed

Data Workers use the cluster and use a “standalone” Ambari Server for Views


Where Can I Learn More about Views?

Ã  hhps://cwiki.apache.org/confluence/display/AMBARI/Views

Ã  hhps://github.com/apache/ambari/blob/trunk/ambari-‐views/docs/index.md

Ã  hhps://github.com/apache/ambari/tree/trunk/ambari-‐views/examples

Ã  hhps://github.com/apache/ambari/tree/trunk/contrib/views


What’s New and Next


What’s New in Ambari 2.4

Ã  Alerts: Retry tolerance (AMBARI-‐15686)

Ã  Alerts: Customizable SCRIPT Parameters (AMBARI-‐14898)

Ã  Alerts: New HDFS Alerts (AMBARI-‐14800)

Ã  New Host Page Filtering (AMBARI-‐15210)

Ã  Remove Service (AMBARI-‐14759)

Ã  Support for SLES 12 Technical Preview (AMBARI-‐16007)

Ã  Stability: Database Consistency Checking (AMBARI-‐16258)

Ã  Customizable Ambari Log + PID Dirs (AMBARI-‐15300)

Ã  New Version RegistraYon Experience (AMBARI-‐15724)

Ã  Log Search Technical Preview (AMBARI-‐14927)

Ã  OperaYonal Audit Logging (AMBARI-‐15241)

Ã  Role-‐Based Access Control (AMBARI-‐13977)

Ã  Automated Setup of Ambari Kerberos (AMBARI-‐15561)

Ã  Automated Setup of Ambari Proxy User (AMBARI-‐15561)

Ã  Customizable Host Reg. SSH Port (AMBARI-‐13450)

Core Features Security Features

Ã  View URLs (AMBARI-‐15821), View Refresh (AMBARI-‐15682)

Ã  Inherit Cluster Permissions (AMBARI-‐16177)

Ã  Remote Cluster RegistraYon (AMBARI-‐16274)

Views Framework Features


What’s Next

Ã  Log Search GA (AMBARI-‐14927)

Ã  Service and Component “auto-‐start” (AMBARI-‐2330)

Ã  Ambari Server HA (AcYve-‐Passive + Load Balancer) (AMBARI-‐17126)

Ã  MulY-‐service instances & versions (AMBARI-‐14714)


Learn More About Apache Ambari

Resource Loca>on

Apache Ambari Project Page hhp://ambari.apache.org

Ambari Project Wiki hhps://cwiki.apache.org/confluence/display/AMBARI

Ambari Project JIRA hhps://issues.apache.org/jira/browse/AMBARI

Stacks hhps://cwiki.apache.org/confluence/pages/viewpage.acYon?pageId=38571133

Blueprints hhps://cwiki.apache.org/confluence/display/AMBARI/Blueprints

Views hhps://cwiki.apache.org/confluence/display/AMBARI/Views


Thank You

past, present and future of apache ambari

Data & Analytics