streamline apache hadoop operations with apache ambari and smartsense
TRANSCRIPT
1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Streamline Apache Hadoop Operations with Apache Ambari and SmartSense
2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Streamlining Apache Hadoop Operations with Ambari & SmartSense
Roni FontaineDirector Product
Marketing
Paul CoddingDirector Product
Management
3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda
Newest features and highlights in Ambari 2.5
How to double Hadoop performance using SmartSense 1.4
Flexible enterprise support model options
4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
What’s New in Ambari 2.5
Service Auto Start (AMBARI-2330)
DB Inconsistency Self-Healing (AMBARI-18990)
Simplified Log Rotation Configuration (AMBARI-16880)
HDFS TopN User & Operation Visualization (AMBARI-19320)
Download All Client Configurations (AMBARI-19275)
Configuration Change Communication (AMBARI-19572)
Add/Remove JournalNodes (AMBARI-7748)
Ignore Host Pre-Check when adding Hosts (AMBARI-18817)
Grafana dashboard for Ambari (AMBARI-17589)
AMS Collector High Availability TP (AMBARI-15901)
Password Credential Store Management (AMBARI-18650)
Post-user-creation script hook (AMBARI-18722)
ZK ACL and SASL Configuration (AMBARI-17324)
Ambari SPNEGO support (AMBARI-18365)
Core Features
Security Features
HDP “Fenton” Support
Port Preserving HS2 Rolling Upgrade (AMBARI-18591)
Log Search TP Update (AMBARI-18821)
Built-In SNMP MIB (AMBARI-19257)
SmartSense Mandatory Install (AMBARI-18346)
RegionServer GC Configuration Optimization (AMBARI-19573)
Core Features Continued
5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Ambari 2.5 + HDP Support Matrix
Added support for HDP 2.6
Deprecated support for HDP 2.4, and removed support for HDP 2.2
HDP 2.6 HDP 2.5 HDP 2.4 HDP 2.3 HDP 2.2
Ambari 2.5
Ambari 2.4
Ambari 2.2.1
Ambari 2.2
deprecated
deprecated deprecated
deprecated
deprecated
6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Ambari 2.5 + OS Support Matrix
Added support for Ubuntu 16 (for HDP 2.6 Only)
Dropped support for Ubuntu 12
RHEL 6 RHEL 7 Debian 7 SLES 11 SLES 12 Ubuntu 12 Ubuntu 14 Ubuntu 16
Ambari 2.5
HDP “Fenton” Only
Ambari 2.4
HDP “Erie” Only
Ambari 2.2
7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
AgendaWhat’s New in Ambari 2.5.0
Feature Highlights: Service Auto Start
8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
How it works
Ambari Agent
Host
Ambari Agent
Host
Ambari Agent
Host
Ambari Server
Ambari DB
LDAPAuthN
Host Restart
• Agent checks in with Server• Server asks agent to check status of
deployed components• Agent reports back state• Server asks agent to start components
not in the desired state
HDP ComponentComponent Stops
9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
AgendaWhat’s New in Ambari 2.5.0
Feature Highlights: Add/Remove JournalNodes
10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
AgendaWhat’s New in Ambari 2.5.0
Feature Highlights: Simplified Log Rotation Configuration
11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
AgendaWhat’s New in Ambari 2.5.0
Feature Highlights: Log Search Improvements
13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
AgendaWhat’s New in Ambari 2.5.0
Feature Highlights: Configuration Change Communication
14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Stack Advisor Behavioral Changes
Add Service
Delete Service
Add Host Component
Move Master
HA Wizards
Delete Host
Add/Remove ZooKeeper Server
Goal: Ensure users are aware of configuration changes related to the activity they are performing.
What we found: Identified multiple locations in which configurations were being changed without notifying the user explicitly
What Changed: New pop-ups and more communication when changes are required
22 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
AgendaWhat’s New in Ambari 2.5.0
Feature Highlights: Download All Client Configurations
26 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
AgendaWhat’s New in Ambari 2.5.0
Feature Highlights: HDFS TopN User & Operation Visualization
27 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
HDFS TopN User & Operation Visualization*
Operations
Users
Users / Operations
*Only works with HDP 2.6
28 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
AgendaWhat’s New in Ambari 2.5.0
Feature Highlights: AMS Collector High Availability TP
29 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
AMS Current state
AMBARICollector API
GRAFANA
HBASE
PHOENIX
SYSTEM
MO
NIT
OR
S
HDPSERVICES SI
NK
S
METRICS COLLECTOR
API consumers
Sinks – HDFS, YARN, HBASE, STORM, KAFKA, FLUME, ACCUMULO, LogSearch, Hive, Nifi
Monitors – System metrics
Grafana
Ambari
31 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
AgendaWhat’s New in Ambari 2.5.0
Feature Highlights: Post User Creation Hook
32 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Post User Creation Hook
Disabled by default
Enabled with two properties– ambari.post.user.creation.hook.enabled=true
– ambari.post.user.creation.hook=/var/lib/ambari-
server/resources/scripts/post-user-creation-hook.sh
Works with manual user creation as well as LDAP sync
Can be run as a one-off command as well
34 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hortonworks Connected Data Platforms and Solutions
Data Services
Hortonworks Solutions
Enterprise DataWarehouse Optimization
Cyber Security andThreat Management
Internet of Thingsand Streaming Analytics
Data CenterHortonworks Data Suite
HDFHDP
HortonworksConnection
CloudHortonworks Data Cloud
AWS HDInsight
Hortonworks Connection
Enablement Subscription
SmartSense™Premier Operational Support
Educational Services
Professional Services
Community Connection
35 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Accelerate Case Resolution
Prevent Issues
Understand Your Cluster
36 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Accelerate Case Resolution
SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture diagnostic information for specific services and hosts into a single “Bundle” that’s automatically uploaded to Hortonworks Support.
Significantly reduces the back-and-forth nature of troubleshooting issues.
A M B A R I
O P SH O R T O N W O R K S
S U P P O R T
S U P P O R TC A S E
S m a r t S e n s eS E R V E R
B U N D L E
G AT E W AY
37 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Accelerate Case Resolution
Prevent Issues
Understand Your Cluster
38 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Prevent Issues
SmartSense analyzes Bundles for configuration issues – recommendations are produced and made available for each cluster in the Hortonworks Support Portal
Recommendations prevent operational issues, and improve performance and overall cluster throughput.
39 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
After Applying only 3 SmartSense Recommendations
They can now run 1200 concurrent jobs
...with only 350 waiting jobs at peak hours
Issue: YARN @ capacity, struggling to add more use cases
Before SmartSense
Could only run 500 jobs concurrently
1100 jobs would be pending waiting for
resources at peak hours
With SmartSense = 2X Throughput Improvement
40 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Accelerate Case Resolution
Prevent Issues
Understand Your Cluster
41 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
“Who’s creating all of these small files in HDFS!?”
“What are my top 10 most active users, and longest running jobs?”
“How much should I charge users for their cluster resource use?”
SmartSense Today – Understand Your Cluster
42 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Understand Your Cluster
Chargeback Reporting
43 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Understand Your Cluster
Chargeback Reporting
HDFS Dashboards
44 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
SmartSense Today – Understand Your Cluster
Chargeback Reporting
HDFS Dashboards
YARN Dashboards
45 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Impact of Hortonworks SmartSense
0200400600800
100012001400
WithoutSmartSense
WithSmartSense
Concurrent Jobs
B U N D L E
2X Throughput Improvement
Address 30% of Issues
Configuration Issues
Avoid 10% of Sev1 Issues
Production Down
Single-Bundle Case Resolution 25% of the Time
SmartSense
Troubleshooting Bundle
46 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hortonworks Connection Ensures Success of Your Big Data Journey
47 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
New Hortonworks Flex Support Subscription
Universal, Usage-based Support Subscription
Cloud & On-Prem
HDCloud
IaaS
On-Prem
Single, flexible, portable support for transition to cloud
New support offering for Spark Data Science, ETL, EDW-Analytics in the Cloud
Performance optimization with Hortonworks SmartSense
48 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Hortonworks Data Cloud for AWS
Pre-tuned for use with AWS
Powered by HDP
Focused on business agility
Prescriptive, ephemeral use cases
• Data Science (Apache Spark & Zeppelin)• Analytics (Apache HIVE)• ETL (Apache HIVE & Spark)• Business Intelligence (OLAP/Druid)
49 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
EASE OF USE: Choose from a set of pre-tuned and pre-configured templates.
50 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Choose from a Set of Prescriptive, Ephemeral Workload Clusters
Data Science
Spark 1.6, 2.1
Business intelligence
(OLAP) Druid TP
Analytics & Reporting with
HIVE 2.2 & LLAP
ETL with Spark
51 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Try it! Hortonworks Data Cloud for AWS Marketplace
How –to Video: https://hortonworks.com/video/hd-cloud-aws/
Try it Now: https://aws.amazon.com/marketplace/pp/B01LXOQBOU
53 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Access to Expertise Plus Freedom, Agility and Flexibility
Expertise for Data Science, ETL and Analytics in HDCloud
Match support costs to usage patterns
Migrate infrastructure on your own terms
Support for Dev, QA & ephemeral production
Built-In Performance Optimization