amazon emr - aws
Post on 25-Dec-2021
20 Views
Preview:
TRANSCRIPT
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Amazon EMREMR Migration Program Assessment
Colm Pruvot, Sr. Manager, WW Analytics Specialists
Kunal Agarwal, CEO, Unravel Data
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
AOL is a web portal and online service marketed by Oath, a subsidiary of Verizon.
CHALLENGEHousing 2PB of data with 3TB new coming in daily, IT and business teams sought ways to drive faster insights at a reduced costs and bring agility.
SOLUTION
Migrate 100 node on premise Hadoop cluster to
cloud using Amazon S3 and EMR
RESULT
Saved 76% of cost vs. on premise Hadoop
deployment
60% performance gains of business insights
Reduced data storage by 76% equating to $70k/
annually
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
The need for change
Challenges of on-premises/EC2 Hadoop/Spark cluster
1. Fixed cost
2. Storage and compute are tightly coupled.
3. Always on
4. Self-service
5. Static, not-scalable & unused Capacity
6. Outages impact
7. Production upgrade
8. Slow deployment cycle
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Benefits of migrating to Amazon EMR
Easily Run Spark, Hive, Presto, HBase, Flink, and more big data apps on AWS
Best Performance
at Lowest cost
Spark workloads run 2.4x faster
compared to Open Source
50–80% reduction in costs with EC2
Spot and Reserved Instances
Per-second billing for flexibility
Use S3 storage
Process data in S3
securely with high performance
using the EMRFS connector
Scale Compute and Storage
independent of each other
Latest versions
Updated with latest open source
frameworks within 30 days
Support for popular OSS like Flink,
Hudi
Fully managed, no cluster
setup, node provisioning or
cluster tuning
Vertical and Horizontal Auto-
Scaling to suit workload
demands
Easy & Scalable
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Amazon EMR
On-premises
Amazon EMR as a migration destination
Third party
providers
Apache SparkApache Hadoop
Apache HBase Apache Hive
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Migration challenges
TCO estimate accuracy
Determine optimal topology
Prescriptive migration options
EMR Migration Program BenefitsMigration Challenges
How will my big data
applications perform on EMR?
How can I be sure about
EMR cost savings before
migrating?
What is the best EMR
migration strategy?
?
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
EMR Migration Program customer journey
Develop total cost of
ownership
Mobilize Migrate & ModernizeAssess
Conduct EMP
Workshop
Engage partners to
install assessment
tooling
Deliver assessment
insights summary
Leverage MAP
funding
Develop Migration
Readiness Plan
Execute lift & shift
data migration
Execute lift & shift
application migration
Modernize migrated
applications
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
EMR Migration Program assessment phaseEMR Migration Workshop is aimed to provide enablement and jumpstart your migration to the cloud.
Meeting #1
EMR Migration Fundamentals
Delivered by SAs
30-day assessment
Estimate level of effort and cost
Delivered by Unravel, ProServe,
or SI partners
Meeting #2
Migration planning
Delivered by ProServe or SI
partners
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
EMP ISV Partners
+
Highlights Collect cluster usage data for up to 30 days
Analyze data and generate Cloud migration report
Identify failed jobs to migrate
Identify applications that vary in duration of execution
Application complexity and effort estimator
Cluster segmentation
Fully funded through the EMR Migration Program
Assess
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Migration Plan Overview Migrate & Modernize
Determine
optimal topologyPhased migration
summaryInstance mapping Cluster-
workload
segmentation
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
EMR migration strategies Migrate & Modernize
Lift and shift Workload fit
Easiest and least risky
strategy
Price-performance
trade-offs
Cost optimized
More cost-effective
than lift and shift
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Unravel Demo
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Unravel deployment: two lightweight options
On Unravel SaaS
Let data hydrate for 7 days
Log in and see analysisIn customer’s virtual
private cloud
Install a lightweight and non-
intrusive sensor in customer’s
virtual private cloud
Auto-installer used for quick install
Let data hydrate for 7 days
Log in and see analysis
© 2021, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark.
Resources & Links
1. Ask us about migrating to EMR
2. Request the ‘Amazon EMR Migration Guide,’ a 160 page guide of best practices for:
• Migrating data, applications, and catalogs
• Using persistent and transient resources
• Configuring security policies, access controls, and audit logs
• Estimating and minimizing costs, while maximizing value
• Leveraging the AWS Cloud for high availability (HA) and disaster recovery (DR)
• Automating common administrative tasks
3. Ask us about our no cost, onsite EMR Migration Program (EMP) Workshop
delivered by AWS Professional Services or trained SI Partners including Cloudwick,
Provectus, 8K Miles, Mactores, TEKsystems, SoftServe and Infosys.
4. Aol case study (min 30)
Business
AWS EMR Home Page
Big Data Analytics Options on AWS
Re:Invent Videos
AWS Big Data Blog
EMR YouTube
Release Notes - What's New?
Technical
Getting Started on EMR
EMR Management Guide
EMR Release Components
History of Application Versions
Details of each Amazon EMR 5.x version
top related