mortar: hadoop-as-a-service + open source framework | aws re: invent public …

42

Upload: mortardata

Post on 28-May-2015

1.095 views

Category:

Technology


1 download

DESCRIPTION

Presentation from AWS re: Invent, where Mortar announced it's public availability on stage. Mortar facilitates Hadoop data development in two parts: an open source data development framework (like Rails for Hadoop) and a PaaS (like Heroku for Hadoop). The open source framework allows for sharing/repeating/maintaining code history, automated testing, and one-button deploy. The PaaS provides high availability execution, full job history, and helps users identify problems.

TRANSCRIPT

Page 1: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
Page 2: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

Open-source softwaredevelopment frameworks

Page 3: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

Open-source software dev frameworks

(and many more...)

Page 4: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

• Avoid repeat work• Remain 100% customizable• Escape vendor lock-in

Every platform needsopen-source, code-based dev frameworks

Page 5: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

But for data?Nothing.

Page 6: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

Data work today• Proprietary / GUI / No framework• Not reusable• Little collaboration• Too many errors, too slow to fix

Page 7: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

an open source dev framework for data

Page 8: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

“Mortar takes something complex and makes it simple and intuitive.”

—Jon Coveney, Twitter

Page 9: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

What big data system should I use? Hadoop, HPCC, Disco, Storm… What Hadoop distro should I use?How much will this cost? Should I do it some other way?How long is this going to take to learn? Should I do it some other way?How many machines should I buy?Can I run in the cloud?How should I interact with it? Pig, Hive, Cascading, Scalding, CascalogCan I use libraries I need?How do I install everything?How can my team share code?How can I reuse code?Is my workload typical?How can I safely deploy?How do I know if what I've written is correct?Are there any libraries I could start with?Can I connect to my key-value store?Can I do machine learning in Hadoop?Is it secure?What if I need help?

Weeks

Page 10: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

More Weeks

Page 11: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

Even More Weeks

Page 12: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

“Our focus in designing Pig has always been to make Hadoop easy...

Mortar's approach is right on—they extend our quick start and ease of use focuses with pre-built Hadoop clusters, clear examples, code organization templates, and github for social sharing of the code.”

OPEN SOURCED PIG—Alan Gates, Hortonworks co-founder

Page 13: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

Mortar is for analyzing lots of data in AWS.

Can Mortar help you?

Page 14: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

Mortar serves companies of all sizes from any industry.

Who is Mortar for?

Page 15: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

“...Mortar fits right in with our vision of the future... With this exciting launch, MongoDB users can now also seamlessly use Mortar.”

—Dwight MerrimanFOUNDER OF 10GEN (MAKER OF MONGODB), DOUBLECLICK (ACQ. GOOGLE), SHOPWIKI, BUSINESS INSIDER, GILT GROUPE

Page 16: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

By and for engineers and data scientists

Page 17: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

> gem install mortar

Page 18: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

> mortar new my_project

Page 19: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
Page 20: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

> git clone your_project

Page 21: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

> mortar run your_project

Page 22: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
Page 23: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
Page 24: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

Pig is easy to learn(and we’ve made it easier)

Page 25: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
Page 26: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
Page 27: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

Illustrate is awesome• Find your mistakes• Understand code before collaborating• Automated tests: a way to test every condition

Page 28: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

Hadoop & Python are powerful data science tools

Page 29: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

...but they haven’t worked together before.

Page 30: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

Now you can use Hadoop & real Python on Mortar

Page 31: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
Page 32: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
Page 33: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

• Installed Mortar• Made a new project• Cloned a project• Ran the project• Illustrated project• Use Python and other libraries on Hadoop

What you just saw

Page 34: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

- Web Projects: zero install, in the browser

2 options for using Mortar:

- Git Projects: modularity, testability, code sharing, local dev, and revision control.

Page 35: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
Page 36: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

• Use your browser• Minutes to connect data• Productive in one hour

One-hour challenge

Page 37: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

As a good citizen, Mortar has a rich API

How does Mortar fit with other

Page 38: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

Full speed, directly on Hadoop

How about speed?

Page 39: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

• Easy start• Keeps you productive• Collaborate with data• No lock-in• Easy to budget

Mortar revolutionizes your data pipeline.

Page 40: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

• Free | Service use unlimited | 10 node-hours• Pay as you Go | $0.89/node-hour | support• Enterprise | $3,000/month | $0.69/node-hour | live

support

Tiers

Page 41: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …
Page 42: Mortar: Hadoop-as-a-Service + Open Source Framework | AWS re: Invent public …

mortardata.com / @mortardata