ironfan: build a hadoop cluster in 20 minutes
DESCRIPTION
The slides from our talk at ChefConf 2012TRANSCRIPT
Ironfan: Build a Hadoop Cluster in 20 Minutes
Dhruv Bansal@dhruvbansal
Nathaniel Eliot@temujin9
Saturday, May 19, 2012
Saturday, May 19, 2012
Outline
Saturday, May 19, 2012
Outline
• Why we love Chef• Why we don’t• Why we wrote Ironfan
Why Ironfan?
Saturday, May 19, 2012
Outline
• Why we love Chef• Why we don’t• Why we wrote Ironfan
Why Ironfan?
• Build a Hadoop cluster• Run a Hadoop job• Put that cluster away
Ironfan in Action
Saturday, May 19, 2012
Outline
• Why we love Chef• Why we don’t• Why we wrote Ironfan
Why Ironfan?
• Build a Hadoop cluster• Run a Hadoop job• Put that cluster away
Ironfan in Action
• Better environments• Cloud agnosticism• Continuous integration• Stacks
Ironfan roadmap
Saturday, May 19, 2012
Why we love Chef
Infrastructure as code:
Version control
Shareable Recapitulable
Testable
Saturday, May 19, 2012
MySQL Nginx SOLR
My Application
Modular, composable units:
Why we love Chef
Saturday, May 19, 2012
Why we love Chef
Saturday, May 19, 2012
Why we don’t love Chef
Anything is possible
Nothing is simple
And there’s much repetition (not DRY)
Saturday, May 19, 2012
Why we don’t love Chef
Hard Coded Dynamic
Too much is hard-coded at development/upload time!
Write Upload
Development Machine
Compile Converge
Managed Server
Saturday, May 19, 2012
Why we don’t love Chef
How do we make @server_ips dynamic?
Saturday, May 19, 2012
We wrote Ironfan to
Enter Ironfan
Saturday, May 19, 2012
We wrote Ironfan to
Enter Ironfan
• Simplify, unify, and standardize our usage of the Chef toolset
Saturday, May 19, 2012
We wrote Ironfan to
Enter Ironfan
• Simplify, unify, and standardize our usage of the Chef toolset
• Build further abstractions on top of Chef
Saturday, May 19, 2012
We wrote Ironfan to
Enter Ironfan
• Simplify, unify, and standardize our usage of the Chef toolset
• Build further abstractions on top of Chef
• Give us superpowers that Chef doesn’t have yet
Saturday, May 19, 2012
We wrote Ironfan to
Enter Ironfan
• Simplify, unify, and standardize our usage of the Chef toolset
• Build further abstractions on top of Chef
• Give us superpowers that Chef doesn’t have yet
http://github.com/infochimps-labs/ironfan
Saturday, May 19, 2012
What does Ironfan do?
Saturday, May 19, 2012
What does Ironfan do?
ChefSaturday, May 19, 2012
What does Ironfan do?
Simple helpers in the silverware cookbook abstract common Chef patterns and keep things DRY
Ironfan
ChefSaturday, May 19, 2012
What does Ironfan do?
Saturday, May 19, 2012
What does Ironfan do?
Dynamic service discovery:
Saturday, May 19, 2012
What does Ironfan do?
Dynamic service discovery:
Saturday, May 19, 2012
What does Ironfan do?
Saturday, May 19, 2012
What does Ironfan do?
A simple DSL for defining clusters of machines
Saturday, May 19, 2012
DEMO TIME!
Saturday, May 19, 2012
Thanks for coming!
Learn more about what Ironfan and Infochimps can do for you.
Come work at a fast-growing, cutting-edge technology company based in Austin, TX.
http://github.com/infochimps-labs/ironfan
http://github.com/infochimps-labs/wukong
Saturday, May 19, 2012