topic 8: enhancements and alternative architectures

8: Enhancements and Alternative Architectures

Zubair Nabi

[email protected]

April 19, 2013

Zubair Nabi 8: Enhancements and Alternative Architectures April 19, 2013 1 / 45

Outline

1 Major shortcomings

2 Pig Latin

3 Dryad

4 CIEL

5 Naiad

Outline

2 Pig Latin

3 Dryad

4 CIEL

5 Naiad

Focusing on some

Low-level programming interface

Iterative and recursive applications

Incremental computations

Focusing on some

Outline

2 Pig Latin

3 Dryad

4 CIEL

5 Naiad

Introduction

MapReduce is too low-level and rigid and leads to lots of custom usercode

Pig Latin is a declarative language atop MapReduce designed byYahoo!

I Finds the sweet spot between the declarative style of SQL and thelow-level interface of MapReduce

The Pig system compiles Pig Latin queries into physical plans that areexecuted atop Hadoop

Introduction

MapReduce is too low-level and rigid and leads to lots of custom usercodePig Latin is a declarative language atop MapReduce designed byYahoo!

Introduction

SQL query to find average pagerank for each large categoryof URLs

1 SELECT category, AVG(pagerank)

2 FROM urls WHERE pagerank > 0.2

3 GROUP BY category HAVING COUNT(∗) > 10^6

Equivalent Pig query

1 good_urls = FILTER urls BY pagerank > 0.2;

2 groups = GROUP good_urls BY category;

3 big_groups = FILTER groups BY COUNT(good_urls)>10^6;

4 output = FOREACH big_groups GENERATE

5 category , AVG(good_urls.pagerank);

Pig Interface

A Pig Latin program is a sequence of steps, reminiscent of traditionalprogramming languages

I In contrast, SQL consists of declarative constraints that collectivelydefine the result

Each step carries out a single data transformation

A Pig Latin program is similar to specifying a query execution or adataflow graph

Due to this dataflow model, it is easier for programmers to understandand control how their data processing task is executed

Pig Interface

Features

Support for a fully nested data model with complex data types

Extensive support for user-defined functions

Ability to operate over plain, schema-less input files

Open-source Apache project

Features

Interoperability

Queries can be performed atop raw data dumps directly

The user needs to provide a function to parse the content of the file intotuples

Similarly, the user also needs to provide a function to convert tuplesinto a byte sequence

Datasets can be laid across diverse data storage sources andapplications

Interoperability

UDFs as first-class citizens

A significant part of large-scale data analysis relies on customprocessing

For instance, the user may be interested in figuring out whether aparticular website is spam

All aspects of processing in Pig Latin including grouping, filtering,joining, and per-tuple processing can be customized via UDFs

UDFs take non-atomic parameters as input and produce non-atomicvalues as output

UDFs are defined in Java

1 groups = GROUP urls BY category;

2 output = FOREACH groups GENERATE

3 category , top10(urls);

Data Model

Pig has four data types:

1 Atom: A single atomic value such as a string or an integer

2 Tuple: A sequence of values, each with possibly a different data type

3 Bag: A collection of tuples

4 Map: A collection of data types, each with an associated key

Data Model

Commands

LOAD: Load and deserialize an input file

FOREACH: Process each tuple of a dataset

FILTER: Filter a dataset based on some condition or UDF

COGROUP: Group together tuples which are related in some way fromone or more datasets

STORE: Materialize the output of a Pig Latin expression to a file

Commands

Outline

2 Pig Latin

3 Dryad

4 CIEL

5 Naiad

Introduction

MapReduce is strictly two stage, single input set and single output set

Awkward architecture to perform multi-stage computation

Introduction

MapReduce is strictly two stage, single input set and single output set

Awkward architecture to perform multi-stage computation

MapReduce: Architecture

Dryad allows computations that can form a Directed Acyclic Graph(DAG)

Each vertice within the graph is a computation while an edge depictscommunication channels

Each computation can take in multiple files as input and producemultiple outputs

Developed by Microsoft Research

Dryad: Architecture

Dryad: Architecture (2)

Files, TCP, FIFO, Networkjob schedule

data plane

control plane

NS PD PDPD

Job manager cluster

Dryad: Job

Job = Directed Acyclic Graph

Processingvertices Channels

(file, pipe, shared memory)

Inputs

Outputs

Channel types and job inputs and outputs

Channel types: File, TCP pipe, Shared-memory FIFO

Encapsulation: Convert a graph into a single vertex, and run withinsame process

Job inputs and outputs: Can be logically concatenated

Vertices

Programming in C++/C#

Runtime library sets up and executes vertices

Map and Reduce classes

Process wrapper: To support legacy executables

Supports event-based programming

Vertices

Dryad: Example

U UN N

select distinct p.objIDfrom photoObjAll pjoin neighbors n – call this join “X”on p.objID = n.objIDand n.objID < n.neighborObjIDand p.mode = 1join photoObjAll l – call this join “Y”on l.objid = n.neighborObjIDand l.mode = 1and abs((p.u-p.g)-(l.u-l.g))<0.05and abs((p.g-p.r)-(l.g-l.r))<0.05and abs((p.r-p.i)-(l.r-l.i))<0.05and abs((p.i-p.z)-(l.i-l.z))<0.05

Operations

Create Vertices using C++ base class

Add edges

Merge two graphs

Operations

Add edges

Merge two graphs

Operations

Add edges

Merge two graphs

Operations (2)

Job Execution

Vertex can specify “hard constraint” or “preference”

Job manager runs greedy scheduling algorithm: Only job running

Simple graph visualizer: State of each vertex and channel for smalljobs

Web-based interface: Regularly-updated statistics

Fault Tolerance: Vertices deterministic. Just re-schedule

Speculative execution within stages

Job Execution

Run-time Graph Refinement

Aggregation tree: Distributed combiner

I Associative, and commutative computation

Run-time Graph Refinement

Aggregation tree: Distributed combinerI Associative, and commutative computation

Outline

2 Pig Latin

3 Dryad

4 CIEL

5 Naiad

Introduction

MapReduce and Dryad are not amenable to iterative and recursiveapplications

Most machine learning and data mining applications are iterative innatureThese applications require a data-dependent control flow

I The ability to spawn new tasks on the fly based on previouscomputations

Introduction

Most machine learning and data mining applications are iterative innature

These applications require a data-dependent control flowI The ability to spawn new tasks on the fly based on previous

computations

Introduction

Most machine learning and data mining applications are iterative innatureThese applications require a data-dependent control flow

I The ability to spawn new tasks on the fly based on previouscomputations

1 Data-centric execution engine from Cambridge: the goal of a CIEL jobis to produce one or more output objects

2 A reference can be obtained to an object without materializing its fullcontents, reminiscent of C pointers

I If objects do not have their full contents, their references are futurereferences; otherwise they are concrete references

3 A job makes progress by executing tasks

1 Each task has dependencies on one of more objects via referencesand it starts executing once all of its references become concrete

2 The purpose of each task is to produce objects1 A task can publish one or more objects by creating a concrete

reference for them2 A task can also spawn new tasks and delegate the creation of output to

3 The dynamic task graph stores the relation between tasks andobjects

2 The purpose of each task is to produce objects

1 A task can publish one or more objects by creating a concretereference for them

2 A task can also spawn new tasks and delegate the creation of output tothem

reference for them

2 A task can also spawn new tasks and delegate the creation of output tothem

CIEL: Architecture

CIEL: Dynamic Task Graph

Executors

1 CIEL maintains a decoupling between tasks and the underlyingframework through the concept of executors

2 Each programming language has a corresponding executor

3 As a result, a task can be written in any programming language, suchas Java, Python, shell-script, etc. as well as the indigenous Skywriting

Executors

Skywriting

1 Scripting language for expressing task-level parallelism atop CIEL

2 Contains data-dependent control flow constructs such as while loopsand recursive functions

3 Ability to spawn new tasks in the middle of execution

Skywriting

Skywriting constructs

1 ref(url): Returns a reference to the object located (both local andremote) at url

2 spawn(f, [arg, ...]): Spawns a task to evaluate f

3 exec(executor, args, n): Runs the given executor toevaluate args

4 spawn_exec(executor, args, n): Spawns a new task to runthe given executor

5 *-: De-references the given reference

Example: Skywriting1 function process_chunk(chunk, prev_result) {

2 return spawn_exec(...);

4 function is_converged(curr_result , prev_result) {

5 return spawn_exec(...)[0];

7 input_data = [ref("ciel://host137/chunk0"),

8 ref("ciel://host223/chunk1"), ...];

9 curr = ...; // Initial guess at the result.

10 do {

11 prev = curr;

12 curr = [];

13 for (chunk in input_data) {

14 curr += process_chunk(chunk, prev);

16 } while (!∗is_converged(curr, prev));17 return curr;

Outline

2 Pig Latin

3 Dryad

4 CIEL

5 Naiad

Introduction

A class of applications requires support for both iterative andincremental computation

For instance, to maintain in real-time the strongly connectedcomponent structure in the graph induced by Twitter mentions

Currently, MapReduce itself has no support for either iterative orincremental computation

Introduction

Data-intensive computing framework from Microsoft Research thatsupports both incremental and iterative computation by leveragingdifferential computation

Differential computation adds two novelty factors to the framework:1 The state of the computation varies according to a partially ordered set

of versions rather than a total ordering2 The set of updates required to reconstruct the state at any version is

retained in an indexed data-structure

The state and updates to that state are associated with amulti-dimensional logical timestamp (called a version)

Data-intensive computing framework from Microsoft Research thatsupports both incremental and iterative computation by leveragingdifferential computationDifferential computation adds two novelty factors to the framework:

1 The state of the computation varies according to a partially ordered setof versions rather than a total ordering

2 The set of updates required to reconstruct the state at any version isretained in an indexed data-structure

Programming environment

Declarative query language based on the .NET Language IntegratedQuery (LINQ)

LINQ extends C# with declarative operators, such as Select,Where, Join, and GroupBy, among othersNaiad adds two more operators:

1 FixedPoint that takes a source collection and a function thatmutates the collection to another collection of the same type to achievefixed-point convergence

2 PrioritizedFP additionally takes a priority function to apply toevery record in the source collection

LINQ extends C# with declarative operators, such as Select,Where, Join, and GroupBy, among others

Naiad adds two more operators:1 FixedPoint that takes a source collection and a function that

mutates the collection to another collection of the same type to achievefixed-point convergence

Runtime

The Naiad runtime transforms declarative queries to a dataflow graph

The user program can insert differences into the input collections andregister callbacks to be invoked when differences are received at theoutput collection

The runtime transparently distributes the execution of the data flowgraph (similar to Dryad) across several cores and nodes

Runtime

References

1 Christopher Olston, Benjamin Reed, Utkarsh Srivastava, Ravi Kumar,and Andrew Tomkins. 2008. Pig latin: a not-so-foreign language fordata processing. In Proceedings of the 2008 ACM SIGMODinternational conference on Management of data (SIGMOD ’08). ACM,New York, NY, USA, 1099-1110.

2 Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and DennisFetterly. 2007. Dryad: distributed data-parallel programs fromsequential building blocks. In Proceedings of the 2nd ACMSIGOPS/EuroSys European Conference on Computer Systems 2007(EuroSys ’07). ACM, New York, NY, USA, 59-72.

3 Derek G. Murray, Malte Schwarzkopf, Christopher Smowton, StevenSmith, Anil Madhavapeddy, and Steven Hand. 2011. CIEL: a universalexecution engine for distributed data-flow computing. In Proceedings ofthe 8th USENIX conference on Networked systems design andimplementation (NSDI’11). USENIX Association, Berkeley, CA, USA.

References (2)

4 Frank McSherry, Derek G. Murray, Rebecca Isaacs, and Michael Isard.Differential dataflow. 2013. In Conference on Innovative Data SystemsResearch (CIDR), 2013.

topic 8: enhancements and alternative architectures

Technology