dmapply: A functional primitive to express distributedmachine learning algorithms in R

Edward Ma Vishrut Gupta Meichun Hsu Indrajit Roy+

HPE Vertica, +Hewlett Packard Labsema,vishrut.gupta,meichun.hsu,indrajitr@hpe.com

ABSTRACTDue to R’s popularity as a data-mining tool, many dis-tributed systems expose an R-based API to users who needto build a distributed application in R. As a result, datascientists have to learn to use different interfaces such asRHadoop, SparkR, Revolution R’s ScaleR, and HPE’s Dis-tributed R. Unfortunately, these interfaces are custom, non-standard, and difficult to learn. Not surprisingly, R appli-cations written in one framework do not work in another,and each backend infrastructure has spent redundant effortin implementing distributed machine learning algorithms.

Working with the members of R-core, we have createdddR (Distributed Data structures in R), a unified systemthat works across different distributed frameworks. In ddR,we introduce a novel programming primitive called dmapply

that executes functions on distributed data structures. Thedmapply primitive encapsulates different computation pat-terns: from function and data broadcast to pair-wise com-munication. We show that dmapply is powerful enough toexpress algorithms that fit the statistical query model, whichincludes many popular machine learning algorithms, as wellas applications written in MapReduce. We have integratedddR with many backends, such as R’s single-node parallelframework, multi-node SNOW framework, Spark, and HPEDistributed R, with few or no modifications to any of thesesystems. We have also implemented multiple machine learn-ing algorithms which are not only portable across differentdistributed systems, but also have performance comparableto the “native” implementations on the backends. We be-lieve that ddR will standardize distributed computing in R,just like the SQL interface has standardized how relationaldata is manipulated.

1. INTRODUCTIONR is one of the top choices for statisticians and data scien-

tists [26]. While R is commonly used as a desktop tool, datascience teams in most enterprises install RHadoop, SparkR,

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. To view a copyof this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. Forany use beyond those covered by this license, obtain permission by emailinginfo@vldb.org.Proceedings of the VLDB Endowment, Vol. 9, No. 13Copyright 2016 VLDB Endowment 2150-8097/16/09.

Distributed R, and other R-based interfaces to run analyt-ics on the corresponding backend distributed system, suchas Hadoop MapReduce and Spark [4, 5, 30, 34]. By usingR, data scientists can continue to program in the familiar Rlanguage, even on massive data. Unfortunately, each back-end exposes a custom R API, which leads to non-portableapplications. Many interfaces also expose low-level detailsspecific to a particular backend, making them difficult tolearn. For example, SparkR, which is an R interface for theApache Spark engine, exposes dozens of Spark’s functionsthat can be non-intuitive to data scientists, and a programwritten in SparkR’s API will not run on other frameworks.

The core reason for this disarray is the lack of any stan-dardized way to perform distributed computing in R. WhileSQL semantics have unified how relational data is manipu-lated, there has been little effort in providing a similar stan-dard for advanced analytics. Even though the R communityhas contributed more than 6000 algorithms packages, thereare hardly any parallel or distributed versions of these algo-rithms. A simple, standard interface for distributed comput-ing in R can potentially kickstart contributions on R-basedparallel machine learning applications without the fear ofvendor lock-in.

There are two key challenges in providing a standardizedsystem for advanced analytics. First, the system should im-plement an interface that is not only easy to use by datascientists but also flexible enough to express many advancedanalytics tasks. Second, generic interfaces typically havehigh overheads due to their flexible nature. One needs toensure that applications expressed in this system, such asmachine learning algorithms, have good performance.

In this paper we describe ddR (Distributed Data struc-tures in R), a system that unifies distributed computing inR. We created ddR in collaboration with members of R-core, those who maintain and release the R language. ddRdefines the following three distributed versions of R datastructures, providing an intuitive way to partition and storemassive datasets: dlist, darray, and dframe, which arethe distributed equivalents of the fundamental R contain-ers: list, matrix, and data frame, respectively. These datastructures store different types and shapes of data such asstructured, unstructured, sparse and dense data.

To manipulate these distributed data structures, ddRintroduces dmapply, a single programming primitive withfunctional semantics. We chose functional semantics be-cause it ensures there are no side-effects, thus making it easyto parallelize a computation. The dmapply operator is mul-tivariate, which means programmers can use it to operate

1293

on multiple data structures at the same time, e.g., by usingdmapply(A,B,C..). Much of the power of dmapply comesfrom the fact that it can express different kinds of data com-putation and communication patterns that are common inanalytics tasks. For example, programmers can easily ex-press embarrassingly parallel computations, where a func-tion operates on each partition of data and no data move-ment is required. Programmers can also express cases wherecertain data structures, or parts of them, are broadcast toall computations. In fact, users can even express patterns inwhich computation on a worker node receives data from anysubset of workers, and thus implement different kinds of ag-gregations. In section 4.2 we confirm the expressive powerof dmapply by showing that it can implement algorithmsthat fit the statistical query model [16], which includes mostof the popular machine learning algorithms. We also showthat dmapply can express applications written in the popularMapReduce paradigm [10].

A key advantage of ddR is that it can be integrated withbackends by writing a simple driver code, typically with-out making any modification to the underlying system. Wehave integrated ddR with several popular data processingengines, such as R’s parallel package (which leverages mul-tiple cores on a single server), a socket-based backend calledSNOW, HPE’s open-sourced Distributed R [24, 30], andSpark [34]. ddR includes a growing list of common R opera-tors such as distributed sum, colMeans, and others, that havebeen implemented using dmapply. These operators ensurethat any backend that integrates with ddR gets these oper-ators for free. For example, programmers who use SparkR’sdataframes or HPE Distributed R’s dataframes can call dis-tributed colMeans (average of each individual column) evenif the respective backend does not have a native implemen-tation of that function. These helper operators ensure thatR programmers can continue to use the same standard func-tions that they expect from single-threaded R. These oper-ators also provide an incentive to different backend vendorsto integrate with ddR and help the standardization effort.

We have implemented a number of reference machine learn-ing algorithms in ddR that perform clustering, classifica-tion, and regression. Our empirical evaluation indicatesthat these algorithms have very good performance. On asingle server, regression with the parallel backend on 12Mrows and 50 columns converges in 20 seconds, while cluster-ing 1.2M 500-dimensional points takes only 7 seconds per-iteration. Notably, these ddR algorithms have performancesimilar to, and sometimes even better than, machine learn-ing products such as H2O [3] and Spark’s MLlib, whosealgorithms are custom written and not portable. In a dis-tributed setting, such as with HPE Distributed R as thebackend, ddR algorithms such as K-means and regressionshow near linear scalability and can easily process hundredsof gigabytes of data. When used with SparkR or HPE Dis-tributed R, ddR algorithms have similar performance asSpark’s native MLlib algorithms or Distributed R’s algo-rithms. These results show that users will benefit from theease of a standardized interface to express applications, andyet obtain performance at par with custom or non-portableimplementations.

The core components of ddR are available as an R pack-age from R’s public package repository called CRAN [20]. Inthe last four months, the ddR package has been downloadedmore than 2,500 times [25].

Backend drivers

Distributed data structures(darray, dframe, dlist) dmapply

Parallel(multi‐core)

HPE Distributed R

MPI(R’s SNOW)

Apache Spark

Applications + Machine learning algorithms

New Unmodified

Common distributed operators

SQL

backends

ddRpackage

Figure 1: High level architecture. Shaded regionsshow new components.

The contributions of this paper are:

• Design of a standard interface, ddR, for distributedcomputing in R. We introduce a simple yet powerfulfunctional primitive, dmapply, and show that it is ex-pressive enough to implement many machine learningalgorithms.

• Implementation of ddR and its integration with mul-tiple backends including R’s parallel and SNOW back-ends, HPE Distributed R, and Spark. The ddR pack-age makes distributed systems easy to use for data sci-entists.

• Implementation of multiple machine learning applica-tions in ddR, which run on different backends, andhave good performance. We demonstrate that theseapplications have performance comparable to nativeversions of these algorithms written in their respectivebackends.

While our focus in this paper is to improve R, we be-lieve that the concepts in ddR, such as dmapply, are gen-eral enough that they may help other analytics languages,such as Python, that are popular wrapper interfaces fordistributed systems. Following the paradigm of dplyr [32]where SQL-like functions in R are transparently pushed intoa database, a standard primitive for distributed computingin R could also lead to more intelligent integration betweenan R application and a parallel database system.

2. ARCHITECTURAL OVERVIEWThe ddR project started as a collaboration between R-

core, i.e., maintainers of the R language, HPE, and the open-source Distributed R community, which has benefited fromcross-industry feedback through workshops and the R Con-sortium. The goal of the project is to extend R, streamlineintegration with popular advanced analytics systems, andcreate portable distributed application and algorithms in R.In this section we provide an overview of ddR and how itinteracts with different backends.

Figure 1 provides a high level view of ddR. The ddR pro-gramming model exposes distributed data structures, thedmapply function to operate on these data structures, and aset of commonly used operators. Programmers use theseAPIs to create distributed applications, such as machinelearning algorithms, and utility libraries, such as parallel

1294

data loaders. One beneficial outcome of implementing ap-plications in ddR is that users can choose to switch back-ends on which their applications execute. For example, acustomer may initially use a ddR clustering algorithm ontheir laptop using Rs parallel package, which provides ef-ficient multi-core processing, and then run the exact samecode on HPE Distributed R or SparkR, in production, andon massive data.

We have implemented the ddR programming model ontop of multiple backends. To integrate with a backend, thekey effort is to implement the three distributed data struc-tures, the dmapply function, and a couple of basic operatorsthat provide functionality such as moving all contents of adata structure from workers to the master node. This codelives in the “backend driver” (see Figure 1), a piece of con-necting logic that binds the frontend interface to the backendby translating invocations to the dmapply primitive to thebackend’s native API.

The amount of effort required to integrate a backend islow if the backend already has an implementation of thecore data structures. For example, Distributed R alreadyimplements the three types of distributed data structuresand we only had to implement the dmapply function. Sim-ilarly, Spark provides data frames and we were able to useits RDD API to implement dmapply. However, for ddR’sdefault multi-core engine, which uses R’s parallel package,we had to implement everything from scratch since standardR does not have a notion of distributed data structures.

As shown in Figure 1, programmers who use ddR to ex-press their R applications can continue to also use APIs,such as the SQL interface, that the underlying backend ex-poses. As an example, on the Spark backend users willstill be able to call SQL operators on RDDs, while on theparallel backend users will be able to invoke the dplyr

package that implements a subset of SQL operators on Robjects [32]. Section 5 provides more details about differentcomponents.Example. Figure 2 shows how programmers can invokeddR’s distributed clustering algorithm. Line 1 imports theddR package, while line 2 imports a distributed K-meanslibrary written using the ddR API. Line 4 determines thebackend on which the functions will be dispatched. In thisexample the backend used in the default parallel back-end, which is single-node but can use multiple cores. Inline 6, the input data is generated in parallel by calling auser-written function genData using dmapply. The input isreturned as a distributed array with as many partitions asthe number of cores in the server. Finally, in line 8, theddR version of the K-means algorithm is invoked to clusterthe input data in parallel. The key advantage of this ddRprogram is that the same code will run on a different back-end, such as HPE Distributed R, if line 4 is simply changedto useBackend(distributedR).

3. DISTRIBUTED DATA STRUCTURESData scientists use a variety of data structures and oper-

ators to express their analytics workflows. With ddR, ourgoal is to ensure that data scientists can continue to usesimilar interfaces for distributed computing in R. The ddRprogramming model exposes three types of distributed datastructures: arrays, data frames, and lists. Distributed ar-rays (darray) store data of a single type. Arrays can repre-sent vectors and matrices. While it is common to use dense

1 l i b r a r y (ddR)2 l i b r a r y ( kmeans . ddR)3 #Spe c i f y which backend to use4 useBackend ( p a r a l l e l )5 #Popu la te a d i s t r i b u t e d a r r a y w i th s y n t h e t i c data6 f e a t u r e s <− dmapply ( genData , i d = 1 : ncores ,

MoreArgs = l i s t ( c e n t e r s = cen , nrow = R, nco l= C) , output . t ype = ” da r r a y ” , combine = ” r b i n d” , npa r t s = c ( ncores , 1 ) )

7 #Ca l l d i s t r i b u t e d K−means l i b r a r y8 model <− dkmeans ( f e a t u r e s ,K)

Figure 2: A distributed K-means application.

1 1 2 2

1 1 2 2

3 3 4 4

3 3 4 4

1 1

1 1

2 2

2 2

3 3

3 3

4 4

4 4

A = darray(nparts=(2,2)…) Logical view

2D partitions

Figure 3: Storing distributed arrays as a set of par-titions.

arrays, where each element in the cell has a value, many ap-plications need sparse arrays to efficiently store their data.Therefore, ddR also supports sparse matrices which arestored in the column compressed format [13]. By default,distributed arrays are stored in column major order and eachpartition of a distributed array can be operated by highlyoptimized matrix libraries such as BLAS [17]. Distributeddata frames (dframe) are similar to arrays, with the excep-tion that each column can store elements of a different type.Distributed data frames are also stored in columnar format.Distributed lists (dlist) store a sequence of elements whereeach element can be a complex data type such as a matrix.

The combination of these three data structures has beensufficient to express many real world applications. Considerthe example of graph analysis. The connectivity in a graphcan be represented as a dlist of edges or a sparse darray forfast matrix manipulation (e.g., PageRank is a series of sparsematrix-vector multiplication). If each vertex has attributes,a dframe can be used to store those attributes.Collection of partitions. Each distributed data structureis internally stored as a set of partitions of the same type.For example, a darray is a collection of array partitions,while a dlist is a collection of local lists. A partition isalways stored in a single server, and the distributed objectcontains metadata information, such as locations and sizesof partitions, to manage data. Unlike dlist, a darray anddframe may be partitioned not only by rows or columns,but also in blocks (i.e., 2-D partitioning). Figure 3 showshow a darray may be stored across multiple servers. Inthis example, the darray is partitioned into 4 blocks, andeach server holds only one partition. The darray argumentnparts in the figure specifies how the partitions are located

1295

1 #Create a d i s t r i b u t e d l i s t . By d e f a u l t eache l ement becomes a p a r t i t i o n

2 A <− d l i s t ( 1 , 2 , 3 , 4 , 5 )3 #Access p a r t i t i o n s4 p <− pa r t s (A)5 #Mu l t i p l y e l ement s i n each p a r t i t i o n by a con s t an t6 B <− dmapply ( f u n c t i o n ( x ) 2∗x [ [ 1 ] ] , p )7 #Fetch the r e s u l t (=2 ,4 ,6 ,8 ,10) on the master8 p r i n t ( c o l l e c t (B) )

Figure 4: Declaring and using distributed datastructures.

in a grid. For example, if the array was row partitionedinstead of blocks, one would use nparts(4,1) to state thatthe four partitions should be stitched row wise by placingthem one below the other.Write-once semantics. A key feature of distributed ob-jects is that they have write-once semantics. Once created,a data structure is immutable. However, a data structurecan be read-shared, even concurrently, by any number of op-erators. This choice of immutable data structures is a resultof embracing the functional programming approach in R.Data structures are operated on by functions dispatched us-ing dmapply, which is side effect free, and returns a new datastructure as the result. Therefore, the programming modelprevents in-place updates to data structures, thus making itmuch easier to parallelize a computation.Accessing partitions. Code may also operate on only asubset of the partitions in a distributed object. In ddR,programmers can use parts(A) to access the partitions of adata structure. The parts function returns a list that con-tains references to individual partitions. Note that there isno data movement between the master and workers whenusing the parts function; it just returns a reference, whichis primarily metadata, about the partitions. In the nextsection, we will explain in greater detail how partitions pro-vide flexibility in expressing algorithms. Finally, program-mers can use the collect keyword to gather the whole datastructure, or a subset of partitions, and materialize them atthe master.Example. Figure 4 shows a simple example that creates adistributed list and accesses its partitions. Line 2 declares adistributed list which holds the numbers 1 to 5. By default itwill be stored as five partitions, each containing one number.In line 4, p is a local R list (not a distributed list) which hasfive elements and each element is a reference to a partitionof A. Line 6 executes a function on each element of p, whichmeans each partition of A, and multiplies each partition by2. The result B is a dlist, has five partitions, and is storedacross multiple nodes. Line 8 gathers the result into a singlelocal R list and prints it.

4. DISTRIBUTED PROCESSINGThe ddR programming model introduces only one paral-

lelism primitive, distributed multivariate apply (dmapply),for distributed processing. When used in combination withdistributed data structures, it provides a flexible way toexpress multiple types of computation and communicationpatterns. Therefore, programmers can use it to implementmany kinds of analytics applications, as well as utility li-braries.

1 A <− d l i s t ( 1 , 2 , 3 , 4 )2 B <− d l i s t (11 ,12 ,13 ,14 )3 #C w i l l be a d l i s t =12 ,14 ,16 ,184 C <− dmapply (FUN=sum , A, B)5 #D w i l l be a d l i s t =13 ,15 ,17 ,196 D <− dmapply (FUN=sum , A, B, MoreArgs= l i s t ( z=1) )

Figure 5: Using dmapply.

The key arguments of distributed apply, dmapply(FUN,

A,B,..., MoreArgs), are (1) FUN that is a function, (2) mul-tiple R data structures on whose elements the function is ap-plied, and (3) a list of R data structures passed as MoreArgsthat are additional input to the function FUN. Note that theR data structures can be both basic R types, such as vectorsand lists, as well as distributed data structures. There arethree additional arguments that are used to define the typeand shape of the output, but we will not focus on them inthis paper.

The dmapply primitive applies the function FUN to eachelement of the input data structure. Let us assume thereare only two distributed lists, A and B, as inputs, and thefunction is sum. The runtime will extract the first elementof A and B and apply sum on it, extract the correspondingsecond elements, and so on. Line 4 in Figure 5 shows thecorresponding program and its results. The MoreArgs argu-ment is a way to pass a list of objects that are available asan input to each invocation of the function. As an example,in line 6 of Figure 5, the constant z is passed to every invo-cation of sum, and hence 1 is added to each element of theprevious result C.

4.1 Communication and computation patternsThe dmapply interface is powerful because it can express

many types of computation patterns.Function broadcast. A common programming paradigmis to apply a function on each element of a data structuresimilar to the map function in prior systems [10]. This canbe achieved in ddR by simply calling dmapply(FUN,A). Infact, programmers can also express that a function should beapplied to each partition at a time instead of each elementat a time by calling dmapply(FUN, parts(A)). In this case,the runtime will invoke FUN on a full partition. If thereis only one input data structure, the runtime ensures thatthere is no data movement by shipping the function to thecorresponding location of the data. Figure 6(A) illustrateshow a function is broadcast to different partitions of a data.Data broadcast. In some cases, programmers need to in-clude the same data in all invocations of a function. Asan example consider the K-means clustering algorithm thatiteratively groups input data into K clusters. In each itera-tion, the distance of the points to the centers has to be cal-culated, which means the centers from the previous iterationhave to be available to all invocations of the distance cal-culation function. In dmapply, programmers use MoreArgs

to specify what data structures need to be present at allfunction invocations. The ddR runtime supports both stan-dard R objects as well as distributed data structures, andpartitions of it, in the MoreArgs field. The runtime may re-assemble a distributed data structure or parts of it, and thenbroadcast it to all worker nodes so that the data is availableduring computation. Figure 6(B) shows how data may bebroadcast to all workers.

1296

1 1 2 2 3 3

sum() sum() sum()

A) Function broadcast: dmapply(sum, A)

Worker 1 Worker 2 Worker 3

Masterfunction

1 1 2 2 3 3

sum() sum() sum()

B) Data broadcast: dmapply(sum, A, MoreArgs=list(z=10))

Worker 1 Worker 2 Worker 3

Master

z=10 z=10 z=10broadcast values

function + data

1 1 2 2 3 3

sum() sum()

C) Partition based: dmapply(sum, parts(A)[1:2], MoreArgs=list(parts(A)[3]))

Worker 1 Worker 2 Worker 3

Master

3 33 3 data

function

Figure 6: Example computation patterns in ddR.

Figure 7 shows one implementation of distributed ran-domforest using ddR. Randomforest is an ensemble learningmethod that is used for classification and regression. Thetraining phase creates a number of decision trees, such as500 trees, that are later used for classification. Since train-ing on large data can take hours, it is common to parallelizethe computationally intensive step of building trees. Line 3in Figure 7 uses a simple parallelization strategy of broad-casting the input to all workers by specifying it in MoreArgs.Each worker then builds 50 trees in parallel (ntree=50) bycalling the existing single threaded randomforest function.At the end of the computation, all the trees are collected atthe master and combined to form a single model (line 5). Inthis example, the full contents of a single data structure arebroadcast to all workers.Accessing a subset of data partitions. The dmapply

approach allows programmers to operate on any subset ofpartitions that contain distributed data. As an example,dmapply(length, parts(A)[1:2]) will find the lengths ofthe two partitions. In this example, the user invokes thelength function only on two partitions of the complete datastructure. Similarly, a subset of the data can be passed toeach invocation of the function by referring to the relevantpartitions in MoreArgs. In general, this kind of usage canresult in data movement, since the runtime has to aggregatethe corresponding data partitions. Figure 6(C) shows howa function is applied to a subset of data partitions, after acertain partition from a remote worker is made available.Multivariate traversal. An important aspect of dmapplyis that it supports simultaneous iteration over multiple datastructures. Irrespective of the number of data structures,the invariant is that the function FUN will be invoked on the

1 #Inpu t data i s a d i s t r i b u t e d data frame2 i n pu t<−dframe ( . . )3 dmodel <− dmapply ( randomfore s t , 1 : 10 , MoreArgs =

l i s t ( data=input , n t r e e =50) )4 #Combine d i s t r i b u t e d t r e e s5 model <− do . c a l l ( randomForest : : combine , c o l l e c t (

dmodel ) )

Figure 7: Distributed randomforest.

first element of each data structure, followed by the secondset of elements, and so on. If the number of elements in allthe data structures is not the same, then the user can specifydmapply to repeat elements of the smaller data structure orthrow an error.

There is a subtle point about what constitutes an element.For a dlist A, the first element is A[[1]], even if that el-ement is another data structure such as a vector or matrix(since a list can hold complex data types). For a darray

B, each cell is an element, and the elements are iterated incolumn major order. For example, the first element in atwo dimensional distributed array is B[1,1]. For a dframe

C each column is an element, i.e., the first element is thefirst column, the second element is the second column, andso on. In the above example if a user writes a statementdmapply(FUN, A,B,C) then the runtime will aggregate thefirst element of A, first cell of B, and first column of C, andapply FUN on them , and so on. Ideally, the number of el-ements in list A should equal the number of cells in B (no.of rows × no. of cols), which should equal the number ofcolumns in C. These iteration rules of the distributed datastructures follow those of their non-distributed counterpartsin R (lists, matrices, and data.frames), so R users can easilyreason about them as though they were standard R datastructures.

In addition to iterating on the the data structure elementby element, the user can also iterate on respective partitionsof different data structures. For example, dmapply(FUN,

parts(A), parts(B), parts(C)) applies FUN to the corre-sponding partitions of the dlist, darray, and dframe. Thisiteration policy conforms to the rules we mentioned earliersince parts returns a list and iteration occurs on each el-ement of a list, which is a partition in this case. Wheniterating over multiple data structures, the physical layoutof the data, i.e., how data is stored on each machine, deter-mines whether data movement will occur. If correspondingelements of the data structures are co-located, then no datamovement is required. Otherwise, the runtime will movedata before dispatching the function FUN.

4.2 ExpressivenessSince ddR advocates the use of a single parallelism prim-

itive, dmapply, a natural question to ask is whether it issufficient to express common machine learning algorithms.We use two approaches to show the expressiveness of ddR.First, we show how algorithms that fit the statistical querymodel [16] can be expressed in ddR. Since a number ofcommon algorithms, such as K-means, regression, PCA andothers, adhere to the statistical query model, it implies thatthese algorithms can be expressed in ddR. Second, we takethe MapReduce programming model as an example andshow how its functionality can be achieved using dmapply.

1297

4.2.1 Machine Learning algorithmsA statistical learning query is a query made over data

that computes PData [F (x, y) = 1], where F is an arbitraryboolean-valued function, x is an observation, and y is thecorresponding label for that sample. Intuitively, the queryruns the function F over the entire training dataset, andreturns the proportion of samples on which F evaluates toone. Algorithms that can be expressed via repeated appli-cation of these learning queries are said to fit the statisticallearning query model. While it may appear that the statis-tical query model is restrictive, it has been shown that mostlearning problems can be reduced to a statistical learningquery formulation [16, 27].Implementation using dmapply. Chu et. al. have shownhow algorithms that fit the statistical query model can bewritten as summations over data, and hence be implementedin MapReduce [27]. We use a similar reduction strategy toshow that algorithms in the statistical query model can beexpressed in ddR. In ddR, we use a series of dmapply andcollect statements to execute statistical learning queries.Let us assume that the training samples X and their respec-tive classes Y are sampled from the distribution D. Assum-ing the function F returns a boolean value, we can computePData [F (x, y) = 1] using the ddR code in Figure 8.

1 FUN <− f u n c t i o n (X,Y, F) sum( s app l y ( 1 : nrow (X) ,f u n c t i o n ( i , x , y , f ) f ( x [ i , ] , y [ i , ] ) ,X ,Y, F) )

2 P <− c o l l e c t ( dmapply (FUN, pa r t s (X) , pa r t s (Y) ) )3 Q <− sum(P)/nrow (X)

Figure 8: Computing summations in parallel.

In Figure 8, the function FUN computes the summation∑x∈Xk

F (x, y) for each partition k of the input data X andY. We use dmapply to execute this summation in paralleland then collect the total sum P =

∑x∈X F (x, y). Finally,

to calculate the probability we divide the summation by thenumber of rows of X using nrow. Note this same formulationcan be used to calculate E [F (x, y)] when F is not a booleanfunction.

Let’s consider a concrete example such as linear regres-sion to show how the summation formulation can be usedto implement algorithms. The goal of linear regression isto find β such that (Xβ − y)T (Xβ − y) is minimized. Onemay use the gradient descent method to iteratively calcu-late βi, which is the estimate of β at iteration i. Since thegradient of (Xβ − y)T (Xβ − y) is 2XTXβ − 2XT y, βi+1 =βi − (2XTXβi − 2XT y). Therefore, the main computationin linear regression is to calculate XTX =

∑x∈X xTx and

XT y =∑

x∈X XT y, both of which are summation problems.These summations can be calculated in a distributed fash-ion which significantly reduces the overall execution timewhen X is large. As it turns out, common machine learn-ing algorithms such as K-means, PCA, Gaussian mixturemodels and others can also be expressed as series of simplesummations [27].Tree-based algorithms. Many learning algorithms, suchas randomforest and gradient boosted trees, are an ensembleof decision trees. We show that such tree-based algorithmscan also be expressed in ddR. The main goal of these al-gorithms is to segment the sample space Ω into Ωi where⋃

Ωi = Ω, and minimize the loss function F(Y, Y

)where

Yi := EΩi [Yi|Xi, Xi ∈ Ωi] on each subset Ωi. Common lossfunctions are information entropy and variance. We showthat the evaluation of these loss functions can be expressedusing P [F (x, y) = 1] and E [F (x, y)], which are statisticalqueries on data and can be calculated in ddR using thecode in Figure 8.

If the response variable Y is numeric, one can use vari-ance as the loss function. This loss function can be eval-uated by calculating, in parallel, the first and second mo-ment of Y , EΩi [Y ] and EΩi

[Y 2], on a subset Ωi. EΩi [Y ]

and EΩi

[Y 2]

can be calculated with the same ddR codein Figure 8 since EΩi [y] = E [y] /P ((x ∈ Ωi) = 1) andEΩi

[y2]

= E[y2]/P ((x ∈ Ωi) = 1).

If the response Y is a categorical variable, then one canuse gini or information gain as the loss function. The firststep in such a computation is to obtain the probability dis-tribution of Y on Ωi which requires computing the densityfunction PΩi [y = k] for all values of k on the sets Ωi. Again,PΩi [y = k] /P ((x ∈ Ωi) = 1) can be calculated in parallelwith the same ddR code as calculating P [F (x, y) = 1] (Fig-ure 8) where the function F returns 1 if y = k,X ∈ Ωi and0 otherwise.Performance. We have shown that ddR’s primitives arepowerful enough to express a number of parallel algorithms.However, expressiveness does not imply efficient implemen-tation or high performance. Therefore, we use the extensiveevaluation in Section 6 to confirm that these ddR algo-rithms are indeed scalable and have good performance, evenwhen compared to custom machine learning libraries.

4.2.2 Relationship with MapReduceMapReduce is a functional programming interface pop-

ularized by the open source Hadoop ecosystem. We showthat ddR can express algorithms and applications writtenin MapReduce. Our goal is not to promote the use of theMapReduce interface or port applications written in MapRe-duce to ddR. Instead, we simply focus on the expressivepower of dmapply by demonstrating its relationship withMapReduce.

Let us assume that a program uses M as the map functionfollowed by R as the reduce function. The following code inddR gives the same result as running MapReduce with M

and R:

1 P = l e ng t h ( pa r t s (X) )2 mapped = dmapply (M,X)3 temp = dmapply (H(p , i ) , p = rep ( pa r t s (mapped ) ,P) , i

= s app l y ( 1 :P , f u n c t i o n ( i i ) r ep ( i i , P) ) )4 r educed = dmapply (R , l a p p l y ( 0 : ( P−1) , f u n c t i o n ( i )

pa r t s ( temp ,P∗ i + 1 :P) ) )

In the above code, the first dmapply statement appliesthe map function M on the input data and stores the resultsin the distributed object mapped. The second dmapply calluses the splitting function H (typically a hash function onkeys) to divide the contents of each partition of mapped intosub-partitions that are stored in the distributed object temp.Finally, the third dmapply call gathers all the partitions oftemp which have the same key and applies the reductionfunction R.

5. IMPLEMENTATIONAs shown earlier in Figure 1, ddR is implemented in three

layers. The top layer is the application code, such as a dis-tributed algorithm, which makes calls to the ddR API (e.g.,

1298

dmapply) and associated utility functions (e.g., colSums).The second layer is the core ddR package, which containsthe implementations of the ddR API. This layer is responsi-ble for error checking and other tasks common across back-ends, and invokes the underlying backend driver to delegatetasks. It consists of about 2,500 lines of code that providegeneric definitions of distributed data structures and classesthat the backend driver can extend. Finally, the third layerconsists of the backend driver (usually implemented as a sep-arate R packaged such as distributedR.ddR) and is respon-sible for implementing the generic distributed classes andfunctions for that particular backend. Typically, a backenddriver implementation may involve 500–1,000 lines of code.Distributed algorithms. We have implemented four ddRalgorithms and made them available on CRAN: K-meansclustering, generalized linear models, PageRank, and ran-domforest. Each algorithm was implemented using the ab-stractions provided by ddR (e.g., dmapply, parts, collect),and store their data in dlists, darrays, or dframes. Someof the algorithms were implemented in only 250 lines of Rcode while others required more than 2,000 lines. The coreof these distributed algorithms require only a hundred orso lines of code. Most of the remaining code is related toproviding functionality similar to existing single-threaded Ralgorithms, such as robust error handling.Distributed operators. A major limitation of existingdistributed frameworks is that they do not provide com-mon utility functions that R users expect. As an example,a data scientist will routinely use the summary function onan array to obtain the min, max, and quantile like statis-tical measures of the data. Similarly, a data scientist mayrequire functions to combine two arrays (rbind), find sumof each row (rowSums), or list the last few elements in thedata structure (tail). ddR removes this limitation of dis-tributed frameworks by expressing common utility functionsusing distributed data structures and dmapply. As long asthe existing frameworks implement dmapply, the users willbe able to benefit from these utility functions. We have im-plemented a number of utility functions in ddR. As an ex-ample, the distributed implementation of rowSums in ddRfirst collects the local summation for each partition usingcollect(dmapply(rowSums, parts(x))) and then mergesthe results at the master node.Extensibility. ddR follows an object-oriented program-ming pattern, implemented as S4 classes in R [7]. The mainddR package defines the abstract classes for distributed ob-jects, while backend drivers are required to extend theseclasses via inheritance. This permits drivers to override de-fault generic operators in ddR (e.g., sum, rowMeans) if thebackend has a more optimized implementation of the samefunctionality. For example, one can implement a genericgroupBy operation in ddR by using a dlist with k× p par-titions (where k is the number of grouping classes and pis the number of partitions of the distributed object to begrouped), and then shuffle the partitions using parts anddmapply. However, Spark already provides a groupBy oper-ation that is more efficient, and the ddR driver for Sparkreuses the version provided by Spark instead of the genericimplementation.

6. EVALUATIONThe strength of ddR is its unified interface for distribut-

ing computing, and the flexibility to choose the best backend

for the task in hand. In this section, we empirically showthree aspects of ddR:

• We show that the same ddR algorithm can indeed beexecuted on a variety of backends such as R’s parallel,SNOW, HPE Distributed R, and Spark, both in single-server and multi-server setups.

• We show that these ddR algorithms have good per-formance and scalability, and are competitive with al-gorithms available in other products.

• We show there is very little overhead of using ddR’sabstractions. Our algorithms implemented in ddRhave similar performance to algorithms written directlyin the respective backend.

Setup. All experiments use a cluster of 8 HP SL390 serversrunning Ubuntu 14.04. Each server has 24 hyperthreaded2.67 GHz cores (Intel Xeon X5650), 196 GB of RAM, 120GB SSD, and are connected with full bisection bandwidth ona 10Gbps network. We use R 3.2.2 with parallel and SNOW,Distributed R 1.2.0, and Spark 1.5. We also compare againstestablished open source machine learning products such asH2O 3.6 [3] and Spark MLlib. H2O is a high-performancemachine learning library written in Java. It provides an Rinterface to invoke the H2O algorithms. Spark MLlib is themachine learning project of Spark and uses optimized linearalgebra libraries for performance. We use three machinelearning algorithms in our evaluation: (a) randomforest, adecision tree based ensemble learning method, (b) K-meansclustering algorithm, and (c) linear regression.

6.1 Single server setupWe first evaluate the case when a user requires only a sin-

gle server to run computations, either because the input datais small enough or the CPU cores in a single server providesufficient performance benefits. When using ddR, the usercan choose R’s parallel or SNOW as the backend which runonly on a single server. One can also use HPE DistributedR or Spark in a single node mode, though these systemsare primarily targeted at multi-server computations. Themain difference between parallel and SNOW is in the mech-anisms they use for managing multi-process execution. Theparallel backend uses Unix fork command to start multi-ple processes that communicate through inter-process com-munication. Because of its reliance on Unix system calls,the fork based parallel package does not run on Windowsservers. The SNOW package starts multiple processes thatcommunicate using sockets, and can be used even in a Win-dows environment.

6.1.1 ScalabilityFigure 9 shows the performance of parallel randomfor-

est as we increase the number of cores. We use a datasetwith 1M observations, 10 features, and run the randomfor-est algorithm to create 500 decision trees. Randomforest isa compute-intensive algorithm, and the single-threaded de-fault algorithm in R takes about 28 minutes to converge.Using ddR, we can parallelize the tree building phase byassigning each core to build a subset of the 500 trees. Byusing multiple cores, each of the backends parallel, SNOW,and HPE Distributed R can reduce the execution time toabout 5 minutes with 12 cores. For this algorithm all the

1299

1 2 4 8 12

05

1015202530

No. of cores

Per-i

tera

tion

time

(min

.)

Rsnow.ddRspark.ddRdistributedR.ddRparallel.ddR

Figure 9: Single server randomforest. Lower is bet-ter.

backends have similar performance, and achieve substantialspeedups, about 6× by going from a single core to 12 cores.

Figure 10 shows how long it takes to cluster 1.2M pointswith 100 attributes into 500 groups. R’s default single-threaded algorithm takes 482s for each iteration of K-means.When using SNOW, the ddR version of K-means takes 96swith 12 cores. HPE Distributed R and parallel provide thebest performance in this setup, completing each K-means it-eration in just 10s with 12 cores. In addition these systemsshow near linear speedup on K-means, where the perfor-mance improves by 10× when going from a single core to12 cores. The performance of SNOW is worse than othersbecause of its inefficient communication layer. SNOW incurshigh overheads when moving the input data from the mas-ter to the worker processes using sockets. In comparison,the parallel workers receive a copy-on-write version of theinput data when the parent process creates child processesusing the fork system call.

Finally, Figure 11 shows the scalability of parallel linearregression algorithm. Since regression is less compute inten-sive than K-means or randomforest, we use a larger datasetwith 12M records each with 50 features. R’s single-threadedregression algorithm converges in 141s. The ddR regressionalgorithm on HPE Distributed R takes 155s with a singlecore but converges in 33s with 12 cores. The parallel ver-sion is faster and converges in around 20s with 12 cores,which corresponds to about 5× speedup over its single coreperformance. Since this dataset is multi-gigabyte, SNOW takestens of minutes to converge, of which most of the time isspent in moving data between processes. Therefore, we ex-clude SNOW from the figure.

6.1.2 Comparison with H2O and Spark MLlibIn addition to scalability of ddR algorithms, we also mea-

sure how these algorithms compare to state-of-the-art ma-chine learning products such as H2O and Spark MLlib.

H2O is a multi-threaded machine learning product andhas been embraced by the R community for its performance.Unlike other parallel R packages, H2O provides parallel im-plementations of algorithms itself instead of a generic inter-face to write parallel programs. Figure 10 and Figure 11 in-clude the performance of the corresponding H2O algorithms.Our evaluation shows that the ddR version of K-means onparallel is about 1.5× faster than H2O’s K-means (Fig-ure 10). For example, ddR can complete each iteration in

1 2 4 8 12

0

100

200

300

400

500

No. of cores

Per-i

tera

tion

time

(sec

.)

Rsnow.ddRspark.ddRdistributedR.ddRH2OSpark MLlibparallel.ddR

Figure 10: Single server K-means. Lower is better.

1 2 4 8 12

050

100150200250300

No. of cores

Con

verg

ence

tim

e (s

ec.)

RSpark MLlibdistributedR.ddRH2Oparallel.ddR

Figure 11: Single server regression. Lower is better.

less than 7s with parallel with 12 cores compared to morethan 10s by H2O. Figure 11 shows that ddR’s regressionimplementation with parallel is comparable to H2O, withthe H2O implementation slightly faster at 8 and 12 cores.The reason for the slight performance advantage is becasueH2O uses multi-threading instead of multi-processing, as byparallel, which lowers the cost of sharing data across work-ers. However, the ddR algorithms on parallel and HPEDistributed R outperform H2O in their ability to handlevery large data. As an example, ddR algorithms on thesebackends have similar scalability even on 5× the input datasize, while the H2O algorithms crash on large datasets.

Spark MLlib provides native implementation of machinelearning algorithms using Spark’s Scala API. Figure 10 showsthat Spark MLlib’s K-means algorithm has similar perfor-mance as H2O, and is slightly slower than the ddR algo-rithm running on parallel. Figure 11 shows that the re-gression implementation in Spark MLlib, when using 4 orless cores, is about 2× slower than both H2O and ddR’simplementation on parallel or HPE Distributed R. At 8or more cores the performance of Spark MLlib is compara-ble, but still less, than the other systems.

6.2 Multi-server setupThe same ddR algorithms that work on a single server

also run in multi-server mode with the appropriate backend.In this section we show that ddR algorithms can processhundreds of gigabytes of data and provide similar scalabilityas custom implementations.

1300

1 2 4 8

050

100150200250

No. of servers

Per-i

tera

tion

time

(sec

.)distributedR.ddRDistributed R

Figure 12: Regression: Strong scaling. Lower is bet-ter.

6.2.1 Strong scaling resultsRegression is a popular algorithm in the financial sector

and is commonly applied on millions of records. We use Dis-tributed R to show how linear regression with ddR can bescaled horizontally to utilize multiple servers. For the ex-periments, we synthetically generated a dataset with 120Mrows by creating vectors around coefficients that we expectto fit the data. This methodology ensures that we can checkfor accuracy of the answers. The dataset size is approxi-mately 95GB and has 100 features per record. Both Dis-tributed R and ddR’s implementation of regression use theNewton-Raphson method. Figure 12 shows how distributedlinear regression scales on upto 8 nodes, each using 12 cores.The custom regression algorithm in Distributed R takes 227sper-iteration with a single server which reduces to 74s with8 servers. Therefore, on this dataset the Distributed R algo-rithm shows about 3× speedup as the number of servers isincreased to 8. The ddR version of regression, running onDistributed R as the backend, shows similar performanceand scalability. On a single server it takes about 251s tocomplete an iteration which reduces to 97 seconds with 8servers. Therefore, the custom regression algorithm in Dis-tributed R is only 23% faster than the ddR version. TheddR algorithm has the added advantage that it runs onother backends as well, thus giving a single interface to theR users.

6.2.2 Weak scaling resultsNext, we show the scalability of ddR’s K-means cluster-

ing algorithm as the dataset size increases. We use syntheticdatasets, with 30M, 60M, 120M, and 240M rows. Eachdataset has 100 features, and we set the number of centers(K) to 1,000. The 240M row datasets corresponds to ap-proximately 180GB of data. Figure 13 compares the perfor-mance of ddR’s K-means clustering algorithm on Spark andDistributed R. Additionally, we plot the execution time ofthe custom K-means algorithms that ship with DistributedR and Spark MLlib. As we increase the number of nodesfrom 1 to 8, we proportionally increase the number of rowsin the dataset from 30M to 240M (i.e., 8×). In an idealdistributed system the per-iteration execution time shouldremain constant for this setup.

There are three interesting observations from Figure 13.First, the ddR version of K-means, on both Spark and Dis-tributed R, scales almost perfectly as the dataset size and

1 node/30M 2 node/60M 4 node/120M 8 node/240M

02468

10

Per-i

tera

tion

time

(min

.)

Spark MLlibspark.ddRdistributedR.ddRDistributed R

Figure 13: K-means: Weak scaling. Lower is better.

the number of machines is increased. When Spark is usedas the backend, the ddR algorithm takes around 7 min-utes per iteration. With Distributed R as the backend theper-iteration time of ddR is around 6 minutes. Second,on this dataset the Distributed R backend outperforms theSpark backend. Therefore, if a user has both the backendsinstalled, it can choose to run the application written inddR, without any modifications, on Distributed R for betterperformance. Finally, our evaluation shows that the ddRversion of the algorithm gives the same or sometimes evenbetter performance than the custom algorithm implementa-tion on the same framework. The ddR algorithm with Dis-tributed R as the backend (distributedR.ddR) is within 5%of the K-means library that ships with Distributed R. TheddR algorithm with Spark as the backend is in fact slightlyfaster than Spark’s MLlib algorithm when the dataset sizeis 120M and 240M.

6.3 SummaryThe single server and multi-server results validate our hy-

pothesis that ddR provides application portability alongwith good performance. For example, algorithms written inddR can run on different backends without any code mod-ifications or the user fearing loss of accuracy in the results.The performance of the same ddR algorithm will dependon the backend used (e.g., how well a backend handles datatransfers), but we found that most backends, with the ex-ception of SNOW, result in good performance. For the singleserver case, we recommend using parallel which providesthe best performance, and is readily available since it shipswith R. On multi-server environment, a user may choose torun ddR with Distributed R or Spark, based on availabil-ity of the infrastructure. Even though the ddR algorithmshave been written independent of the backend, our resultsshow that their performance is comparable to, and some-times even better than, the native algorithms in other ma-chine learning frameworks such as H2O and Spark MLlib.

7. RELATED WORKDistributed frameworks. MapReduce started the waveof programming models and infrastructures that provide asimple yet fault tolerant way to write distributed applica-tions [10]. Even though simple, many found the MapRe-duce model too low level to program. As a result, the sec-ond wave of distributed systems, such as Pig [23], HIVE [28]and DryadLINQ [33], focused on programmer productivityby providing a SQL like interface. These systems are batchprocessing systems and not well suited for iterative machine

1301

learning or streaming tasks. Even the machine learning Ma-hout library, based on the open source MapReduce imple-mentation, has fallen out of favor due to its poor perfor-mance [1]. Finally, the third wave of distributed systemsfocus on domain specific applications and leverage hard-ware, such as main memory, to improve performance. Sparkuses in-memory processing and lineage based fault toler-ance to improve performance of distributed applications [34].Pregel [21], GraphLab [19], and Concerto [18] advocate avertex-centric programming model to express both graph al-gorithms as well as machine learning algorithms. Storm [29],Spark Streaming [35] and Naiad [22] provide support forstreaming applications.

For data scientists, implementing applications on thesedistributed frameworks with custom interfaces is a night-mare. There have been efforts to provide R-friendly in-terfaces to many of these systems. Notable examples areRicardo [9], RHadoop [6], and SparkR for the MapReduceand Spark programming model. Unfortunately, these inter-faces typically expose the custom underlying functions ofthe framework but in R, such as map and reduce in the caseof RHadoop. The data scientist still needs to learn the un-derlying system. Moreover, applications written using theseinterfaces are not portable across frameworks. ddR solvesthese issues by exposing a simple unified interface with simi-lar semantics as R’s current single-threaded data structures.

SystemML provides a declarative approach for machinelearning which initially focused on MapReduce and now onSpark [12]. By relying on a declarative syntax it inheritscertain powerful aspects of databases such as a plan opti-mizer [8]. ddR differs from the approach of SystemML byimproving R itself, instead of proposing a new declarativelanguage, introducing a simple but expressive distributedparallelism primitive, dmapply, and implementing portabledistributed algorithms and utility functions. One could en-vision integrating ddR and certain optimization techniquesof SystemML similar to how compilers provide program op-timizations.Databases and machine learning. Most databases pro-vide support for machine learning algorithms either throughthe ability to call custom UDFs or by exposing in-built algo-rithms or by integrating with external tools. Many vendorssuch as Oracle, HPE Vertica, Microsoft SQL server, andothers embed R in their database. Users can call any ofthe thousands of R packages inside a single UDF, thoughperformance is limited by the single threaded R package.As an example, customers may funnel a table through asingle threaded R K-means UDF, but they cannot create adistributed K-means function by simply invoking multipleK-means UDFs. In all these cases, the R user interacts withthe database primarily through a thin SQL wrapper, suchas the hugely popular dplyr package [32]. These wrapperR packages are natural for data wrangling in SQL but donot provide a portable way to implement machine learningalgorithms. MADlib like approaches show how in-database,machine learning algorithms can be implemented via user-defined functions, and SQL statements [15, 2]. Unfortu-nately, to contribute an in-database algorithm one needsto follow the programming paradigm and the low level lan-guage API proposed by MADlib. Finally, many databasessupport fast connectors to external frameworks, from singlethreaded R with SAP HANA [14] to distributed frameworkslike Spark and HPE Distributed R with Vertica [24]. When

using external frameworks to run machine learning applica-tions outside of the database, the user still has to expressher application using the custom interface exposed by Sparkor Distributed R.Parallel libraries in R. R is inherently single-threaded,but there are over 25 packages that provide some form ofparallelism extensions to R [11]. Some well known packagesinclude parallel, SNOW, foreach, and Rmpi. All of thesepackages expose their custom syntax for expressing paral-lelism, and none of them have the concept of distributeddata structures. Not surprisingly, even with so many par-allel extensions, R has hardly any parallel machine learningalgorithms written using them. While the parallel pack-age exposes parallelism based on the functional semanticsof the apply class of functions, it is a single node solutionand does not tackle distributed systems issues such as par-titioned distributed data structures. HPE Distributed pro-vides an infrastructure for distributed computing in R, buthas similar shortcomings: it exposes a custom syntax andapplications written on it cannot run on other frameworkssuch as SparkR [30, 31].

8. ONGOING WORKWhile we have focused on expanding the data structures

available in R and integrating with recent distributed ana-lytics systems, one could imagine using a parallel databaseas a ddR backend. By using a parallel database and ddR,R users can continue to program in their familiar API, whilebenefiting from the optimized execution engines of databases.In our ongoing work to integrate ddR with a database (suchas Vertica), we leverage the fact that many databases sup-port R based user-defined functions. Here we outline oneway to implement a database driver for ddR.Maintaining order and partitions. Elements of R ob-jects (e.g., matrix) have implicit structure and ordering withrespect to other elements. We expect a database table to in-clude specific columns that denote row or column indices ofeach element. Similarly, the ddR interface also maintainsa correspondence between data and the partitions to whichthey belong. One could create these partitions on the fly us-ing indices or store an explicit database column for partitionids.Database query. We can express a multivariate dmapply

by using joins and R user-defined functions. One way toimplement the generic dmapply(FUN(x,y,z), X, Y,

MoreArgs = list(z=3.0)) function in Vertica, while ensur-ing parallel UDF processing, is as follows:

SELECT FUN(COMBINED. data1 ,COMBINED. data2 , 3 . 0 ) OVER(PARTITION BY COMBINED. p a r t i t i o n )

FROM (SELECT X. data AS data1 ,Y . data AS data2 ,X .p a r t i t i o n AS p a r t i t i o n FROM

X JOIN Y ON X. i ndex=Y. i ndex ) AS COMBINED;

9. CONCLUSIONR is a powerful tool for single-threaded statistical analy-

sis, but lacks proper language bindings for distributed com-puting. Our design and implementation of ddR is a firststep in extending the R language and providing a unifiedinterface for distributed computing. We have shown the ex-pressive power of the ddR interface, and how algorithmsimplemented in this interface have performance compara-ble to custom implementations. Customers will no longer

1302

need to struggle with custom interfaces for each distributedbackend; instead, they can write portable R applicationsusing ddR, and leverage the already implemented ddR al-gorithms across their favorite distributed computing infras-tructure.

10. ACKNOWLEDGMENTSWe thank Michael Lawrence from R-core who worked with

us to design and implement ddR. We thank Arash Fard andother members of the HPE Vertica team for contributingalgorithms to ddR. We also thank the R Consortium andattendees of the HPE Workshop on Distributed Computingin R for their support and feedback.

11. REFERENCES[1] Apache Mahout. http://mahout.apache.org.

[2] Comparing pattern mining on a billion records withHP Vertica and Hadoop.http://www.vertica.com/2013/04/08/comparing-

pattern-mining-on-a-billion-records-with-hp-

vertica-and-hadoop/.

[3] H2O: Machine learning library. http://www.h2o.ai/.

[4] MATLAB–parallel computing toolbox.http://www.mathworks.com/products/parallel-

computing/.

[5] Revolution R enterprise scaler.http://www.revolutionanalytics.com/revolution-

r-enterprise-scaler.

[6] RHadoop and MapR. https://www.mapr.com/resources/rhadoop-and-mapr.

[7] The s4 object system.http://adv-r.had.co.nz/S4.html.

[8] M. Boehm, S. Tatikonda, B. Reinwald, P. Sen,Y. Tian, D. R. Burdick, and S. Vaithyanathan. Hybridparallelization strategies for large-scale machinelearning in systemml. Proc. VLDB Endow.,7(7):553–564, Mar. 2014.

[9] S. Das, Y. Sismanis, K. S. Beyer, R. Gemulla, P. J.Haas, and J. McPherson. Ricardo: Integrating R andHadoop. In Proceedings of the 2010 ACM SIGMODInternational Conference on Management of Data,SIGMOD ’10, pages 987–998, New York, NY, USA,2010. ACM.

[10] J. Dean and S. Ghemawat. Mapreduce: Simplifieddata processing on large clusters. In Proceedings of the6th Conference on Symposium on Opearting SystemsDesign & Implementation - Volume 6, OSDI’04, pages10–10, Berkeley, CA, USA, 2004. USENIXAssociation.

[11] D. Eddelbuettel. High-performance and parallelcomputing with r. https://cran.r-project.org/web/views/HighPerformanceComputing.html.

[12] A. Ghoting, R. Krishnamurthy, E. Pednault,B. Reinwald, V. Sindhwani, S. Tatikonda, Y. Tian,and S. Vaithyanathan. Systemml: Declarative machinelearning on mapreduce. In Proceedings of the 2011IEEE 27th International Conference on DataEngineering, ICDE ’11, pages 231–242, Washington,DC, USA, 2011. IEEE Computer Society.

[13] J. R. Gilbert, C. Moler, and R. Schreiber. Sparsematrices in matlab: Design and implementation.SIAM J. Matrix Anal. Appl., 13(1):333–356, Jan.1992.

[14] P. Große, W. Lehner, T. Weichert, F. Farber, andW.-S. Li. Bridging two worlds with RICE integratingR into the SAP in-memory computing engine.PVLDB, 4(12), 2011.

[15] J. M. Hellerstein, C. Re, F. Schoppmann, D. Z. Wang,E. Fratkin, A. Gorajek, K. S. Ng, C. Welton, X. Feng,K. Li, and A. Kumar. The MADlib analytics library:Or MAD skills, the SQL. Proc. VLDB Endow.,5(12):1700–1711, Aug. 2012.

[16] M. Kearns. Efficient noise-tolerant learning fromstatistical queries. J. ACM, 45(6):983–1006, Nov.1998.

[17] C. L. Lawson, R. J. Hanson, D. R. Kincaid, and F. T.Krogh. Basic linear algebra subprograms for fortranusage. ACM Trans. Math. Softw., 5(3):308–323, Sept.1979.

[18] M. M.-J. Lee, I. Roy, A. AuYoung, V. Talwar, K. R.Jayaram, and Y. Zhou. Views and transactionalstorage for large graphs. In Middleware, volume 8275of Lecture Notes in Computer Science, pages 287–306.Springer, 2013.

[19] Y. Low, D. Bickson, J. Gonzalez, C. Guestrin,A. Kyrola, and J. M. Hellerstein. Distributedgraphlab: A framework for machine learning and datamining in the cloud. Proc. VLDB Endow.,5(8):716–727, Apr. 2012.

[20] E. Ma, I. Roy, and M. Lawrence. ddR: Distributeddata structures in R. https://cran.r-project.org/web/packages/ddR/index.html.

[21] G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert,I. Horn, N. Leiser, and G. Czajkowski. Pregel: Asystem for large-scale graph processing. In Proceedingsof the 2010 ACM SIGMOD International Conferenceon Management of Data, SIGMOD ’10, pages135–146, New York, NY, USA, 2010. ACM.

[22] D. G. Murray, F. McSherry, R. Isaacs, M. Isard,P. Barham, and M. Abadi. Naiad: A timely dataflowsystem. In Proceedings of the Twenty-Fourth ACMSymposium on Operating Systems Principles, SOSP’13, pages 439–455, New York, NY, USA, 2013. ACM.

[23] C. Olston, B. Reed, U. Srivastava, R. Kumar, andA. Tomkins. Pig latin: A not-so-foreign language fordata processing. In Proceedings of the 2008 ACMSIGMOD International Conference on Management ofData, SIGMOD ’08, pages 1099–1110, New York, NY,USA, 2008. ACM.

[24] S. Prasad, A. Fard, V. Gupta, J. Martinez, J. LeFevre,V. Xu, M. Hsu, and I. Roy. Large-scale predictiveanalytics in vertica: Fast data transfer, distributedmodel creation, and in-database prediction. InProceedings of the 2015 ACM SIGMOD InternationalConference on Management of Data, SIGMOD ’15,pages 1657–1668, New York, NY, USA, 2015. ACM.

[25] D. Robinson. Package downloads over time.https://dgrtwo.shinyapps.io/cranview/.

[26] D. Smith. New surveys show continued popularity ofR. http://blog.revolutionanalytics.com/2015/11/new-

surveys-show-continued-popularity-of-r.html.

[27] C. tao Chu, S. K. Kim, Y. an Lin, Y. Yu, G. Bradski,K. Olukotun, and A. Y. Ng. Map-reduce for machine

1303

learning on multicore. In B. Scholkopf, J. C. Platt,and T. Hoffman, editors, Advances in NeuralInformation Processing Systems 19, pages 281–288.MIT Press, 2007.

[28] A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka,S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive:A warehousing solution over a map-reduce framework.Proc. VLDB Endow., 2(2):1626–1629, Aug. 2009.

[29] A. Toshniwal, S. Taneja, A. Shukla, K. Ramasamy,J. M. Patel, S. Kulkarni, J. Jackson, K. Gade, M. Fu,J. Donham, N. Bhagat, S. Mittal, and D. Ryaboy.Storm@twitter. In Proceedings of the 2014 ACMSIGMOD International Conference on Management ofData, SIGMOD ’14, pages 147–156, New York, NY,USA, 2014. ACM.

[30] S. Venkataraman, E. Bodzsar, I. Roy, A. AuYoung,and R. S. Schreiber. Presto: Distributed machinelearning and graph processing with sparse matrices. InProceedings of the 8th ACM European Conference onComputer Systems, EuroSys ’13, pages 197–210, NewYork, NY, USA, 2013. ACM.

[31] S. Venkataraman, I. Roy, A. AuYoung, and R. S.Schreiber. Using R for iterative and incrementalprocessing. In 4th USENIX Workshop on Hot Topicsin Cloud Computing, HotCloud’12, Boston, MA, USA,

June 12-13, 2012, 2012.

[32] H. Wickham. dplyr:A grammar for data manipulation.https://cran.r-

project.org/web/packages/dplyr/index.html.

[33] Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson,P. K. Gunda, and J. Currey. Dryadlinq: A system forgeneral-purpose distributed data-parallel computingusing a high-level language. In Proceedings of the 8thUSENIX Conference on Operating Systems Designand Implementation, OSDI’08, pages 1–14, Berkeley,CA, USA, 2008. USENIX Association.

[34] M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma,M. McCauley, M. J. Franklin, S. Shenker, andI. Stoica. Resilient distributed datasets: Afault-tolerant abstraction for in-memory clustercomputing. In Proceedings of the 9th USENIXConference on Networked Systems Design andImplementation, NSDI’12, pages 2–2, Berkeley, CA,USA, 2012. USENIX Association.

[35] M. Zaharia, T. Das, H. Li, T. Hunter, S. Shenker, andI. Stoica. Discretized streams: Fault-tolerantstreaming computation at scale. In Proceedings of theTwenty-Fourth ACM Symposium on OperatingSystems Principles, SOSP ’13, pages 423–438, NewYork, NY, USA, 2013. ACM.

1304