mapr, implications for integration

04/11/2023 © MapR Confidential 1

MapR, Implications for Integration

CHUG – August 2011


Outline

• MapR system overview• Map-reduce review• MapR architecture• Performance Results• Map-reduce on MapR

• Architectural implications• Search indexing / deployment• EM algorithm for machine learning• … and more …


Map-Reduce

Input Output

Shuffle


Bottlenecks and Issues

• Read-only files• Many copies in I/O path• Shuffle based on HTTP• Can’t use new technologies• Eats file descriptors

• Spills go to local file space• Bad for skewed distribution of sizes


MapR Areas of Development

Map Re-

duce

Storage Service

s

Ecosystem

HBase

Management


MapR Improvements

• Faster file system• Fewer copies• Multiple NICS• No file descriptor or page-buf competition

• Faster map-reduce• Uses distributed file system• Direct RPC to receiver• Very wide merges


MapR Innovations

• Volumes• Distributed management• Data placement

• Read/write random access file system• Allows distributed meta-data• Improved scaling• Enables NFS access

• Application-level NIC bonding• Transactionally correct snapshots and mirrors


MapR's Containers

Each container contains Directories & files Data blocks

Replicated on servers No need to manage

directly

Files/directories are sharded into blocks, whichare placed into mini NNs (containers ) on disks

Containers are 16-32 GB segments of disk, placed on nodes


Container locations and replication

CLDB

N1, N2

N3, N2

N1, N2

N1, N3

N3, N2

N1

N2

N3

Container location database (CLDB) keeps track of nodes hosting each container


MapR ScalingContainers represent 16 - 32GB of data

Each can hold up to 1 Billion files and directories 100M containers = ~ 2 Exabytes (a very large cluster)

250 bytes DRAM to cache a container 25GB to cache all containers for 2EB cluster

But not necessary, can page to disk Typical large 10PB cluster needs 2GB

Container-reports are 100x - 1000x < HDFS block-reports Serve 100x more data-nodes Increase container size to 64G to serve 4EB cluster

Map/reduce not affected


MapR's Streaming Performance

Read Write0

250

500

750

1000

1250

1500

1750

2000

2250

Read Write0

250

500

750

1000

1250

1500

1750

2000

2250

HardwareMapRHadoopMB

persec

Tests: i. 16 streams x 120GB ii. 2000 streams x 1GB

11 x 7200rpm SATA 11 x 15Krpm SAS

Higher is better


Terasort on MapR

1.0 TB0

10

20

30

40

50

60

3.5 TB0

50

100

150

200

250

300

MapRHadoop

Elapsed time (mins)

10+1 nodes: 8 core, 24GB DRAM, 11 x 1TB SATA 7200 rpm

Lower is better


HBase on MapR

Recordsper

second

Higher is betterZipfian Uniform0

5000

10000

15000

20000

25000

MapRApache

YCSB Random Read with 1 billion 1K records10+1 node cluster: 8 core, 24GB DRAM, 11 x 1TB 7200 RPM


# of files (m)

Rate

(file

s/se

c)

Op: - create file - write 100 bytes - close

Notes:

- NN not replicated

- NN uses 20G DRAM

- DN uses 2G DRAM

Out of box

Tuned

Small Files (Apache Hadoop, 10 nodes)


MUCH faster for some operations

# of files (millions)

CreateRate

Same 10 nodes …


What MapR is not

• Volumes != federation• MapR supports > 10,000 volumes all with

independent placement and defaults• Volumes support snapshots and mirroring

• NFS != FUSE• Checksum and compress at gateway• IP fail-over• Read/write/update semantics at full speed

• MapR != maprfs


New Capabilities


NFS mounting models

• Export to the world• NFS gateway runs on selected gateway hosts

• Local server• NFS gateway runs on local host• Enables local compression and check summing

• Export to self• NFS gateway runs on all data nodes, mounted

from localhost


Export to the world

NFSServerNFS

ServerNFSServerNFS

ServerNFSClient


Client

NFSServer

Local server

Application

Cluster Nodes


ClusterNode

NFSServer

Universal export to self

Task

Cluster Nodes


ClusterNode

NFSServer

Task

ClusterNode

NFSServer

Task

ClusterNode

NFSServer

Task

Nodes are identical


Application architecture

• So now we have a hammer

• Let’s find us some nails!


Sharded text Indexing

MapReducer

Input documents

Localdisk Search

EngineLocal

disk

Clustered index storage

Assign documents to shards

Index text to local disk and then copy index to

distributed file store

Copy to local disk typically required before

index can be loaded


Sharded text indexing

• Mapper assigns document to shard• Shard is usually hash of document id

• Reducer indexes all documents for a shard• Indexes created on local disk• On success, copy index to DFS• On failure, delete local files

• Must avoid directory collisions • can’t use shard id!

• Must manage and reclaim local disk space


Conventional data flow

MapReducer

Input documents

Localdisk Search

EngineLocal

disk


Failure of a reducer causes garbage to accumulate in the

local disk

Failure of search engine requires

another download of the index from clustered storage.


SearchEngine

Simplified NFS data flows

MapReducer

Input documents


Failure of a reducer is cleaned up by

map-reduce framework

Search engine reads mirrored index directly.

Index to task work directory via NFS


Simplified NFS data flows

MapReducer

Input documents

SearchEngine

Mirrors

SearchEngine

Mirroring allows exact placement

of index data

Aribitrary levels of replication also possible


How about another one?


K-means

• Classic E-M based algorithm• Given cluster centroids,• Assign each data point to nearest centroid• Accumulate new centroids• Rinse, lather, repeat


Aggregatenew

centroids

K-means, the movie

Assignto

Nearestcentroid

Centroids

Input


But …


Averagemodels

Parallel Stochastic Gradient Descent

Trainsub

model

Model

Input


Updatemodel

Variational Dirichlet Assignment

Gathersufficientstatistics

Model

Input


Old tricks, new dogs

• Mapper• Assign point to cluster• Emit cluster id, (1, point)

• Combiner and reducer• Sum counts, weighted sum of points• Emit cluster id, (n, sum/n)

• Output to HDFS

Read fromHDFS to local disk by distributed cache

Written by map-reduce

Read from local disk from distributed cache


Old tricks, new dogs

• Mapper• Assign point to cluster• Emit cluster id, (1, point)

• Combiner and reducer• Sum counts, weighted sum of points• Emit cluster id, (n, sum/n)

• Output to HDFSMapR FS

Read fromNFS

Written by map-reduce


Poor man’s Pregel

• Mapper

• Lines in bold can use conventional I/O via NFS

37

while not done: read and accumulate input models for each input: accumulate model write model synchronize reset input formatemit summary


Click modeling architecture

Featureextraction

anddown

sampling

Input

Side-data

Datajoin

SequentialSGD

Learning

Map-reduce

Now via NFS


Click modeling architecture

Map-reduceMap-reduce

Featureextraction

anddown

sampling

Input

Side-data

Datajoin

SequentialSGD

Learning

Map-reduce cooperates

with NFSSequential

SGDLearning

SequentialSGD

Learning

SequentialSGD

Learning


And another…


??

Hybrid model flow

Map-reduce

Map-reduce

Feature extraction and

down sampling

SVD(PageRank)(spectral)

DeployedModel

Down stream

modeling


Map-reduceSequential

Hybrid model flow

Feature extraction and

down sampling

SVD(PageRank)(spectral)

DeployedModel

Down stream

modeling


And visualization…


Trivial visualization interface

• Map-reduce output is visible via NFS

• Legacy visualization just works

$ R> x <- read.csv(“/mapr/my.cluster/home/ted/data/foo.out”)> plot(error ~ t, x)> q(save=‘n’)


Conclusions

• We used to know all this• Tab completion used to work• 5 years of work-arounds have clouded our

memories

• We just have to remember the future

mapr, implications for integration

Technology