apache mesos
Post on 22-Mar-2016
311 Views
Preview:
DESCRIPTION
TRANSCRIPT
Benjamin Hindman – @benh
Apache Mesoshttp://incubator.apache.org/mesos@ApacheMesos
historyBerkeley research project including Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, Ion Stoica
http://incubator.apache.org/mesos/research.html
Mesos aims to make it easier to build distributed applications/frameworks and share cluster resources
applications/frameworks
services analytics
analyticsservices
applications/frameworks
how?
MesosNode Node Nod
e Node
Hadoop service …
Node
Node
Hadoop
Node
Node
service…
level of abstractionmore easily share the resources via multi-tenancy and elasticity (improve utilization)run on bare-metal or virtual machines – develop against Mesos API, run in private datacenter (Twitter), or the cloud, or both!
Hadoop
Spark
service
shared cluster
static partitioning sharing with Mesos
featuresAPIs in C++, Java, Pythonhigh-availability via zookeeperisolation via linux control groups (LXC)
in progress»official apache release»more linux cgroup support (oom and i/o, in
particular, networking)»resource usage monitoring, reporting»new allocators (priority based, usage
based)»new frameworks (storm)»scheduler management (launching,
watching, re-launching, etc)
400+ nodes running production servicesgenomics researchers using Hadoop and SparkSpark in use by Yahoo! ResearchSpark for analyticsHadoop and Spark used by machine learning researchersYour Name
Here
demonstration
linux environment$ yum install -y gcc-c++$ yum install -y java-1.6.0-openjdk-devel.x86_64$ yum install -y make.x86_64$ yum install -y patch.x86_64$ yum install -y python26-devel.x86_64$ yum install -y ant.noarch
get mesos$ wget http://people.apache.org/~benh/mesos-0.9.0-incubating-RC3/mesos-0.9.0-incubating.tar.gz$ tar zxvf mesos-0.9.0-incubating.tar.gz$ cd mesos-0.9.0
build mesos$ mkdir build$ cd build$ ../configure.amazon-linux-64$ make$ make install
deploy mesos (1)/usr/local/var/mesos/deploy/masters:
ec2-50-17-28-135.compute-1.amazonaws.com
/usr/local/var/mesos/deploy/slaves:
ec2-184-73-142-43.compute-1.amazonaws.comec2-107-22-145-31.compute-1.amazonaws.com
deploy mesos (2)on slaves (i.e., ec2-184-73-142-43.compute-1.amazonaws.com, ec2-107-22-145-31.compute-1.amazonaws.com)
/usr/local/var/mesos/conf/mesos.conf:
master=ec2-50-17-28-135.compute-1.amazonaws.com
deploy mesos (3)$ /usr/local/sbin/mesos-start-cluster.sh
build hadoop$ make hadoop$ mv hadoop/hadoop-0.20.205.0 /etc/hadoop$ cp protobuf-2.4.1.jar /etc/hadoop$ cp src/mesos-0.9.0.jar /etc/hadoop
configure hadoop (1)conf/mapred-site.xml:
<configuration> <property> <name>mapred.job.tracker</name> <value>ip-10-108-207-105.ec2.internal:9001</value> </property> <property> <name>mapred.jobtracker.taskScheduler</name> <value>org.apache.hadoop.mapred.MesosScheduler</value> </property>
<property> <name>mapred.mesos.master</name> <value>ip-10-108-207-105.ec2.internal:5050</value> </property></configuration>
configure hadoop (2)conf/hadoop-env.sh:
#!/bin/sh
export JAVA_HOME=/usr/lib/jvm/jre
# Google protobuf (necessary for running the MesosScheduler). export PROTOBUF_JAR=${HADOOP_HOME}/protobuf-2.4.1.jar
# Mesos.export MESOS_JAR=${HADOOP_HOME}/mesos-0.9.0.jar
# Native Mesos library. export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
export HADOOP_CLASSPATH=${HADOOP_HOME}/build/contrib/mesos/classes:${MESOS_JAR}:${PROTOBUF_JAR}
...
configure hadoop (3)conf/core-site.sh:
<configuration> <property> <name>fs.default.name</name> <value>hdfs://ip-10-108-207-105.ec2.internal:9000</value> </property></configuration>
configure hadoop (4)conf/masters:
ec2-50-17-28-135.compute-1.amazonaws.com
conf/slaves:
ec2-184-73-142-43.compute-1.amazonaws.comec2-107-22-145-31.compute-1.amazonaws.com
configure hadoop (4)conf/masters:
ec2-50-17-28-135.compute-1.amazonaws.com
conf/slaves:
ec2-184-73-142-43.compute-1.amazonaws.comec2-107-22-145-31.compute-1.amazonaws.com
starting hadoop$ pwd/etc/hadoop$ ./bin/hadoop jobtracker
running wordcount$ ./bin/hadoop jar hadoop-examples-0.20.205.0.jar wordcount macbeth.txt output
starting another hadoop<configuration> <property> <name>mapred.job.tracker</name> <value>ip-10-108-207-105.ec2.internal:9002</value> </property>
<property> <name>mapred.job.tracker.http.address</name> <value>0.0.0.0:50032</value> </property>
<property> <name>mapred.task.tracker.http.address</name> <value>0.0.0.0:50062</value> </property></configuration>
get and build spark$ git clone git://github.com/mesos/spark.git$ cd spark$ git checkout --track origin/mesos-0.9$ sbt/sbt compile
configure spark$ cp conf/spark-env.sh.template conf/spark-env.sh
conf/spark-env.sh:
#!/bin/sh
export SCALA_HOME=/root/scala-2.9.1-1export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.soexport SPARK_MEM=1g
run spark shell$ pwd/root/spark$ MASTER=$HOSTNAME:5050 ./spark-shell
setting log_diron slaves (i.e., ec2-184-73-142-43.compute-1.amazonaws.com, ec2-107-22-145-31.compute-1.amazonaws.com)
/usr/local/var/mesos/conf/mesos.conf:
master=ec2-50-17-28-135.compute-1.amazonaws.comlog_dir=/tmp/mesos
re-deploy mesos$ /usr/local/sbin/mesos-stop-slaves.sh
$ /usr/local/sbin/mesos-start-slaves.sh
top related