Introduction to Introduction to

Peadar Coyle @springcoil

Luxembourg - Early 2016

Aims of this talkAims of this talk

Explain what Spark is.Explain what Spark is.I'm more a data scientist than an engineer...

Who am I?Who am I?Math and Data nerdInterested in machine learning and data processingSpeaker at PyData/ PyCons throughout Europe

'Big data' so far'Big data' so far

Why care?Why care?big data analytics in memoryResilient Distributed Datasets (RDD)Flexible programming modelscomplements Hadoopbetter performance than Hadoophttps://github.com/springcoil/scalable_ml

Who uses it?Who uses it?Current tech not future tech!Current tech not future tech!

Supported LanguagesSupported Languages

val data = Array(1, 2, 3, 4, 5)val distData = sc.parallelize(data)

sc is 'Spark ContextHere is one RDD

CodeCode

https://github.com/springcoil/scalable_ml/

package scalable_ml

import org.apache.spark.mllib.regression.LabeledPointimport org.apache.spark.mllib.linalg.DenseVectorimport org.apache.spark.rdd.RDDimport breeze.linalg.{DenseVector => BDV}import breeze.linalg.{DenseMatrix => BDM}

class LeastSquaresRegression { def fit(dataset: RDD[LabeledPoint]): DenseVector = { val features = dataset.map { _.features }

val covarianceMatrix: BDM[Double] = features.map { v => val x = BDM(v.toArray) x.t * x }.reduce(_ + _) val featuresTimesLabels: BDV[Double] = dataset.map { xy => BDV(xy.features.toArray) * xy.label }.reduce(_ + _)

val weight = covarianceMatrix \ featuresTimesLabels

new DenseVector(weight.data) }}

Resilient DistributedResilient DistributedDatasets (RDD)Datasets (RDD)

Process in parallelActions on RDDs = transformations and actionspersistance: Memory, Disk, Memory and Disk

Spark EcosystemSpark Ecosystem

Spark streamingSpark SQL - Really the creation of a data frameMore stuff will come soon... IBM and others heavily investing in this.

Any questions?Any questions?