Parallel implementation of ML algorithms on Spark
Dalei Li EIT Digital
https://github.com/lidalei/LinearLogisticRegSpark
1
Overview• Linear regression + l2 regularization
• Normal equation
• Logistic regression + l2 regularization
• Gradient descend
• Newton’s method
• Hyper-parameter optimization
• Experiments
2
Tools
• IntelliJ + sbt
• Scala 2.11.8 + Spark 2.0.1
3
Linear regression• Problem formulation
• Closed-form solution
• Computation reformulation
4
Linear regression• Data set - UCI YearPredictionMSD, text file
• 515,345 songs, (90 audio numerical features, year)
• Core computation - norm terms and rmse
5
Implemented outer product + vector addition
Workflow
6
Read file RegexTokenizer StandardScaler Solve normal equation
Spark SQL textAdd l2 regularization
LAPACK
Center data
Evaluation
rmse
Validation
7
Spark ML linear regression with norm solver vs. my implementation (both with 0.1 l2 regularization)
Randomly split data set into train 70% + test 30%. The RMSEs on test set are also identical, less than 0.5% difference.
Logistic regression• Problem formulation
• Gradient descent
• Newton’s method
• Computation reformulation - gradient and Hessian matrix
8
Logistic regression• Data set - UCI HIGGS, csv file
• 11 million instances, (21+7 numerical features, binary label)
• Core computation - gradient and Hessian matrix
9
treeReduce can reduce the pressure of final ops in driver.
Workflow
10
Read file VectorAssembler DF to RDDgradient descent/
newton’s method
Spark SQL csv Gradient - add l2 regularization
Scala case class Instance (features, label),
Newton’s - append all-one column
Evaluation
cross entropy confusion matrix
Validation
11
Spark ML logistic regression with L-BFGS vs. my implementation of Newton’s method
Randomly split data set into train 70% + test 30%. The learned THETAs are almost identical, the last one is bias.
• Grid search to find optimal hyper-parameters with best generalization error
• Estimate generalization error
• k-Fold cross validation
Hyper-parameter optimization
12
Hyper-parameter is a parameter used in a training process but not a part of a classifier itself. It controls what kind of parameters can / tend to be selected. For example, polynomial expansion will make non-linear relationship between a label and features be learned possibly.
Grid search
• Grid - [polynomial expansion degree] x [l2 regularization]
• Polynomial expansion is memory killer
• Degree 3 on 7 features results in 119 features
• Be careful with exploiting parallelism
13
To increase temporal locality - accesses to a data frame are clustered in time.
Polynomial expansion does not include constant column.
K-Fold
14
DF Persist, randomSplit map=> [([train_i], test)] map=>[(train, test)]
Spark SQL data frame
[([DF], DF)]
[(union[DF], DF)]
15
k-Fold
PE
Experiments
16
Spark 2.0.2 standalone mode
3 cores + 5GB mem exact copy of read-in file
http://spark.apache.org/docs/latest/cluster-overview.html
In total, we have 3 physical machines with 12GB mem + 8 cores.Driver - execute scala programWorker - execute tasksExecutor - each application runs a or more processes on a worker nodeJob - triggered by an actionTask - a unit of work executed on an executor, related with number of partitions >= number of blocks (128MB). If set manually, 2-4 partitions for each CPU in your cluster.Stage - a set of tasks
Local file - path + content on each worker node.
Performance test• ML Settings
• Logistic regression on HIGGS
• Train-test split, 70% + 30%
• Only 7 high level features were used
• Test unit 1 - 100 times full gradient descent + training error on training set, initial learning rate 0.001, l2 regularization 0.1
• Test unit 2 - compute confusion matrix on test set and make predictions
17
Performance and speedup curve
18
0
1.25
2.5
3.75
5
0
225
450
675
900
local 1 executor 2 executors 3 executors 4 executors 5 executors
training time (s) training-speed up
1
1.822
2.372
2.693
3.641
4.43
Running time vs. #executors (2 times average). Except for local, all tests have enough memory
Local mode does not have enough memory, causing data cannot be persist in memory. Thus, the running time is much higher.
Having more executors will reduce the running time linearly.
Grid search• 10% of original data, i.e., 1.1 million instances, 7 high level features only
• Grid
• Polynomial degrees - 1, 2, 3
• l2 regularization - 0, 0.001, 0.01, 0.1, 0.5
• 3-Fold cross validation
• 100 times gradient descent with initial learning rate 0.01
• 2 executors with 10GB mem + 5 cores each
• Result - 4400s training time, final test accuracy 62.4%
19
Confusion matrix: truePositive: 117605, trueNegative: 88664, falsePositive: 66529, falseNegative: 57786
Conclusion• Persist data - use more than once (incl. having branches)
• Change default cluster settings, e.g., executor memory per executor is 1GB
• Make use of Spark UI to find bottlenecks
• Using Spark builtin functions if possible
• Good examples for missing functions
• Don’t use accumulators in a transformation, except only need approximations
• Always start from small data to debug faster
• Future work - obey train-test split
20
Q&A• Thank you!
• Useful links
• Master - spark://ip:7077, e.g., spark://b2.lxd:7077
• Cluster - http://ip:8080/
• Spark UI - http://ip:4040/
• https://spark.apache.org/docs/latest/programming-guide.html
• http://spark.apache.org/docs/latest/submitting-applications.html, package a jar - sbt package
21
Backend slides
22
Training time vs. # executors
23
0
0.25
0.5
0.75
1
0
225
450
675
900
local 1 executor 2 executors 3 executors 4 executors 5 executors
training time (s) test accuracy
Spark UI
24
Jobs timeline
Spark UI
25
Executor summary
Numerical stability
26
