scala for machine learning

Scala for Machine Learning

Patrick NicolasDecember 2014

patricknicolas.blogspot.comwww.slideshare.net/pnicolas

What challenges?

Building scientific and machine learning applications requires ….

1. Clearly defined abstractions2. Flexible, dynamic models3. Scalable execution

What makes Scala particularly suitable to solvemachine learning and optimization problems?

... and may involve mathematician, data scientists,software engineers and dev. ops.

Scala tool box

Which elements in the Scala tool box are useful to meetthese challenges?

Actors

Composed futures

F-boundReactive

AbstractionNon-linear learning models <= functorial tensors

Kernel monadic composition <= monads

Extending library types <= implicits

Flexibility

Scalability

Low dimension features space (manifold) embedded into an observation space (Euclidean)

Abstraction: Non-linear learning models

Tensors

𝑓(𝑥, 𝑦, 𝑧)

𝛻𝑓 =𝜕𝑓

𝜕𝑥 𝑖 +𝜕𝑓

𝜕𝑦 𝑗 +𝜕𝑓

𝜕𝑧𝑘

Each type of tensors is a category, associated with a functor category.

• Field

• Vector field (contravariant)

• Inner product

• Covariant vector field (one-form/map)

• Tensor product ,exterior product , …

< 𝑣,𝑤 > = f

𝛼 𝑤 =< 𝑣,𝑤 >

𝑇𝑚𝑛⨂𝑇𝑝

𝑞𝑑𝑥𝑖 ∧ 𝑑𝑥𝑗

Tensor fields are geometric entities defining linear relation between vector fields, differential forms, scalars and other tensor fields


Machine learning consists of identifying a low dimension features space, manifold within an Euclidean observations space. Computation of smooth manifolds relies on tensorsand tensor metrics (Riemann, Laplace-Beltrami,…)

Problem: How to represent tensors and metrics?

Solution: Functorial representation of tensors, tensor products and differential forms.


One option is to define a vector field as a collection (i.e. List) and leverage the functor for the list.

Functor: f: U => V F(f): F(U) => F(V)


Convenient but incorrect…

Let’s define a generic vector field and covector fields types


Define a tensor as a Higher kind being either a vector or a co-vector accessed through type projection.

The functor for the vector field relies on the projection (Homfunctor) of 2 argument type functor Tensor on covariant and contravariant types.

Covariant Functor f: U => V F(f): F(U) => F(V)


Contravariant functors are used for morphisms or transformation on Covariant tensors (type CoVField)

Contravariant functor f: U => V F(f): F(V) => F(U)


Product Functor

Tensor metrics and products requires other type of functors …

BiFunctor

(*) Paul Phillips’ cats framework https://github.com/non/cats


Abstraction: Kernel monadic composition

Clustering or classifying observations entails computation of inner product of observations on the manifold

Kernel functions are commonly used in training to separate classes of observations with a linear decision boundary (hyperplane).

Problem: Building a model entails creating, composing and evaluating numerous kernels.

Solution: Define kernels as a 1st class programming concept with monadic operations.

Define a kernel function as the composition of 2 functions g o h

𝒦𝑓 𝐱, 𝐲 = 𝑔(

𝑖

ℎ(𝑥𝑖 , 𝑦𝑖))

Abstraction: Kernel monadic composition

We create a monad to generate any kind of kernel functions Kf, by composing their component g: g1 o g2 o … o gn o h

A monad extends a functor with binding method (flatMap)

The monadic implementation of the kernel function component h

Abstraction: Kernel functions composition

Declaration explicit kernel function

𝒦 𝐱, 𝐲 = 𝑒−12𝐱−𝐲𝜎

2

h: 𝑥, 𝑦 → 𝑥 − 𝑦 g: 𝑥 → 𝑒−1

2𝜎2( 𝑥)2

Polynomial kernel𝒦 𝐱, 𝐲 = (1 + 𝐱. 𝐲)𝑑 h: 𝑥, 𝑦 → 𝑥. 𝑦 g: 𝑥 → (1 + 𝑥)𝑑


Radius basis function kernel

Our monad is ready for composing any kind of explicit kernels on demand, using for-comprehension


Notes

• Quite often monads defines filtering capabilities (i.e. Scala collections).

• Accidently, the for-comprehension closure can be also used to create dynamic workflow


Abstraction: Extending library types

Scala libraries classes cannot always be sub-classed. Wrapping library component in a helper class clutters the design.

Implicit classes extends classes functionality without cluttering name spaces (alternative to type classes)

The purpose of reusability goes beyond refactoring code. It includes leveraging existing well understood concepts and semantic.

Data flow micro-router for successful and failed computation by transforming Try to Either with recovery and processing functions

scala.util.Try[T]

recover[U >: T](f: PartialFunction[Throwable, U]): Try[U]

getOrElse[U >: T](f: () => U): U

orElse[U :> T](f: () => Try[U]): Try[U]

toEither[U](rec: () => U)(f: T => T): Either[U, T]


.. as applied to a normalization problem.

4 lines of Scala code to extend Try with Either concept.


Notes


• Type conversion such as toDouble, toFloat can be extended to deal rounding error or rendering precision

• Creating a type class is a more generic (appropriate?) methodology to extends functionality of a closed model or framework. Is there a reason why Try in Scala standard library does not support conversion to Either ?

Abstractionnon-linear learning models <= functorial tensors



FlexibilityModeling <= Stackable traits

Scalability

Flexibility: modeling

Building machine learning apps requires configurable, dynamic workflows that preserve the model formalism

Leverage mixins, inheritance and abstract values to create models and weave data transformation.

Factory design patterns have been used to model dynamic systems (GoF). Are they adequate to model dynamic workflow?


Traditional programming languages compare unfavorably to scientific related language such as R because their inability to follow a strict mathematical formalism:

1. Variable declaration

2. Model definition

3. Instantiation

Scala stacked traits and abstract values preserve the core formalism of mathematical expressions.

𝑓 ∈ ℝ𝑛 → ℝ𝑛

𝑓 𝑥 = 𝑒𝑥

𝑔 ∈ ℝ𝑛 → ℝ

ℎ = 𝑔𝑜𝑓

g 𝒙 = 𝑖 𝑥𝑖

Declaration

Model

Instantiation


Multiple models and algorithms are typically evaluated by weaving computation tasks.

A learning platform is a framework that• Define computational tasks• Wires the tasks (data flow)• Deploys the tasks (*)

Overcome limitation of monadic composition (3 level of dynamic binding…)

(*) Actor-based deployment


Even the simplest workflow (model of data transformation) requires flexibility …..


Data scientists should be able to

1. Given the objective of the computation, select the best sequence of module/tasks (i.e. Modeling: Preprocessing + Training + Validating)

2. Given the profile of data input, select the best data transformation for each module (i.e. Data preprocessing: Kalman, DFT, Moving average….)

3. Given the computing platform, select the best implementation for each data transformation (i.e. Kalman: KalmanOnAkka, Spark…)


Implementation of Preprocessing module


Implementation of Preprocessing module using discrete Fourier

… and discrete Kalman filter


d

dPreprocessing

Loading

Reducing Training

Validating

Preprocessor

DFTFilter

Kalman

EM

PCA SVM

MLP

Reducer Supervisor

Clustering

Clustering workflow = preprocessing task -> Reducing task

Modeling workflow = preprocessing task -> model training task -> model validation

Modeling


A simple clustering workflow requires a preprocessor &reducer. The computation sequence exec transform a time series of element of type U and return a time series of type W as option


A model is created by processing the original time series of type TS[T] through a preprocessor, a training supervisor and a validator


Putting all together for a conditional path execution …


1

AbstractionNon-linear learning models <= functorial tensors



FlexibilityModeling <= Stackable traits

ScalabilityDynamic programming <= tail recursion

Online processing <= streams

Data flow control <= back-pressure strategy

Scalability: dynamic programming

Many machine learning algorithms (HMM,RL, EM, MLP, …) relies on dynamic programming techniques

Tail recursion is very efficient solution because it avoids the creation of new stack frames

Choosing between iterative and recursive implementation of algorithms is a well-documented dilemma.

Viterbi algorithm for hidden Markov Models

The objective is to find the most likely sequence of states {qt} given a set of observations {Ot} and a λ-model


The algorithm recurses along the observations with N different states.


Relative performance of the recursion w/o tail elimination for the Viterbi algorithm given the number of observations


Scalability: online processing

Some problems lend themselves to process very large data sets of unknown size for which the execution may have to be aborted or re-applied

Streams reduce memory consumption by allocating and releasing chunk of data (or slice or time series) while allowing reuse of intermediate results.

An increasing number of algorithms such as reinforcement training relies on online (or on-demand) training.

X0 X1 ….... Xn ………. Xm

Data stream

1

2𝑚 𝑦𝑛 − 𝑓 𝒘|𝑥𝑛

2+ 𝜆 𝒘 2

Garbage collector

Allocate slice .take

Release slice .drop

Heap

Traversal loss function


The large data set is converted into a stream then broken down into manageable slices. The slices are instantiated, processed (i.e. loss function) and released back to the garbage collector, one at the time

Slices of NOBS observations are allocated one at the time, (.take) processed, then released (.drop) at the time.


The reference streamRef has to be weak, in order to have the slices garbage collected. Otherwise the memory consumption increases with each new batch of data.

(*) Alternatives: define strmRef as a def or use StreamIterator


Comparing list, stream and stream with weak references.


Operating zone

Notes:

Iterators: • computation cannot not memoized. (“Iterators are the

imperative version of streams”)• One element at a time• Non-recursive (tail elimination)

Views:• No intermediate results preserved• One element at a time

Stream iterators: • Lazy tails


The execution of workflow may create a stream bottleneck, for slow tasks and overflow local buffers.

A flow control mechanism handling back pressure on bounded mail boxes of upstream actors.

Actors provides a very efficient and reliable way to deploy workflows and tasks over a large number of cores and hosts.

Scalability: flow control


Actor-based workflow has to consider• Cascading failures => supervision strategy• Cascading bottleneck => Mailbox back-pressure strategy

Workers

Router, Dispatcher, …

Akka has reliable mechanism to handle failure. What about temporary disruptions?


Messages passing scheme to process various data streams with transformations.

Dataset

Workers

Controller

Watcher

Load->

Compute->

Bounded mailboxes

<- GetStatus

Status ->

Completed->

Worker actors processes data chunk msg.xt sent by the

Controller with the transformation msg.fct

Message sent by collector to trigger computation


Watcher actor monitors messages queues report to collector with

Status message.

GetStatus message sent by the collector has no payload


Controller creates the workers, bounded mailbox for each worker actor (msgQueues) and the watcher actor.


The Controller loads the data sets per chunk upon receiving the message Load from the main program. It processes the results of the computation from the worker (Completed) and throttle the input to workers for each Status message.


The Load message is implemented as a loop that create data chunk which size is adjusted according to the load computed by the watcher and forwarded to the controller, Status


Simple throttle increases/decreases size of the batch of observations given the current load and specified watermark.


Selecting faster/slower and less/more accurate version of algorithm can also be used in the regulation strategy

Feedback control loop adjusts the size of the batches given the load in mail boxes and complexity of the computation


• Feedback control loop should be smoothed (moving average, Kalman…)

• A larger variety of data flow control actions such as adding more workers, increasing queue capacity, …

• The watch dog should handle dead letters, in case of a failure of the feedback control or the workers.

• Reactive streams introduced in Akka 2.2+ has a sophisticated TCP-based propagation and back pressure control flows

Notes


… and there is more

There are many other Scala programming language constructs I found particularly intriguing as far as for machine learning is concerned …

Reactive streams (TCP)

Domain Specific Language

Emulate ‘R’ language for scientists to use the application.

Effective fault-tolerance & flow control mechanism

Delimited continuationSave, restore, reuse computation states

Donate to Apache software and Eclipse foundations

Monads are Elephants J. Ivy –james-iry.blogspot.com/2007/10/monads-are-elephans-part2.html

Extending the Cake pattern: Dependency injection in Scala A. Warski –www.warski.org/blog/2010/12/di-in-scala-cake-pattern

Programming in Scala $12.5 Traits as stackable modification M. Odersky, M. Spoon, L. Venners - Artima 2008

Introducing Akka J. Boner - Typesafe 2012www.slideshare.net/jboner/introducing-akka

Scala in Machine Learning: $1 Getting started P. Nicolas –Packt publishing 2014

Exploring Akka Stream’s TCP Back Pressure: U. Peter – Xebia 2015blog.xebia.com/2015/01/14/exploring-akka-streams-tcp-back-pressure/

References

Cats functional library P. Phillips – https://github.com/non/cats

scala for machine learning

Data & Analytics

fv abstraction

fu abstraction

comprehension abstraction

comnoncats abstraction

manifold kernel functions

polynomial kernel

kind of kernel functions

dynamic workflow abstraction