machine learning for java developers
TRANSCRIPT
Machine Learning for Java Developers
Nirmal FernandoWSO2 Inc.
{Java Colombo}
Few things about me...
● Associated Technical Lead at WSO2● Team Lead of WSO2 Machine Learner● Just completed 4th year in the industry● Graduated from Department of Computer Science, University
of Moratuwa.● Schooled at St. Sebastian’s College, Moratuwa.● Can sing a bit :-)
https://goo.gl/qbAXLz
Predictive Analytics
Extract information from existing datasets to determine patterns and predict future outcomes and trends.
It does not tell you what will happen in the future.
But forecasts what might happen in the future with an acceptable level of reliability. source: http://insidebigdata.com/2014/08/25/salespredict-
marketo-partner-using-predictive-analytics/
Predictive Analytics
“Big Data Predictive Analytics” Forrester Research report is the
second most read Forrester report in Q3, 2015
https://www.forrester.com
Predictive Analytics - Use cases
http://californialoanfind.com/what-and-who-is-teletrack/
Predictive Analytics - Use cases
http://www.chrisdunn.com/
Machine Learning
Field of study that gives computers
the ability to learn
without being explicitly
programmed.
- Arthur Samuel (1959)
Machine Learning - Pipeline
Machine Learning - Terminology
● Input data must be in tabular format ● Each row is called a data point ● Each column is called a feature ● Value you are going to predict is called the “response
variable”
● Next value prediction
● Classification
● Clustering
● Recommendations
etc…
Machine Learning - What type of a problem?
Next value prediction
Example of linear regression on one independent variable
Predicting a discrete value
Classification
Grouping similar data points
together.
Clustering
Seek to predict preferences a user
would give to an item/product.
Recommendations
● Supervised learning
● Unsupervised learning
● Reinforcement learning
Machine Learning - Which algorithm category?
Supervised vs Unsupervised
Supervised Learning Algorithms
Regression Classification
Linear RegressionLasso RegressionRidge Regression
Logistic RegressionSupport Vector Machine (SVM)Decision TreeRandom ForestNaive BayesBayesian Network
Unsupervised Learning Algorithms
Clustering
K-meansK-mediansHierarchical Clustering….
Java tools for Machine Learning
Tool License URL
Weka GNU General Public License
http://www.cs.waikato.ac.nz/ml/weka/
JSAT GPL v3 https://github.com/EdwardRaff/JSAT
Mahout Apache v2 https://mahout.apache.org/
Spark MLlib Apache v2 http://spark.apache.org/mllib/
Speed
Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk.
Ease of Use
Write applications quickly in Java, Scala, Python, R.
Easy to Deploy
Runs on existing Hadoop clusters and data.
Apache Spark MLlib - scalable machine learning library
SparkConf - Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.
SparkContext / JavaSparkContext - Main entry point for Spark functionality. A SparkContext represents the connection to a Spark cluster. Only one SparkContext may active per JVM.
RDD / JavaRDD - A Resilient Distributed Dataset (RDD), the basic abstraction in Spark. Represents an immutable, partitioned collection of elements that can be operated in parallel.
Apache Spark - few terms
Filter - Return a new dataset formed by selecting those elements of the source on which function returns true.
Map - Return a new distributed dataset formed by passing each element of the source through a function.
Random Split - Split a dataset randomly based on a given ratio.
Cache - Persisting (or caching) a dataset in memory across operations.
Apache Spark - few operations on a RDD
● Dataset
Pima Indian diabetes dataset
https://archive.ics.uci.edu/ml/datasets/Pima+Indians+Diabetes
Number of instances : 768
Number of features : 8
Let’s solve a classification problem using Apache Spark
● Response variable
Name : class
Values : 0 or 1
Interpretation : Whether a given Pima Indian has diabetes or not
Let’s solve a classification problem using Apache Spark
● Objective
Build a classification model to predict whether a given Pima Indian has diabetes or not.
Let’s try to build a Logistic Regression model for this.
Let’s solve a classification problem using Apache Spark
Code:
https://github.com/nirmal070125/ml-java-meetup
Solution using Apache Spark
Powered by Apache Spark and Apache Spark MLlib.
● Manage and explore your data ● Analyze the data using machine learning algorithms● Build machine learning models● Compare and manage generated machine learning models● Predict using the built models● Use the built models with WSO2 CEP and WSO2 ESB.
http://wso2.com/products/machine-learner/
Few words on WSO2 Machine Learner