data science in action - college of william &...
TRANSCRIPT
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science in Action
Ji Li
Data Scientist
March 25, 2015
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Overview
Table of Contents
1 What is Data ScienceOverviewData Science Workflow
2 Churn ModelBusiness UnderstandingDemo in RTop FeaturesPrescriptive Analysis
3 Yesware Email AnalysisEmail Analysis
4 Data Science in the IndustryData Science ToolboxBig Data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Overview
Data science on Google search
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Overview
Data science from my point of view
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Table of Contents
1 What is Data ScienceOverviewData Science Workflow
2 Churn ModelBusiness UnderstandingDemo in RTop FeaturesPrescriptive Analysis
3 Yesware Email AnalysisEmail Analysis
4 Data Science in the IndustryData Science ToolboxBig Data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Domain expertise guides the workflow
Data Science Workflow
Data Munging
Data Mining
Delivery of actionable Insights
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Data Munging
Data Munging
Data Munging means some or allof the following tasks:
ETL
Data Integration
Data Cleansing
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Data Munging
Data Munging
Data Munging means some or allof the following tasks:
ETL
Data Integration
Data Cleansing
ETLThe process of extract,transform, and load data.
To acquire data fromexternal sources.
To migrate multiple datasources internally.
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Data Munging
Data Munging
Data Munging means some or allof the following tasks:
ETL
Data Integration
Data Cleansing
Data Integration
To combine data from disparatesources into meaningful andvaluable information.
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Data Munging
Data Munging
Data Munging means some or allof the following tasks:
ETL
Data Integration
Data Cleansing
Data Cleansing
Data cleansing, also called datascrubbing, is the process ofamending or removing data in adatabase that is incorrect,incomplete, improperlyformatted, or duplicated.
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Data Mining
Data Mining
Data Mining is the key step toturn data into insights:
Data Exploration
Machine Learning
Model Evaluation
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Data Mining
Data Mining
Data Mining is the key step toturn data into insights:
Data Exploration
Machine Learning
Model Evaluation
Data Exploration
The process of visually examineand explore the data.
To gain basic understandingof the data.To identify relationshipsbetween different attributes.To answer basic questionsusing data.
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Data Mining
Data Mining
Data Mining is the key step toturn data into insights:
Data Exploration
Machine Learning
Model Evaluation
Machine Learning
To obtain statistical models, weusually need to go throughmultiple steps like the following:
1 Construct new features.2 Remove redundant features.3 Choose one or more suitable
machine learning algorithm.
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Data Mining
Data Mining
Data Mining is the key step toturn data into insights:
Data Exploration
Machine Learning
Model Evaluation
Model EvaluationModel evaluation is often usednot only to select the best modelfrom the set of models, but alsoto get ready for producingactionable insights.
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Skills of a data scientist
Data Science skills
Programming
Statistics
Visualization
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Workflow
Doing data science
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Business Understanding
Table of Contents
1 What is Data ScienceOverviewData Science Workflow
2 Churn ModelBusiness UnderstandingDemo in RTop FeaturesPrescriptive Analysis
3 Yesware Email AnalysisEmail Analysis
4 Data Science in the IndustryData Science ToolboxBig Data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Business Understanding
The Goal
GoalTo predict which customers will churn.To find ways to prevent customers from churning.
Workflow
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Business Understanding
Customer renewal data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Business Understanding
Customer service call data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Business Understanding
Customer plan type and demographic
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Business Understanding
Customer usage data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Business Understanding
Churn data structure
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Demo in R
Table of Contents
1 What is Data ScienceOverviewData Science Workflow
2 Churn ModelBusiness UnderstandingDemo in RTop FeaturesPrescriptive Analysis
3 Yesware Email AnalysisEmail Analysis
4 Data Science in the IndustryData Science ToolboxBig Data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Demo in R
R Script
http://goo.gl/IV3HDs
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Top Features
Table of Contents
1 What is Data ScienceOverviewData Science Workflow
2 Churn ModelBusiness UnderstandingDemo in RTop FeaturesPrescriptive Analysis
3 Yesware Email AnalysisEmail Analysis
4 Data Science in the IndustryData Science ToolboxBig Data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Top Features
Variable importance from random forest
Top Features
cust surv calls
is intl plan
day mins
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Top Features
Top feature - customer service calls
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Top Features
Top feature - customer service calls
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Top Features
Top feature - customer day time minutes
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Top Features
Top feature - international plan and international calls
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Prescriptive Analysis
Table of Contents
1 What is Data ScienceOverviewData Science Workflow
2 Churn ModelBusiness UnderstandingDemo in RTop FeaturesPrescriptive Analysis
3 Yesware Email AnalysisEmail Analysis
4 Data Science in the IndustryData Science ToolboxBig Data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Prescriptive Analysis
Churn analysis
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Prescriptive Analysis
Churn analysis
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Email Analysis
Table of Contents
1 What is Data ScienceOverviewData Science Workflow
2 Churn ModelBusiness UnderstandingDemo in RTop FeaturesPrescriptive Analysis
3 Yesware Email AnalysisEmail Analysis
4 Data Science in the IndustryData Science ToolboxBig Data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Email Analysis
When to send an email
To Optimize Email Reply
Yesware users want to know how to get more replies.
Emails Sent Volume and Reply Rate by Sent HourEmails Sent Volume and Reply Rate by Sent Weekday
Best time to send email: http://goo.gl/lVdD31
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Email Analysis
When to send an email
To Optimize Email Reply
Yesware users want to know how to get more replies.
Emails Sent Volume and Reply Rate by Sent HourEmails Sent Volume and Reply Rate by Sent Weekday
An Email Reply Model1 Construct features from the email data.2 Create a model to predict the reply on each email.3 Identify some top features that contributed most to the reply:
Sent HourSent Weekday
Best time to send email: http://goo.gl/lVdD31
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Email Analysis
When to send an email
To Optimize Email Reply
Yesware users want to know how to get more replies.Emails Sent Volume and Reply Rate by Sent Hour
Emails Sent Volume and Reply Rate by Sent Weekday
Best time to send email: http://goo.gl/lVdD31
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Email Analysis
When to send an email
To Optimize Email Reply
Yesware users want to know how to get more replies.Emails Sent Volume and Reply Rate by Sent HourEmails Sent Volume and Reply Rate by Sent Weekday
Best time to send email: http://goo.gl/lVdD31
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Email Analysis
D3 Visualization: Subject Line Keywords
Subject line keywords: http://goo.gl/PK9xhO
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Toolbox
Table of Contents
1 What is Data ScienceOverviewData Science Workflow
2 Churn ModelBusiness UnderstandingDemo in RTop FeaturesPrescriptive Analysis
3 Yesware Email AnalysisEmail Analysis
4 Data Science in the IndustryData Science ToolboxBig Data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Toolbox
Why do we need a toolbox?
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Toolbox
Tools that helped me do data science fast
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Data Science Toolbox
Data visualization tools
D3: http://d3js.org/
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Big Data
Table of Contents
1 What is Data ScienceOverviewData Science Workflow
2 Churn ModelBusiness UnderstandingDemo in RTop FeaturesPrescriptive Analysis
3 Yesware Email AnalysisEmail Analysis
4 Data Science in the IndustryData Science ToolboxBig Data
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Big Data
Big Data
Big Data Ecosystem
Hadoop - file system
Spark - computing systemSpark Stack
Spark SQL - Data MungingSpark Streaming - RealTime ProcessingMLlib - Machine LearningGraphX - Visualization
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Big Data
Big Data
Big Data Ecosystem
Hadoop - file systemSpark - computing system
Spark Stack
Spark SQL - Data MungingSpark Streaming - RealTime ProcessingMLlib - Machine LearningGraphX - Visualization
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Big Data
Big Data
Big Data Ecosystem
Hadoop - file systemSpark - computing systemSpark Stack
Spark SQL - Data MungingSpark Streaming - RealTime ProcessingMLlib - Machine LearningGraphX - Visualization
https://spark.apache.org/
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Big Data
Big Data
Big Data Ecosystem
Hadoop - file systemSpark - computing systemSpark Stack
Spark SQL - Data MungingSpark Streaming - RealTime ProcessingMLlib - Machine LearningGraphX - Visualization
https://spark.apache.org/
Data Science In Action Ji Li
What is Data Science Churn Model Yesware Email Analysis Data Science in the Industry
Big Data
Thank You
Please send your questions and feedback to me!
Data Science In Action Ji Li