real time analytics using cloudera impala in manufacturing use case
DESCRIPTION
CSCI E-185 Big Data Analytics -- Final project, Fall 2013TRANSCRIPT
Final Project
Real Time Analyticsusing Cloudera Impala in Manufacturing use case
Rapheephan Thongkham-uan (Nancy)CSCI E-185 Big Data Analytics
@Rapheephan Thongkham-Uan
Friday, May 10, 13
To make Big Data makes MoneyIn manufacturing, ...
• We want to improve the supply chain management by tracking the defective parts, finding the bottlenecks, etc.
• We are doing the analysis on the big amount of data using traditional tools that takes too much time.
• People in the factory are familiar to SQL query.
• The faster we analyze the big data,
- faster defects/bottlenecks detection
- near real-time problem solving, decision-making
- less time and money spending on the defects
That’s why we need Cloudera Impala
@Rapheephan Thongkham-Uan
Friday, May 10, 13
Requirements
• Cloudera Manager 4.5.2 installation guide
- http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Free/latest/Cloudera-Manager-Free-Edition-Installation-Guide/Cloudera-Manager-Free-Edition-Installation-Guide.html
• My VM
- Ubuntu 12.04 (Precise) 64-bits
- CDH 4.2
- Cloudera Management 4.5.2
• I installed Impala via Cloudera Manager
@Rapheephan Thongkham-Uan
Friday, May 10, 13
After finishing cloudera manager installation
@Rapheephan Thongkham-Uan
Friday, May 10, 13
We will use Hue Web UI to query Impala
From the Services menu bar, click
HUE1and choose Hue
Web UI
@Rapheephan Thongkham-Uan
Friday, May 10, 13
Create table in HiveCreate Hive table with user impala then load the data from local into the table
$ sudo -E -u impala hive -e “CREATE TABLE khsample (id INT, sdate STRING, seq INT, product STRING, ope STRING, resource_grp STRING, resource STRING, inflow FLOAT, proclot FLOAT, wip FLOAT, ope_rate FLOAT) ROW FORMAT DELIMITED FILEDS TERMINATED BY ‘,’;”
$ sudo -E -u impala hive -e “LOAD DATA LOCAL INPATH ‘KH_RESULT.csv’ INTO TABLE khsample;”
@Rapheephan Thongkham-Uan
Friday, May 10, 13
Sample table in Hue Web UI
We can view the table we just created in Hive shell on Hue Web UI
*the input data is included japanese characters which cannot be read.
@Rapheephan Thongkham-Uan
Friday, May 10, 13
Create table in HiveBefore querying Impala on Hue Web UI, we have to refresh the Impala first. In the Impala-shell, input the following command
$ impala-shell
[impala-server:21000] > refresh;
@Rapheephan Thongkham-Uan
Friday, May 10, 13
Query in Impala
In Hue Web UI, click Impala icon the query editor page will be shown.
input the query and execute
@Rapheephan Thongkham-Uan
Friday, May 10, 13
Bottlenecks query
- To find the groups of machines which are the bottlenecks, we can calculate from WIP by day. The group of machines which WIP value is higher than the day before can be predicted as bottleneck.
- The simulation dates were from 12/13 to 12/22. I will get the summation of WIP values from the sampling dates (12/14, 12/16, 12/18, 12/20, 12/22).
- We have to do 5 sub-queries in FROM statement.
@Rapheephan Thongkham-Uan
Friday, May 10, 13
Bottlenecks query (2)
SELECT A.resource_grp,
A.awip as wip22, --12/22 wip
B.bwip as wip20, --12/20 wip
C.cwip as wip18, --12/18 wip
D.dwip as wip16, --12/16 wip
D.dwip as wip14 --12/14 wip
FROM (SELECT resource_grp, sum(wip) as awip
FROM khsample
WHERE id = 118 and sdate =’”2012/12/22”’) A join
(SELECT resource_grp, sum(wip) as bwip
FROM khsample
WHERE id = 118 and sdate =’”2012/12/20”’) B join
(SELECT resource_grp, sum(wip) as cwip
FROM khsample
WHERE id = 118 and sdate =’”2012/12/18”’) C join
@Rapheephan Thongkham-Uan
(SELECT resource_grp, sum(wip) as dwip
FROM khsample
WHERE id = 118 and sdate =’”2012/12/16”’) D join
(SELECT resource_grp, sum(wip) as ewip
FROM khsample
WHERE id = 118 and sdate =’”2012/12/14”’) E
WHERE A.resource_grp = B.resource_grp
and A.resource_grp = C.resource_grp
and A.resource_grp = D.resource_grp
and A.resource_grp = E.resource_grp
and A.awip >= B.bwip and B.bwip >= C.cwip
and C.cwip >= D.dwip and D.dwip >= E.ewip
ORDER BY A.awip DESC
LIMIT 20;
Friday, May 10, 13
Comparing the result of Impala with Oracle SQL
@Rapheephan Thongkham-Uan
Friday, May 10, 13
Results
• join 5 sub-queries in Oracle SQL took 50s.
• join 5 sub-queries in Impala took 6.67s.
• Impala can query 7x faster with the same results.
• In the real use, we could configure Impala to work with HBase, also change Hive Metastore to OracleDB.
@Rapheephan Thongkham-Uan
Friday, May 10, 13