analysis of economic data using big data
TRANSCRIPT
![Page 1: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/1.jpg)
Analysis of Economic Data Using Bigdata
Presented BySHIVUMANJESH P
[4JC13MCA51] VI SEM MCA
SJCE
Internal Guide C J HARSHITHA Assistant Professor
Dept. Of MCA SJCE
External Guide Imran basha Senior Consultant
Snipe IT Solutions
JSS MAHAVIDYAPEETHA SRI JAYACHAMARAJENDRA COLLEGE OF ENGINEERING MYSURU-570006
AN AUTONOMOUS INSTITUTE AFFILIATED TOVISVESVARAYA TECHNOLOGICAL UNIVERSITY, BELGAVI.
Presentation on
![Page 2: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/2.jpg)
Problem Definition1. Inflation is rising as a serious threat for countries
development.
2. Unscientific farming
![Page 3: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/3.jpg)
3. Big-picture problem, economic indicators and decision makers rely on the native economic transactions and on the data records.
![Page 4: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/4.jpg)
Objective
To examine economic data and recording the increasing and decreasing vegetables and food items prices year to year.
Preferring the fresh and edible food products and to overcome various problems of deficiency and malnutrition.
To maintain the continuous connectivity between Demand-Supply Chain
•
![Page 5: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/5.jpg)
Scope of The Project• The Economic data analysis make an immense impact on E-
commerce and also builds a potential to the business activities and also in the investments
• The analysis is limited to the particular products and can be future extended based on the requirements and developments.
• The big data analysis can be presented using the android application by providing simple and smart user interfaces about products they use in the daily life
• It requires high end specification of the system on which it is implementing, dealing with large data set with diversified features and functionalities.
![Page 6: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/6.jpg)
User characteristics
• The system will provide a very precise and simple platform to the respective users.
• The admin will provide the access to the developer as well as to the user and provides data sets.
• The developer collects the data sets clusters the data based on the particular criteria and analyze the behavior of the data elements.
• The user gets the desired result by firing a query.
![Page 7: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/7.jpg)
General constraints• The big data usage is efficient for large data sets and it is
not suitable for data with less volume.
• Since the main objective is based on data analysis user interface section is given least priority.
• Sometimes it may find tedious to deal with complete unstructured data items.
• The data which is obtained from the various source may not be of same parameters
![Page 8: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/8.jpg)
Functional Requirements Storage • Hadoop Distributed File System is designed for storing very large
files with streaming data access patterns, running on clusters of commodity hardware.
• The economic data is a highly diversified data set which is both large and variety in nature.
• A dataset is typically generated or copied from source, and then various analyses are performed on that dataset over time.
• Applications that require low-latency access to data, in the tens of milliseconds range, will not work well with HDFS.
![Page 9: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/9.jpg)
Computation• MapReduce is a processing technique that allows for
massive scalability across hundreds or thousands of servers in a Hadoop cluster.
• The MapReduce algorithm contains two important tasks, namely Map and Reduce.
• This algorithm in economic data analysis helps in finding the demand for the particular goods based on certain key words.
• The shuffle and sort process is dependent mainly on volume of the data sets.
![Page 10: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/10.jpg)
Performance Requirements• The major aim for choosing the domain of big data for
economic analysis is for the velocity criteria of data processing.
• Connecting of the commodity systems and forming the node between them helps in quick retrieval of the data items.
• There is a vast development of flexibility in distributed system environment.
![Page 11: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/11.jpg)
• Hardware Requirements
Processor : Core i3 onwards RAM : 4GB + Hard disk space : 40GB +
• Software Requirements
Technology : Hadoop Tools : Apache Hive Apache Pig
Apache SqoopApache oozie
R Studio Operating System : Linux
![Page 12: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/12.jpg)
System Architecture
![Page 13: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/13.jpg)
Class Diagram
![Page 14: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/14.jpg)
Algorithm Design
![Page 15: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/15.jpg)
Dataflow Diagram
![Page 16: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/16.jpg)
Level 1 DFD
![Page 17: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/17.jpg)
Activity Diagram
![Page 18: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/18.jpg)
Use case Diagram – Admin & User
![Page 19: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/19.jpg)
Use case Diagram-Developer
![Page 20: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/20.jpg)
Sequence Diagram
![Page 21: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/21.jpg)
Requirements
![Page 22: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/22.jpg)
Unstructured Datasets
![Page 23: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/23.jpg)
Structured Datasets
![Page 24: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/24.jpg)
System Implementation
![Page 25: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/25.jpg)
![Page 26: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/26.jpg)
![Page 27: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/27.jpg)
![Page 28: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/28.jpg)
![Page 29: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/29.jpg)
![Page 30: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/30.jpg)
![Page 31: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/31.jpg)
![Page 32: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/32.jpg)
![Page 33: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/33.jpg)
![Page 34: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/34.jpg)
![Page 35: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/35.jpg)
![Page 36: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/36.jpg)
R Environment
![Page 37: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/37.jpg)
![Page 38: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/38.jpg)
![Page 39: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/39.jpg)
Experimental Results
![Page 40: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/40.jpg)
![Page 41: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/41.jpg)
![Page 42: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/42.jpg)
![Page 43: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/43.jpg)
![Page 44: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/44.jpg)
![Page 45: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/45.jpg)
![Page 46: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/46.jpg)
![Page 47: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/47.jpg)
![Page 48: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/48.jpg)
![Page 49: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/49.jpg)
![Page 50: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/50.jpg)
![Page 51: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/51.jpg)
![Page 52: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/52.jpg)
![Page 53: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/53.jpg)
Test Cases
![Page 54: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/54.jpg)
Testcase no Testcase Discription
Required input
Expected output
Actual output
Test pass/fail
#TC 01 Verification of the nodes
Command to start Hadoop nodes (Start-
all.sh)
All nodes should start
All nodes are present
P
#TC 02 Verification of Hive Installation
Command Hive version
It should return Installed Hive
Hive Version is returned
P
#TC 03 Verification of Pig Installation
Command to start Pig (/opt/pig)
It should return grunt shell
grunt shell is returned
P
#TC 04 Verification of Sqoop
Installation
Command Sqoop version
It should return Installed Sqoop
Sqoop Version is returned
P
#TC 05 Verification of Data Imported to
HDFS from RDBMS
Entering to Hadoop file system from
local file system
Imported data should be
present in HDFS
Imported data is present in HDFS
P
#TC 06 Validating user Query
Entering Query Valid query should be entered
Valid query is entered
P
#TC 07 Testing the processed data
Post Query Processed Data should be correct
Processed Data should be valid
P
#TC 08 Importing the processed data to
R
Import Dataset Processed data should be imported
Processed Data is imported
P
![Page 55: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/55.jpg)
#TC 09 Mapping of processed dataset
Barplot() Processed dataset should
be mapped correctly
Processed data is mapped correctly
P
#TC 10 Mapping in Pie chart
Pie() Processed data should be mapped in
percent
Processed data is not mapped with percent
F
#TC 11
Retrieving the
result Less than 5 seconds
Dump()
Result should be displayed
within 5 seconds
Results is displaying more than 5 seconds
F
#TC 12
Plotting the
values obtained in R
Plot()
All the values
should be obtained
Some values are missing
F
![Page 56: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/56.jpg)
Conclusion• The statistical analysis is carried out for fruits and
vegetables from the 1970-2013
• The major Requirements is based on the context of inflation problem
• The analysis is done mainly on product based and Year based
• This analysis serves as a vital input for machine learning mechanism
![Page 57: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/57.jpg)
Future Enhancements• The analysis can be extended further on the food grains
• The enterprise application can be build by embedding a search engine which will be helpful for end user
• The data sets can be tuned which may leads in deriving of some other requirements of different paradigm
• The graphical representation can be changed further by displaying of accurate value rather than range
![Page 58: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/58.jpg)
Company Details
Company Name : Snipe IT Solutions
Address : # 123, 3rd floor, 70th Cross, 5th Block, Rajajinagar Nagar, Bengaluru.
External guide : Imran basha Senior Consultant
Snipe IT Solutions
Email : [email protected] Ph no : 9590071811
![Page 59: Analysis of economic data using big data](https://reader033.vdocuments.mx/reader033/viewer/2022051404/58f074f61a28abd30c8b4643/html5/thumbnails/59.jpg)
Thank You