Upload File

big data & hadoop by mr.nataraj smallest unit is bit 1 byte=8 bits 1 kb (kilo byte)= 1024 bytes...

  • Home
  • Documents
  • Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits
34

Upload: elinor-tyler

Post on 19-Dec-2015

239 views

Category:

Documents


3 download

Report

Tags:

TRANSCRIPT

Page 1: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits
Page 2: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

Big Data & HadoopBy Mr.Nataraj

Page 3: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

smallest unit is bit1 byte=8 bits1 KB (Kilo Byte) = 1024 bytes =1024*8 bits1MB (Mega Byte) =1024 KB =(1024)^2 * 8 bits1 GB (Giga Byte) =1024 MB =(1024)^3 * 8 bits1 TB (Tera Byte) =1024GB =(1024)^4 * 8 bits1 PB (Peta Byte) =1024 TB =(1024)^5 * 8 bits1 EB (Exa Byte) =1024 PB =(1024)^6 * 8 bits1 ZB (Zetta Byte) =1024 EB =(1024)^7 * 8 bits1 YB (Yotta Byte) =1024 ZB =(1024)^8 * 8 bits1 XB (Xenotta Byte) =1024 YB =(1024)^9 * 8 bits

UNITS OF DATA

Page 4: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

1 byte =A single character1 KB = A very short story1 MB=A small novel (6 seconds of TV-quality video)1 Gigabyte: A pickup truck filled with paper1 Terabyte : 50000 trees made into paper2 PB: All US academic research libraries5 EB: All words ever spoken by human beings

HOW BIG ARE THOSE NUMBERS

Page 5: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

WHAT IS BIG DATA

Page 6: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

SOME INTERESTING FACTS• Google: 20,00,000 query per second• Facebook 34000 likes per minute• Online Shopping of USD 300,000 per minute• 1,00,000 tweets in twitter per minute• 600 new videos are uploaded per minute in yT• Barack Obama used Big Data to win election• Driver-less cars uses Big Data Processing for

driving vehicles

Page 7: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

• AT&T transfers about 30 petabytes of data through its networks each day

• Google processed about 24 petabytes of data per day in 2009

• The 2009 movie Avatar is reported to have taken over 1 petabyte of local storage at Weta Digital for the rendering of the 3D CGI effects

http://en.wikipedia.org/wiki/AT&T_Inc.
http://en.wikipedia.org/wiki/Google
http://en.wikipedia.org/wiki/Avatar_(2009_film)
http://en.wikipedia.org/wiki/Weta_Digital
http://en.wikipedia.org/wiki/Weta_Digital
Page 8: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

As of January 2013, Facebook users had uploaded over 240 billion photos, with 350 million new photos every day. For each uploaded photo, Facebook generates and stores four images of different sizes, which translated to a total of 960 billion images and an estimated 357 petabytes of storage

Processing capabiltiyGoogle process 20 PB a dayFacebook 2.5 PB of User data + 15 TB/dayebay 6.5 PB of data +50TB/day

Page 9: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

Evolution of Hadoop• Doug Cutting working on Lucene

Project(A Search engine to search document)got problem of Storage and computation, was looking for distributed Processing.

• Google publish a Paper GFS(Google File System)

• Doug cutting & Michael Cafarella implemented GFS to come out with Hadoop

Page 10: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

WHAT IS HADOOP• A framework written in Java for running

applications on large clusters of commodity hardware.

• Mainly contains 2 parts– HDFS for Storing data– Map-Reduce for processing data

• Maintains fault-tolerant using replication factor.

Page 11: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

• employee.txt(eno,ename,empAge,empSal,empDes)• 101,prasad,t20,1000,lead

Page 12: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

Assume you have around 100,00,000,00000,0000000000 records and you would like to find out all the employees above 60 years of age.How do you program them traditionally. 10 GB= 10 min 1 TB= 1000 minutes =16 hoursGoogle process 20 PB of data per dayTo process 20 PB it will take 3200 hours = 133 days

INSPIRATION FOR HADOOPTo store huge data(unlimited)To process huge data

Page 13: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

• Node -A single computer with its own processor and memory.

• Cluster-combination of nodes as a single unit• Commodity Hardware-cheap non-reliable

hardware• Replication Factor-data getting duplicated &

saved in more than one place• Data Local Optimization-data will be processed

locally

Basic Terminology

Page 14: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

• Block:- A part of data• node1 node2 node3• data1 data2 data3• 1 file 200 MB(50MB 50MB 50MB 50MB)

• Block size:- The size of data that can stored as a single unit

• apache hadoop:- 64 MB(configurable)• 1GB in apache hadoop=16 blocks• 65MB(apache)=64MB+ 1MB• Replication:- duplicate the data replication factor is: 3

Page 15: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

NODE CLUSTER

Page 16: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

SCALING• Vertical Scaling

Adding more powerful hardware to an existing system.Will Scale only up to certain limit.

• Horizontal Scaling Adding a completely new node to an existing

cluster. will scale up to many nodes

Page 17: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

3 V's of Hadoop• Volume: The amount of data generated• Variety: structured data,unstructed

data.Database table data• Velocity: The frequency at which data is

generated

Page 18: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

1.hadoop believes on scale out instead of scale up when needed buy more oxes dont grow your oxe more powerful

2.hadoop on structured as well unstructured RDBMS only works with structured data.(However now a days many no-sql database has comeout in the market like mongo db,couch base.)

3.hadoop believes on key-value pair rather than data in the column

HADOOP VS RDMS

Page 19: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

No doubt Hadoop is a framework for processing big data. But it is not the only framework to do so. Below are few more alternative.

Apache SparkGraphLabHPCC Systems- (High Performance Computing Cluster)

DryadStratosphere

HADOOP ALTERNATIVES

Storm R3 Disco Phoenix Plasma

http://spark.incubator.apache.org/
http://graphlab.org/
http://hpccsystems.com/
http://research.microsoft.com/en-us/projects/Dryad/
http://www.stratosphere.eu/
http://storm-project.net/
http://heynemann.github.io/r3/
http://discoproject.org/
http://mapreduce.stanford.edu/
http://plasma.camlcity.org/plasma/index.html
Page 20: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

You can download hadoop from link http://hadoop.apache.org/releases.htmlhttp://apache.bytenet.in/hadoop/common/ · 18 November, 2014: Release 2.6.0 available · 27 June, 2014: Release 0.23.11 available · 1 Aug, 2013: Release 1.2.1 (stable) available

DOWLOADING HADOOP

http://hadoop.apache.org/releases.html
http://hadoop.apache.org/releases.html
http://hadoop.apache.org/releases.html
Page 21: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

1. Name Node2. Secondary Name Node3. Job Tracker4. Task Tracker5. Data Node

HADOOP1. Storing Huge Data 2. Processing Huge Data

Hadoop Daemons

Page 22: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

HADOOP CORE COMPONENTS

Page 23: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

Modes in HadoopStandalone ModePseudo Distributed ModeFully Distributed Mode

Standalone modeIt is the default mode1 nodeNo separate process will be running(daemons)Everything runs in a single JVMSmall development,Test,Debugging

Page 24: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

Pseudo Distributed Mode1. A single node, but cluster will be simulated2. Daemons will run on separate process separate JVMs3. Development and Debugging

Page 25: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

1. Multiple nodes2. Hadoop will run in a cluster of machines/nodes used in Production Environment

Fully Distributed Mode

Page 26: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

Hadoop Architecture

Page 27: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits
Page 28: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

HivePigScoopAvro

ECOsystem Components

FlumeOozieHBaseCassandra

Page 29: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

Job Tracker

Page 30: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

Job Tracker contd..

Page 31: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

Job Tracker contd..

Page 32: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

Job Tracker Contd..

Page 33: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

HDFS write

Page 34: Big Data & Hadoop By Mr.Nataraj smallest unit is bit 1 byte=8 bits 1 KB (Kilo Byte)= 1024 bytes =1024*8 bits 1MB (Mega Byte)=1024 KB=(1024)^2 * 8 bits

HDFS write


Fabián E. Bustamante, Spring 2007 Bits and Bytes Today Why bits? Binary/hexadecimal Byte representations Boolean algebra Expressing in C

Chapter 4 Bits and Bytes · 2012-01-31 · Page 1 Chapter 4 Bits and Bytes • Why bits? • Representing information as bits –Binary/Hexadecimal –Byte representations • Representing

Informática Aplicada I Tecnicatura Higiene y Seguridad 1€¦ · Los dispositivos de almacenamiento y las memorias se miden en: 1 byte = 8 bits 1 Kb = 1024 bytes 1Mb = 1024 Kb 1

IRJET-ENHANCING DES AND AES WITH 1024 BITS KEY

COS217:Introductionto ProgrammingSystems · 47 Javavs.C:Details Java C Character&type char // 16-bit Unicode char /* 8 bits */ Integral&types byte // 8 bits short // 16 bits int

ISDB-T seminario técnico(2007) en Argentina · 6 DiBEG Digital Broadcasting ... Byte -> Bits MSB first Bits -> Byte MSB first Byte -> Bits ... receptor decodificador RS deben de

Http://proglit.com/. bits and text BY SA byte (the size of a cell of addressable memory) 8 bits on all modern systems octet = 8 bits

Informazione Analogica Versus Informazione Digitale · Zettabyte ZB 270 yottabyte YB 280 12 KB= ? Byte 123 MB = ? Bit 2 GB = ? KB NB: 210=1024. Using Bits to Represent Numeric Values

Mòdul de: Edició digital d'imatges amb el GIMP (1) · document informàtic) conté 22 Kb, aquesta és la seva mida de l'arxiu: 22 Kb. [ ] 8 bits = 1 byte, 1024 bytes = 1 Kilobyte

Bits and Bytes January 17, 2002 Topics Why bits? Representing information as bits –Binary/Hexadecimal –Byte representations »numbers »characters and strings

Higher Computing Science · 1 Megabyte 1024 Kilobytes 1 Gigabyte 1024 Megabytes 1 Terabyte 1024 Gigabytes 1 Petabyte 1024 Terabytes Bit Byte Kilobyte Megabyte Gigabyte Terabyte Petabyte

BĠLGĠSAYARIN (COMPUTER) TANIMI - web.firat.edu.trweb.firat.edu.tr/mbaykara/TemelBilgisayar.pdf · 8 Bit = 1 Byte(bayt) 1024 Byte = 1 Kilobyte(kilobayt) 1024 KB = 1 Megabyte(megabayt)

Dati e Programmi - Massimo Lauria · Due stati sono pochi:insiemi di bit n bit assumono2n configurazioni 1 byte 8 bit 1 KB 210 = 1024 byte 1 MB 210 = 1024 KB 1 GB 210 = 1024 MB 4

Nú bi á iNúmeros binários e Linggguagens de ...adriano/lc/2009/3-intro-linguagens... · 1 byte = 8 bits 1 Kb (Kilobyte) = 1024 bytes 1 MB (Megabyte) = 1024 KB 1 GB (Gigabyte)

Sistem kode - emiiryanti.dosen.ittelkom-pwt.ac.idemiiryanti.dosen.ittelkom-pwt.ac.id/.../24/2017/03/04_Sistem-kode.pdf · •1 byte = 8 bit •1 Kilobyte =1 KB = 1024 byte = 210 byte

New PC'en & hardware · 2019. 1. 8. · 8 bit = 1 byte 1024 bytes = 1 kilobyte (KB) 1024 kilobytes = 1 megabyte (MB) 1024 megabytes = 1 gigabyte (GB) 1024 gigabytes = 1 terrabyte

Bilgisayar Nedir - kafkas.edu.tr bilişim teknolojileri/01a... · Kilobyte => KB => 1024 Byte Megabyte => MB => 1024 Kilobyte Gigabyte => GB => 1024 Megabyte TeraByte => TB => 1024

Data Encoding CTPS 2018 - intranet.cb.amrita.edu · Byte to Terabyte • Bits can be grouped together to make them easier to work with. • A group of 8 bits is called a byte. Other

“The Class That Gives CMU Its Zip!” Bits and Bytes Aug. 29 ... · Bits and Bytes Aug. 29, 2002 Topics Why bits? Representing information as bits Binary/Hexadecimal Byte representations

“2 bits, 4 bits, 6 bits a byte!” Other bits 4-1-1 ...csbio.unc.edu/mcmillan/Comp411F17/Lecture02.pdf · Special "control" characters set upper two bits to 00 ex. cntl-g → bell,

COMPUTER STORAGE (TERMS)€¦ · Eight little bits make one BYTE. ... Gigabyte>Kilobyte>Megabyte>Bit>Byte Q.4 Convert: (i) 200 kb to bits (ii) 5,120,000 Bytes to Megabytes (iii) 2GB

Bits and Bytes 4/2/2008 Topics Physics, transistors, Moore’s law Why bits? Representing information as bits Binary/Hexadecimal Byte representations »numbers

Bits and Bytes CS 213 Aug. 27, 1998 Topics Why bits? Representing information as bits –Binary/Hexadecimal –Byte representations »numbers »characters and

Memórias do Computador · Unidades de Medidas Unidade Sigla Múltiplo Quantidade de bytes Byte B 2^0 1 = 8 bits KiloByte KB 2^10 1024 MegaByte MB 2^20 1 048 576

OttO Vector - 5 Bits To Byte [Digital Booklet]

DS2431 1024-Bit, 1-Wire EEPROM · General Description The DS2431 is a 1024-bit, 1-Wire® EEPROM chip orga- nized as four memory pages of 256 bits each. Data is written to an 8-byte

Datentypen Genauigkeiten Timo Damm. Bits and Bytes 1 kB = 103 Byte = 1000 Byte 1 MB = 106 Byte = 1000000 Byte (Diskette 1,44 MB) 1 GB = 109 Byte (CD 0,8GB,

“2 bits, 4 bits, 6 bits a Byte”Comp411 – Spring 2013 1/14/13 L02 –Information Theory 1 4-1-1 Information “2 bits, 4 bits, 6 bits a Byte” - Representing information using

Bits and Bytes January 15, 2004 Topics Why bits? Representing information as bits Binary/Hexadecimal Byte representations »numbers »characters and strings

Chapter 2: Computer Hardware€¢ 1 Peta Byte (PB) = 1024 TB = 250 Bytes • 1 Exa Byte (EB) = 1024 PB = 260 Bytes •Also known as the main memory. •It is mostly temporary or working

Lenovosi identifica la capacità del disco rigido, i sistemi operativi utilizzano una radice di 1024, ovvero 1 GB = 1024 MB, 1 MB = 1024 KB, 1 KB = 1024 byte. Lo standard …

Bits and Bytesarcs.skku.edu/.../Courses/ComputerSystems/02.1_Bits_Bytes_BinRep.pdfExamples (char type => 1 byte, 8 bits) ... Variable x has 4-byte representation 0x01234567 ... Word

Funddl ódamentos de la programación · 1 Byte = 8 bits = 1 = 1 carácter / nº (0-255 1 Kilobyte (KB) = 1024 Bytes 1 Megabyte (MB) = 1024 KB Dirección 1 Gigabyte (GB) = 1024 MB

Bits and Bytes Topics Why bits? Tell me Representing information as bits Binary/Hexadecimal Byte representations »numbers »characters and strings »Instructions

TALLER Rol de los profesionales del sector al 2030 · 8 Bits > 1 Byte ++ 1024 Bytes > 1 Kilobyte 1024 Kilobytes > 1 Megabyte 1024 Megabytes > 1 Gigabyte ... petabytes / mes 0,2 a