ics 101 spring 2016 data management - university of …lipyeow/ics101/2016spr/ics101-20160202...ics...

ICS101Spring2016DataManagement

Assoc.Prof.LipyeowLimInforma;on&ComputerScienceDepartment

UniversityofHawaiiatManoa

1LipyeowLim--UniversityofHawaiiatManoa

SurveyQues;on:Whatdoyouthinkdatamanagementis?

A.  Howtofilepapersinafilecabinet.B.  Howtoputdatafilesintherightdirectories

onyourcomputer.C.  Howtoorganize,store,andfinddatausing

computers.D.  Howtoensurethatyourdatacannotbe

accessedoralteredbyanunauthorizedperson.

LipyeowLim--UniversityofHawaiiatManoa 2

Objec;vesAttheendofthislecture,thesuccessfulstudentshouldknow:•  Whatisdataandwheredoesitcomefrom?•  Whatisdatamanagement?•  Whatisadatabase•  Whatistherela;onalmodeldata•  Howtosearchadatabase•  Whatisatransac;on•  Howtosearchunstructureddata


Whatis``data’’?

l  Dataareknownfactsthatcanberecordedandthathaveimplicitmeaning.

l  Threebroadcategoriesofdatal  Structureddatal  Semi-structureddatal  Unstructureddata

l  ``Structure’’ofdatareferstotheorganiza;onwithinthedatathatisiden;fiable.


Wheredoesdatacomefrom?


•  Howmuchdatadowehave?

UnitsofDataMul,plier(Decimal)

Nota,on Name

1 B byte(=8bits)1000 kB kilobyte10002 MB megabyte10003 GB gigabyte10004 TB terabyte10005 PB petabyte10006 EB exabyte10007 ZB zecabyte10008 YB yocabyte


Amusicfile

ADVDqualitymovie

Allthedatainatypicallibrary

Howmuchdataisthereintheworld?


•  AccordingtoIBM:2.5exabytesofdataarewasgeneratedeverydayin2012.

•  75%ofdataisunstructuredcomingfromsourcessuchastext,voiceandvideo.

Whoownsthedata?

Canwehandleallthatdata?

•  How?

•  Structureddata–Databasetechnology

•  Unstructureddata–Searchenginetechnology


TheDataManagementProblem


WhereisthephotoItooklastChristmas?

WheredidIreadabout“TuringMachines”?

Whereistheinvoiceforthiscomputer?

Whichproductisthemostprofitable?

User

Queries

Data

?

Whatisadatabase?l  Adatabase:acollec;onofrelateddata.

l  Representssomeaspectoftherealworld(akauniverseofdiscourse).

l  Logicallycoherentcollec;onofdatal  Designedandbuiltforspecificpurpose

l  Adatamodelisacollec;onofconceptsfordescribing/organizingthedata.

l  Aschemaisadescrip;onofapar;cularcollec;onofdata,usingtheagivendatamodel.


TheRela;onalDataModel•  Rela&onaldatabase:asetofrela&ons•  Arela&onismadeupof2parts:

–  Instance:atable,withrowsandcolumns.#Rows=cardinality,#fields=degree/arity.

–  Schema:specifiesnameofrela;on,plusnameanddomain/typeofeachcolumnoracribute.

•  E.G.Students(sid:string,name:string,login:string,age:integer,gpa:real).

•  Canthinkofarela;onasasetofrowsortuples(i.e.,allrowsaredis;nct).


ExampleInstanceofStudentsRela;on

•  Q1.Whatisthecardinalityoftherela;oninstance?(a)1 (b)2 (c)3 (d)4

•  Q2.Whatisthedegree/arityoftherela;oninstance?(a)2 (b)3 (c)4 (d)5


sid name login age gpa 53666 Jones jones@cs 18 3.4 53688 Smith smith@eecs 18 3.2 53650 Smith smith@math 19 3.8

Whyistherela;onalmodeluseful?•  Supportssimpleandpowerfulquerycapabili;es!

•  StructuredQueryLanguage(SQL)


SELECT S.sname FROM Students S WHERE S.gpa>3.5

sid name login age gpa 53666 Jones jones@cs 18 3.4 53688 Smith smith@eecs 18 3.2 53650 Smith smith@math 19 3.8

WhatisaDBMS?•  Adatabasemanagementsystem(DBMS)isacollec&onofprogramsthatenablesusersto– CreatenewDBsandspecifythestructureusingdatadefini;onlanguage(DDL)

– Querydatausingaquerylanguageordatamanipula;onlanguage(DML)

– Storeverylargeamountsofdata– Supportdurabilityinthefaceoffailures,errors,misuse

– Controlconcurrentaccesstodatafrommanyusers


TypesofDatabasesl  On-lineTransac;onProcessing(OLTP)l  Bankingl  Airlinereserva;onsl  Corporaterecords

l  On-lineAnaly;calProcessing(OLAP)l  Datawarehouses,data

martsl  Businessintelligence(BI)

l  Specializeddatabasesl  Mul;media

l  XMLl  GeographicalInforma;on

Systems(GIS)l  Real-;medatabases

(telecomindustry)

l  SpecialApplica;onsl  CustomerRela;onship

Management(CRM)l  EnterpriseResource

Planning(ERP)

l  HostedDBServicesl  Amazon,Salesforce


ABitofHistoryl  1970EdgarFCodd(aka“Ted”)inventedtherela;onalmodelintheseminalpaper“ARela;onalModelofDataforLargeSharedDataBanks”§  Mainconcept:rela&on=atablewithrowsandcolumns.§  Everyrela;onhasaschema,whichdescribesthecolumns.

l  Prior1970,nostandarddatamodel.l  NetworkmodelusedbyCodasyll  HierarchicalmodelusedbyIMS

l  Aser1970,IBMbuiltSystemRasproof-of-conceptforrela;onalmodelandusedSQLasthequerylanguage.SQLeventuallybecameastandard.


Transac;ons•  Atransac&onistheDBMS’sabstractviewofauserprogram:asequenceofreadsandwrites.– Eg.User1viewsavailableseatsandreservesseat22A.

•  ADBMSsupportsmul;pleusers,ie,mul;pletransac;onsmayberunningconcurrently.– Eg.User2viewsavailableseatsandreservesseat22A.

– Eg.User3viewsavailableseatsandreservesseat23D.


ACIDProper;esofTransac;ons

•  Atomicity:all-or-nothingexecu;onoftransac;ons

•  Consistency:constraintsondataelementsispreserved

•  Isola;on:eachtransac;onexecutesasifnoothertransac;onisexecu;ngconcurrently

•  Durability:effectofanexecutedtransac;onmustneverbelost


Q3.WhyuseaDBMS?

a)  Thedataistoolargetomanageinexcelfilesb)  Idonotwanttowritemyownprogramsto

findsomethinginthedatac)  Idonotwanttowritemyownprogramto

managemul;pleusersandtransac;onsd)  Alloftheabove.


TheDataManagementProblem


WhereisthephotoItooklastChristmas?

WheredidIreadabout“TuringMachines”?

Whereistheinvoiceforthiscomputer?

Whichproductisthemostprofitable?

User

Queries

Data

?

UnstructuredData•  Whataresomeexamplesofunstructureddata?

•  Howdowemodelunstructureddata?•  Howdowequeryunstructureddata?•  Howdoweprocessqueriesonunstructureddata?

•  Howdoweindexunstructureddata?


UnstructuredTextData•  Fieldof“Informa;onRetrieval”•  DataModel

–  Collec;onofdocuments–  Eachdocumentisabagofwords(akaterms)

•  QueryModel–  Keyword+BooleanCombina;ons–  Eg.DBMSandSQLandtutorial

•  Details:– Notallwordsareequal.“Stopwords”(eg.“the”,“a”,“his”...)areignored.

–  Stemming:convertwordstotheirbasicform.Eg.“Surfing”,“surfed”becomes“surf”


InvertedIndexes•  Recall:anindexisamappingofsearchkeytodataentries– Whatisthesearchkey?– Whatisthedataentry?

•  InvertedIndex:–  Foreachtermstorealistofpos;ngs–  Apos;ngconsistsof<docid,posi;on>pairs


DBMS doc01 10 18 20 doc02 5 38 doc03 13

SQL doc06 1 12 doc09 4 9 doc20 12

trigger doc01 12 15 doc09 14 21 doc10 1125 55

... ...

lexicon Posting lists

Whatisthedatainaninvertedindexsortedon?

LookupsusingInvertedIndexes

•  Givenasinglekeywordquery“k”(eg.SQL)–  Findkinthelexicon–  Retrievethepos;nglistfork–  Scanpos;nglistfordocumentIDs[andposi;ons]

•  Whatifthequeryis“k1andk2”?–  RetrievedocumentIDsfork1andk2–  Performintersec;on


DBMS doc01 10 18 20 doc02 5 38 doc01 13

SQL doc06 1 12 doc09 4 9 doc20 12

trigger doc01 12 15 doc09 14 21 doc10 1125 55

... ...

lexicon Posting lists

TooManyMatchingDocuments•  Ranktheresultsby“relevance”!•  Vector-SpaceModel

–  Documentsarevectorsinhi-dimensionalspace

–  Eachdimensioninthevectorrepresentsaterm

–  Queriesarerepresentedasvectorssimilarly

–  Vectordistance(dotproduct)betweenqueryvectoranddocumentvectorgivesrankingcriteria

– Weightscanbeusedtotweakrelevance

•  PageRank(later)


Star

Diet

Doc about astronomy

Doc about movie stars

Doc about behavior

Q4.Whichofthefollowingisthemostsimilartoaninvertedindex?a)  Bookmarks.b)  Contentpageofabook.c)  Theindexattheendofabook.d)  Adeckofplayingcards.


InternetSearchEngines


WorldWideWeb

WebPageRepository

InvertedIndex

WebCrawler

SearchEngineWebServer

Keyword Query

Query

Indexer

Ranked Results

Postings etc Doc IDs

Snipplets

BasicWebSearch•  hcp://www.googleguide.com/advanced_operators_reference.html


QueryExpression Whatitmeans

Bikingitaly BikingANDitaly

RecyclesteelORiron RecycleAND(steelORiron)

“Ihaveadream” “Ihaveadream”treatedasoneterm

Salsa-dance SalsaANDNOTdance

OtherniHyexpressions Whatitmeans

12+34-56*7/8 Evaluatesthearithme;cexpression

300EurosinUSD Converts300eurostoUScurrency

RankingWebPages•  Google’sPageRank

–  Linksinwebpagesprovidecluestohowimportantawebpageis.

•  Takearandomwalk–  Startatsomewebpagep–  Randomlypickoneofthelinksandgotothatwebpage

–  Repeatforalleternity•  Thenumberof;mesthewalkervisitsapageisanindica;onofhowimportantthepageis.


1

3

2

4

5

6

Ver;cesrepresentwebpages.Edgesrepresentweblinks.

Semi-structuredSearch

QueryExpression Whatitmeans

define:imbroglio Finddefini;onsof“imbroglio”

Halloweensite:www.census.gov Restrictsearchfor“halloween”toUScensuswebsite

Form1098-TIRSfiletype:pdf FindtheUStaxform1098-TinPDFformat

link:warriorlibrarian.com FindpagesthatlinktoWarriorLibrarian'swebsite

DanShugarintext:Powerlight Findpagesmen;oningDanShugarwherehiscompany,Powerlight,isincludedinthetextofthepage,i.e.,lesslikelytobefromthecorporatewebsite.

allin,tle:GoogleAdvancedOperators Searchforpageswith;tlescontaining"Google,""Advanced,",and"Operators"


Web pages are not really unstructured! Click “view source” to view HTML.

Summary•  DataManagementProblem

–  Howdoweposeandanswerqueriesondata?•  Structureddata

–  Rela;onalDataModel–  SQL–  Rela;onalDBMS–  Transac;ons

•  Unstructureddata–  Bagofterms–  Booleancombina;onofkeywordqueries–  InvertedIndexes(WebSearchEngines)

•  Semi-structureddata–  Couldusetechniquesfromeitherstructuredorunstructured–  Moresophis;catedkeywordqueries


SurveyQues;on:Ilearnedalotaboutdata

managementfromthislecture.

A.  StronglyAgreeB.  AgreeC.  NeutralD.  DisagreeE.  StronglyDisagree
