nosql and data lake architecture - dama nydama-ny.com/images/meeting/051817/nosql_databases/... ·...

NOSQLandDataLakeArchitecture

IBM590MadisonAvenue(at57thSt.)

NewYork,NY

ByTomHaugheyInfoModel,LLC

[email protected]

Agenda•  DescribeNOSQL•  ProposerelaConshipofNOSQLtothedatalake•  AnalyzethecapabiliCesofNOSQLtosupportBI•  Proposeanewoverallreferencedataarchitecture

©InfoModel,LLC.2017 2

NOSQL

Ontheoutside

Ontheinside–

hundredstothousandsofservers

Server


Other

Author

•  Blog entry could also be absorbed into Blog as an array. •  All arrows are links. •  In many cases, the actual user name would be included rather than just the ID. •  This is an example of a NOSQL data model, not necessarily of notation •  [ ] indicates embedding.

User

User ID User Name Password

Blog

Blog ID Blog Name Blog Description Blog Timestamp User ID Tag [ ] Tag Keyword

Blog Entry

Blog Entry TimeUUID Blog Entry Text Blog ID Comment [ ]

Comment TimeUUID Comment Text User ID Comment Timestamp

Follows

User ID [ ] Followed User ID

Followed By

User ID [ ] Follower User ID

Subscribes To

User ID [ ] Subscribed to Blog ID Subscription Date

Subscribed By

Blog ID [ ] Subscriber User ID Subscription Date

Time Ordered Blog Entries

User ID Blog Entry TimeUUID Posted By User ID Blog Entry Text

BlogNOSQLSampleDesign


UnderstandingNOSQL•  ThefollowingessenLaltoproperunderstandingofNOSQL:

–  DatadistribuLon:Dataisspreadhorizontallyovermanyservers–  Semi-structured:Not“schema-less”;schemaisembeddedinthedata

ortheapplicaCon–  Composite.Astructurecanconsistofotherstructures–  Hierarchical:1:Mrankingfromparenttochild(exceptgraphDBMSs)–  Key-valuestructure:MostNOSQL’sconsistofakeyandavalue

•  Valuecanbeablob,string,orcontainerofotherkey:valuepairs.•  TheexcepGontothisisgraphDBMS

–  Materializedqueries:Basicallyastructureforeachquery–  ApplicaLonorientaLon:Notenterprise-oriented,butqueryoriented.–  RelaLonships:UnidirecConallinks,notjoins.–  Datamodel:ANOSQLdatamodelisaphysical,notlogical,datamodel

•  Fourmaintypes:key:value,widecolumn,document,graph©InfoModel,LLC.2017 5

1-KeyValue

Key Value

Whateveryouwantittocontain,suchasaCSV,ablob,orotherkey:values,e.g.,streamingstockmarketdata:

record={Ccker:…,rate_date:…,price:…,price_change:…,percent_change:…

UseCasesFilestorageLogrecordsSessionstorageContentmanagementStreamingdata(songs,albums,etc.)ProductdatamanagementHighvolumedatafeedsUserdatamanagementHadoop

@InfoModel,LLC.2017 6

2-WideColumn(ColumnFamily)

Column Family: Personal Column Family: Business Row Key

Name Home Phone Office Phone Address

00001 Luke 212 555 5689 201 891 2536 21 McCoy Ave 00021 Bryce 212 232 6785 201 766 2091 26 McCoy Ave 00033 Matthew 201 234 5768 201 766 7381 26 McCoy Ave 00046 James 908 435 6242 908 657 5438 5 Hatfield St 00057 Jack 347 361 5429 973 376 8394 6 Wallace St

Thetableissortedbasedonrowkey

EachcellcanhavemulCpleversionsindividuallyCmestamped

347 361 9876

Timestamp1 Timestamp2

Cells

•  SomeCmescalledcolumn-orientedbutactuallyrow-oriented•  SeveralvariaConsbutthisistheHbaseversion

©InfoModel,LLC.20177

2–WideColumn(Cassandra)

CharacterisLcs:•  Sparsetablestructure•  Foreachkey,therecanbe:

–  Variableahributes–  Newcolumnswithout

indicaCngtheyarenew–  Omissionofcolumns

Key1 A=6 C=3 D=53 E=10Key2 B=42 D=62 E=84Key3 A=1 B=5 F=20 H=9

UseCases•  ProductCatalog/Playlist•  RecommendaCon/

PersonalizaConEngine•  SensorData/InternetofThings•  Messaging•  FraudDetecCon•  UsedbyeBay,Twissandra…


Table’ssortedbasedonrowkey

Columnssortedbasedoncolumnkey

3-Document•  A key : value store that understands the data Posts =

{_id: “A12345” author: “Mickey”, date: 22/6/2012, text: “The Adventures of Mickey Mouse”, keywords: [“drama”, “comic”, “adventure”] } comments : [ { author: “Nick Machiavelli”, date: 11/12/2012, rating: “Less filling, tastes great”, votes: 7000 } , … ] }


•  Contentmanagement•  Highvolumedatafeeds

–  Stockmarketdata–  Streamingmusic

•  OperaConalintelligence–  Atdetailedlevel–  Ataggregatelevel

•  Productdatamanagement•  Userdatamanagement(users,profiles,etc.)•  Hadoop

UseCasesforDocumentDataStores


4 - Graph Databases •  Every element contains a direct pointer to its adjacent elements

hhp://www.infoq.com/arCcles/graph-nosql-neo4j©InfoModel,LLC.2017 11

GraphDatabases•  Based on graph theory •  Addresses data complexity •  Direct path operations are easy •  Can be transactional •  FOAF (Friend Of A Friend) •  Usually paired with an index for search


Joins•  NOSQLstructuresarematerializedquerieswithallor

mostofthequerydataembeddedintotheonestructure•  Joinsarenotjust“unnecessary”inNOSQL,theyarenot

possible(throughtheDBMSs)–  Rememberdataishighlydistributedacrossmany,manyservers–  Joinswouldbefar-reachingandinefficient–  NOSQLembeddingimpliesredundancy

•  Joiningacrossserverswouldrequire(somethinglike):–  UseofaparCConingkeythatcanhash(notnecessarilythePK)–  AhashingalgorithmthatconsistentlyresolvesaparCConingkeytothesamenode

–  An~evendistribuConofdata–  Ideally,collocaConofdatatobejoined


MetadataintheDataLake•  Somemetadata,suchasdatatype,length,domain,

granularity,business/technicaldefiniConandothers,musteventuallybeassignedtodatalakefor:–  Data–  RelaConshipsandmore

•  SayMonthlySalesRevenueisingestedintothedatalakefromdifferentorgs/countries(inwhichcasethesetotalsarerawdata)–  Arethegrainsthesame?Dotheyagree?–  TheyarebothSalesRevenuebutsupposeoneis“salesasoflastdayofthemonth”andtheotheris“salesasofthelastFridayofthemonth”

–  Metadataisrequiredtounderstandthisdata


DataLakeonNOSQL?•  AdatalakecanresideonHadoop,NoSQL,AmazonSimpleStorageService,

arelaConaldatabase,ordifferentcombinaConsofthem•  Fedbydatastreams•  Datalakehasmanytypesofdataelements,datastructuresandmetadata

inHDFSwithoutregardtoimportance,IDs,orsummariesandaggregates•  Importanttounderstandthevariegatednatureofthedatalakedatain

relaContothemetamodeloftheperspecCveNOSQLdatabase–  Semi-structured–  Key:value(mostly)withitshierarchicalstructure–  ThekeyandcolumnnamebeingessenCalpartsofmostNOSQL

•  MoreotendatalakeiskeptonHadoopandfedtoorfromNOSQL–  SomeaddiConallyusedagraphdatabaseontoptokeeptrackofthe

relaConships–  OnceloadedontoNOSQL,thefullpowerofNOSQLcanbeused

•  Note:NOSQLisanoperaConal,notananalyCcal,datastore©InfoModel,LLC.2017 15

BIonNOSQL•  NOSQLdoesnotyethavecommodityBItools•  Inall,thereareseveralapproaches*toBIonNOSQL,

whichdivideintotwomajorgroups:–  ThosethatuseNoSQLonlyandstrengthentheapplicaConwithbeherUIs,beheradhocreporCngandothercustomfeaturesontotheNoSQLproductstheyarecurrentlyusing

–  ThosethatuseNoSQLtoruntheirapplicaCons,butthentakethatdataoutoftheNoSQLsystemandputitintoaRDBMSortradiConaldatawarehouseformore“aterthefact”analysis.

•  EachapproachhasmanysuccessstoriesandthebestapproachforaparCcularcompanyisbasedontheirspecificneeds,budgetsandskills

*BasedonpresentaConbyNicholasGoodman,andarCclesbyCharlesRoe“BI/AnalyCcsonNOSQL”


1-ApplicaLononTopofNOSQL•  ReportsonNOSQL

–  HaveadeveloperbuildanapplicaConforreporCngontopofNOSQL

–  HasthefullrichnessofNoSQLbutisexpensiveduetotheneedforadeveloper

NOSQL ApplicaCon


2-EnhancedNOSQL-Only•  AddsadynamicquerybuilderintothereporCngapp•  Needsadevelopertobuilditisevenmoreexpensive

NOSQLQueryBuilderApplicaCon

IndexesAggregates


3-NOSQLExtracttoRelaLonal•  SimilartotradiConalETLbutHadooporNOSQLarethesource–  ExtractfromNOSQL/HadoopandinsertintoRDBMS–  AllowstheuseofrichBItools

•  Addstofirstapproach,creaCngadynamicquerybuilderintothereporCngsystem–  Guidedadhoc–  Datafreshnessissueduetoday-olddata

RDBMSETLNOSQL/Hadoop

BI


Othersources

4-NOSQLasETLSource•  NOSQLsarejustpartoftheDWsourcing(ETL)•  DataextractedfromNOSQL/HadoopandinsertedintoDWandintegratedwithotherDWdata

•  ProsandCons–  CanusestandardBItools,whicharecostly–  Richflexibility/scalabilityonNOSQLgone–  ETLdevelopmentcost–  Note:Hadoopisabatchenvironment,notreal-Cme

RDBMSETLNOSQL/Hadoop

BI


5-NoSQLAddedtoBITools•  Developerintensive

–  AddsaservicetoastandardcommodityBItool–  FlahenstheNoSQLdataandoutputsitintoareport–  NoneedforSQLoradhocweb-basedaccesstools–  NoneedforloadsofexpensivereportswriheninNOSQL

•  TheprogramiswrihenoneCme–  ItisanapplicaConwithsubstanCaldevelopercosts–  CanuseM:Rforup-to-dateaccessto100%ofthedataset–  AggregaConsmightbeslower

NOSQL

NOSQLAddedto

BI


6–InterfacingNOSQLandBI•  ThirdpartyEnterpriseInformaConIntegraCon(EII)between

thecommodityBIandtheNoSQLordatalake–  TheEIItoolcanspeaktobothNOSQLandBI–  IntegraConwithotherdata,givesliveup-to-dateaccess–  ETLissimplewithINSERT/MERGEsdonenightlyandhasadhocaccess

tolive,cacheddata.–  “Bestofboth”:NoSQLontheback,BIonthefront–  ThecostandcomplicaConsofintroducingathirdsystem–  ThereissCllsomelossoftherichnessofNoSQLlanguage

EII BIToolsNOSQL


TransformingData•  Tousedata,itmustbeputintoauseablestate•  Youcannotputrawdataintoanyrepository(including

thedatalake)andpretendthatipsofactoiteliminatestheneedfortransformaCon(asmanyclaim!)

•  It’sacaseof“Paymenoworpaymelater”•  DataiseithertransformedbyETL/ELTbeforeitisstored

oritistransformedaKerbythequery–  Schemaonwrite-abatchprocessofETLorELTinarelaConalenvironment

–  Schemaonreadbythequerymanager–whetherinRDBMS,NOSQL,orMap:Reduce

•  OnemainreasonforstoringaggregatesintheDWistoprovideconsistentnumbers–  Thedatalaketooshouldensureconsistentnumbers


SampleStrategicQuery•  Strategicqueriesusingmanydimensionsand

aggregaConscouldposeaproblemtoNOSQL–  “GivemeabreakdownoftotalcommissionspaidandtransacConcountbyaccount,summarizedbyproducttypeandproductclass,andorderedbytheorganizaConthatownstheproductandtheorganizaConthatsoldtheproduct.”

•  SuchaquerywouldbedifficulttodoinNOSQLwithoutmaterializingthequeryandusingMap/ReducefuncCons

•  ADWonarobustpla|ormcouldefficientlysupportthiswithouthavingtomaterializetheaggregateandsCllsupportquerieswithothermixesofdimensions


MiningDataLakeData•  MiningistheuseofmathemaCcalalgorithmstofind

hiddenrelaConshipsinthedata•  Justasthepopularityofnewtoolsisexploding,soare

thecapabiliCesindatamining•  NOSQLiswellsuitedfordatamining•  Data-miningtechniquesfallintofourmajorcategories:

–  ClassificaCon–suchastargetedmarkeCng–  AssociaCon–suchasmarketbasketanalysis–  Sequencing–thosewhoboughtthisboughtthat–  Clustering–developingconclusionsusingspaceanddistance

•  NOTE:InHadoop,queryingandminingcanbedonethroughHive,MahoutandPig


DataLakeReferenceArchitecture

Appliance HADOOP NOSQL EDW Mart

RYO data

Consumers

Big Data Data Warehouse

RDBMSs

Real-time Analytics

Files Cloud Web Logs OLAP Tables Docs Sensors Events XML/JSON Streams

Data Streams

Trusted Data

Data Lakes

Raw Data

Data Streams


DW,DataMartandNOSQL•  ADWisageneralizedenvironmentinwhichdataisgatheredfrom

manysources,transformed,storedandthendeliveredwithbusinessmeaning–  Feedsanytypeofquery,includingadhocqueries–  ApplicaConneutral

•  DatamartsareaspecializedcollecConofrelateddataforaparCcularusercommunityandforalimitedCme–  Areuser-andapplicaCon-specific–  WillanswerverywellthosequesConswithindatamartsscope

•  FourtypesofNOSQL–  Key:value,widecolumn,document,graph–  MostNOSQLdatabasesarehierarchicalandoperaConal–  Thismeansmaterializingalmosteveryqueryrequiringanydatastores

•  DataLake–  AstoragerepositorythatholdsavastamountofrawdatainitsnaCve

formatunClitisneeded


QuesLons?

ThetermNOSQLwasfirstusedbyCarloStrozziin1998todescribeafile-based

databasehewasbuilding

nosql and data lake architecture - dama nydama-ny.com/images/meeting/051817/nosql_databases/... ·...

Documents