Effec%veDataScienceinAerospaceApplica%ons

GeoffreyClark|2016-06-25|GR-81-RHOcopyrightLucidata2016

LucidataInforma%cs?

•  GeoffreyClark,PrincipalandCEO– DataModeler:logicalandra%onalDW– Solu%onArchitect:analy%cs– Strategy:risks,what-if,sims,wargames

•  AssociatesatLucidata– SeniorDataModeler,requirementsexpert– AdvancedAnaly%csexperts– SeniorGISanalystsandFOSS4Gdevelopment

copyrightLucidata2015

DualChallengesofAnaly%cs

Market

TMS

CRM

Analytics Solution StandardBIFormal,Trusted,Shared,PublicDataExplora0onAd-hoc,AgileExperimental

individualsPowerUsers

Clean,Load&JoinData

FormNewBIQues%ons

AnalyzeData

FindNewData

Analytics User Segments

FIN

YourData

Knowledge,Defini%ons,Hierarchies,Rela%onships

Ad-HocDataPlayground

Standard,IndustryDataIndustry

Reference

Execu%ves,Managers

Analysts,KnowledgeWorkers

TechnicalChallenge,DataIntegra%on CulturalChallenge,UserIntegra%on

HowUniqueisyourdata?*

*isthatasourceofadvantage,orconfusion?

En%ty-Rela%onship(ER)Modeling

ThisisanexampleEn.ty-Rela.onshipDiagram(ERD),whichexplainshowtoreadthenota.on.Itisalsoanexampleofahighlyabstractmodel,whichis"datadriven",meaningthatnewThingTypes,newThingsandnewThingRela.onshipsmaybeeasilyaddedtoadatabasebasedonthisdesignwithoutneedingtochangethedatabasestructures.Thisprac.cewascommoninearlydatabases,builtforon-linetransac.onprocessing(OLTP).Thisstandsincontrasttotheconcretebusinessseman.csimplementedaspartofDimensionalDataModelingefforts,suppor.ngon-lineanaly.calprocessing(OLAP).

copyrightLucidata2016

Simulate

Op,mize

Forecast

Derive(datamining)

Summarize&Describe(sta%s%cs)

Visualize

Join&Filter(datawarehousing)

Measure&Store(sourcesystems)

En,,es&Rela,onships(datamodeling)

BI

EA

STAT

SOR

Analy,

cs

QualityAMATEUR PROFESSIONAL

TheProgressionofAnaly%cs

copyrightLucidata2015 5

...restuponthisfounda%on

Thesetypesofmodels...

BigDataHistory,viaGoogleTrends

Source:heps://www.google.com/trends/

Whatresearchdidwedo?

"p1sk"-project#1,surrogatekeys.airport_id=2369

"p2nk"-project#2,naturalkeys.

airport_iata_cd='SEA’airport_from_dt='2011-07-01’

"p3uu"-project#3,universallyuniqueiden%fier(UUID),

airport_uuid=7cbcc311-18c9-4497-99c9-62c42fd1ef2b

"p4hk"-project#4,hashkey.airport_key=0ed805d25fc96166a5895857a252de4b

WhathastheBigDatainnova.oncycletaughtusaboutdatadesign?

OriginalT100“GreenBook”

DataSource:T100“GreenBook”

ImportanceofReferenceData

copyrightLucidata2015

Ifyouhaveafloodof.mestamps,beNerknowwhat.mezonetheyrepresent.And,don’tbeliketheMarsClimateOrbiter,getyourunitsofmeasureright!

Nbr Full Number ISO 31 Description Comments 10^0 1 C62 one (or unit) "EA" for each from ANSI 10^1 10 ten 10^2 100 CEN one hundred 10^3 1000 MIL one thousand 10^4 10,000 ten thousand 10^5 100,000 one hundred thousand 10^6 1,000,000 MIO one million Somewhat confusing 10^7 10,000,000 ten million 10^8 100,000,000 one hundred million 10^9 1,000,000,000 MLD one milliard in EU one billion (US) Horribly confusing! 10^12 1,000,000,000,000 BIL one billion in EU (one trillion in US) Horribly confusing! 10^18 1,000,000,000,000,000,000 TRL one trillion (EUR) Horribly confusing!

ISO31-UnitsofMeasure

Moreworkremainstobeeercoordinatethemeasurementac%vi%esofhumankind,thefuturewillappreciateit!

project#1,surrogatekey project#2,naturalkeysproject#3,universallyuniqueidproject#4,hashkey

Addi%onalFactorstoConsider

Source:DanLinstedtonlearndatavault.com.

Addi%onalResearchPlans

•  Howdoesdatadesignchangewhenusingdistributeddatabasetechnology?aliases-massiveparallelprocessing(MPP),“Sharded”

•  Howdoesdatadesignchangewhenusingcolumnardatabasetechnology?

•  Howdoesdatadesignchangewhenusinggraphdatabasetechnology?aliases–“RDF”,“triplestore”.

•  Howdoesperformancechangewithdifferentdiskop%ons–HDD,SSD,SSDRAIDS,etc.

Informa%onDensity CharlesJosephMinard’sCarteFigura.vefrom1869,depic%ngNapolean’s1812invasionofRussia,andaqermathinsevendimensions(la%tude,longitude,%me,temperature,armygroup,andmilitaryphase).“Thebeststa.s.caldrawingevermade”--EdwardTuqe

Source:heps://en.wikipedia.org/wiki/File:Minard.png