top 10 data warehouse features for dbas and ... - oracle · oracle database 12c release 2: top 10...

Copyright©2016,Oracleand/oritsaffiliates.Allrightsreserved.|

OracleDatabase12cRelease2:Top10DataWarehouseFeaturesforDevelopersandDBAs

KeithLakerSeniorPrincipalProductManagerAnalyGcSQLOracleCorporaGon @BigRedDW

oracle-big-data.blogspot.com


SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirecGon.ItisintendedforinformaGonpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfuncGonality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andGmingofanyfeaturesorfuncGonalitydescribedforOracle’sproductsremainsatthesolediscreGonofOracle.

2


• Availablenow– ExadataExpressCloudService

• Comingsoon– DatabaseCloudServices– ExadataCloudMachine

3

AnnouncingOracleDatabase12cRelease2onOracleCloud

OracleispresenGngfeaturesforOracleDatabase12cRelease2onOracleCloud.WewillannounceavailabilityoftheOn-PremreleasesomeGmeaTerOpenWorld.


Agenda

Managinglistsandlanguages

DataconversionandvalidaGon

Queryprocessingenhancements

AnalyGcsforDataLakes

Summary

4

1

3 4

5 6

7 8

2

9 10


Agenda

Managingverylargelistsandlanguages




5

1 2


Pre-12.2LISTAGG• Pre12.2syntaxtomanagelistswasrelaGvelysimple:

!LISTAGG(c.cust_first_name||' '||c.cust_last_name, ',’) ! WITHIN GROUP (ORDER BY c.country_id) AS Customer!

• Issue….keyissueisoverflowerror:– ORA-01489: result of string concatenation is too long!

• SoluGonsin12.2– IncreaseVARCHAR2sizetosupportlargerstrings– Handleoverflowerrors-Newsyntaxsupporttotruncatestring,opGonallydisplaycountoftruncateditemscount,andsettruncaGonindicaGon

!6


SupportForLargerVARCHAR2objects

•  Introducedin12cRelease1– VARCHAR2objectssupportsupto32K

SQL> show parameter MAX_STRING_SIZE!

NAME ! ! TYPE VALUE !--------------- ------ -------- !max_string_size string STANDARD!!ALTER SYSTEM SET max_string_size=extended SCOPE= SPFILE;!

– Needtorunrdbms/admin/utl32k.sqlscript7

AvoidsoverflowingLISTAGGfuncGonbyincreasingsizeofVARCHAR(2)objects


NewKeywordsForUseWithLISTAGG• With12.2wehavemadeiteasiertomanagelists:

LISTAGG(<measure_column>[, <delimiter>] . . .!

!– Whattodowhenanoverflowoccurs

• ON OVERFLOW ERROR (default)• ON OVERFLOW TRUNCATE <delimiter>!

– Controltoshow/not-showmanyvaluesweretruncated• WITHOUT COUNT (default)• WITH COUNT!

8


NewKeywordsForUseWithLISTAGGWITHCOUNTSELECT ! g.country_region,! LISTAGG(c.cust_first_name||' '||c.cust_last_name, ',' !

! ! !ON OVERFLOW TRUNCATE WITHOUT COUNT)! WITHIN GROUP (ORDER BY c.country_id) AS Customer!FROM customers c, countries g!WHERE g.country_id = c.country_id!GROUP BY country_region!ORDER BY country_region;!

9


Keywords:ONOVERFLOWTRUNCATEWITHOUTCOUNT

10


NewKeywordsForUseWithLISTAGGWITHOUTCOUNTSELECT ! g.country_region,! LISTAGG(c.cust_first_name||' '||c.cust_last_name, ',' !

! ! !ON OVERFLOW TRUNCATE ‘***’ WITH COUNT)! WITHIN GROUP (ORDER BY c.country_id) AS Customer!FROM customers c, countries g!WHERE g.country_id = c.country_id!GROUP BY country_region!ORDER BY country_region;!

11


Data-BoundCollaGons“…anamedsetofrulesdescribinghowtocompareandmatchcharacterstringstoputtheminaspecifiedorder…”• BasedontheISO/IEC/ANSISQLstandard9075:1999• Charactersetisalwaysdeclaredatthedatabaselevel• CollaGondeclaredforacolumn

– Doesnotdeterminethecharactersetofdatainthecolumn

• Whyisitimportant?– itsimplifiesapplicaGonmigraGontotheOracleDatabasefromanumberofnon-OracledatabasesimplemenGngcollaGoninasimilarway

12


1

Data-BoundCollaGons• Oraclesupportsaround100linguisGccollaGons

– Parameterizedbyaddingthesuffix_CIorthesuffix_AI•  _CI-Specifiesacase-insensiGvesort•  _AI-Specifiesanaccent-insensiGvesort

CREATE TABLE products!( product_code VARCHAR2(20 BYTE) COLLATE BINARY!, product_name VARCHAR2(100 BYTE) COLLATE GENERIC_M_CI!, product_category VARCHAR2(5 BYTE) COLLATE BINARY!, product_description VARCHAR2(1000 BYTE) COLLATE BINARY_CI!);!

– Product_nameistobecomparedusingGENERIC_M_CI-case-insensiEveversionofgenericmulElingualcollaEon

13


WantmoreinformaGon…• VisittheUnicodeDemoBoothinDatabaseareaofMosconeSouth

– Monday:10:15am–5:30pm– Tuesday:10:15am–5:15pm– Wednesday:10:15am–4:15pm

• DownloadslidesforsessionCON6400:– ConqueringLarge-ScaleUnicodeDatabaseMigraGons

•  ThomasMize,DatabaseAdministrator,NorthropGrumman• WeiranZhang,Director,Oracle

14


Agenda





15

3 4


Pre12.2DataConversionErrorsParsingData•  Issue:Parsingdatainputfromawebformorloadingdatafromexternalfiles,converGngtospecificdatatypetypicallygenerateserror:

SQL Error: ORA-01722: invalid number!

•  SoluGons:– DetectdataconversionerrorswithnewVALIDATE_CONVERSIONfuncGon– EnhancementstomostofconversionfuncGonslikeTO_NUMBER,TO_DATE,CASTetc.tohandledataconversionerrorsandreplacewithuserprovideddefaultvalues

16


DetecGngconversionerrors-VALIDATE_CONVERSION

• UsefultodetectifinputvaluecanbeconvertedtodesGnaGontype.Returns1ifconversionissuccessful,otherwisereturns0

• VALIDATE_CONVERSION('123a'asNUMBER)-->returns0• VALIDATE_CONVERSION('123'asNUMBER)-->returns1

• CanbeefficientlyusedasfiltertoavoidbaddatawhileimporGngforeigndatasources,ETLprocessing

17

IdenGfyinginvaliddataintheinputstreams


TwoMethodsforDealingWithConversionErrors

SELECT! VALIDATE_CONVERSION(empno AS NUMBER) AS is_empno,! VALIDATE_CONVERSION(mgr AS NUMBER) AS is_mgr,! VALIDATE_CONVERSION(hiredate AS DATE) AS is_hiredate,! VALIDATE_CONVERSION(sal AS NUMBER) AS is_sal,! VALIDATE_CONVERSION(comm AS NUMBER) AS is_comm,! VALIDATE_CONVERSION(deptno AS NUMBER) AS is_deptno !FROM staging_emp;!

18

Findrow-columnvaluesthatarecausingerrors:VALIDATE_CONVERSION



19

Findrow-columnvaluesthatarecausingerrors:VALIDATE_CONVERSION


Handlingdataconversionerrors-TO_xxxx(),CAST()

• Pre12.2:TO_NUMBER('123a')-->returnsinvalidnumbererror(ora-01722)New12.2Features• NewsyntaxDEFAULT<default_value>ONCONVERSIONERROR

– Replaceconversionfailurewithuserdefineddefaultvalue– TO_NUMBER('123a'DEFAULT'123'ONCONVERSIONERROR)-->returns123

•  ThisnewsyntaxcanbeusedforTO_NUMBER,TO_DATE,TO_TIMESTAMP,TO_TIMESTAMP_TZ,TO_DMINTERVAL,TO_YMINTERVALandCAST

20

-Replacingincorrectormissingdatawithdefaultvalues



• CASTfuncGonreturnsuser-specifiedvalue,insteadofanerror. . .CAST(mgr AS NUMBER DEFAULT 9999 ON CONVERSION ERROR),!

. . .

• ReducesfailuresduringdatatransformaGonanddataloadingprocesses.

21

EnhancedCASTandTO_xxxfuncGons


UsingCASTandTO_XXXXFUNCTIONS

INSERT INTO emp!SELECT ! empno,! ename,! job,! CAST(mgr AS NUMBER DEFAULT 9999 ON CONVERSION ERROR),! CAST(hiredate AS DATE DEFAULT sysdate ON CONVERSION ERROR),! CAST(sal AS NUMBER DEFAULT 0 ON CONVERSION ERROR),! CAST(comm AS NUMBER DEFAULT null ON CONVERSION ERROR),! CAST(deptno AS NUMBER DEFAULT 99 ON CONVERSION ERROR)!FROM staging_emp!WHERE VALIDATE_CONVERSION(empno AS NUMBER) = 1!

22

UsingenhancedfuncGonstoremoveincorrectdatatypesandcorrectconversionerrors


Agenda





23

5 6


QueryProcessingEnhancements

•  Issue:– Datavolumesincreasingrapidly– Usersrunningmoreadhoc,moresophisGcatedworkloads

•  SoluGons:– OpGmizequeryprocessingformostresourceintensiveworkloads– Returnresultsfaster– Freeupresourcestorunotherqueries

24

Performanceboostersthatrequirezerocodechanges


In-MemoryCursorDuraGonTempTables• ComplexqueriesoTenprocesssamequeryblockmulGpleGmes•  SuchasWITHclausequeriesandstartransformaGons

•  SoluGon:– CreatetemporarytablesforqueryresultsandstoretheminmemoryfortheduraGonofthecursor

• Result:– OpGmizaGonenhancesmaterializaGonofintermediateresultsfromrepeGGvelyusedsub-queries

– ImprovesperformanceandopGmizesI/O

25


In-MemoryCursorDuraGonTempTables

26


BeterJoinProcessingforDWQueries:BandJoins

•  JoincondiGoninvolvesrangeofvaluesratherthanexactmatch– Currentlybandjoinisevaluatedeitherthroughnestedlooporsort-mergejoin

• KeyobjecGves:– Addhashalgorithmtosupportprocessingbandjoins– Detectandevaluatebandjoinsmoreefficiently– DeliverperformanceforbandjoinscomparabletoequijoinoperaGons

Whatisabandjoin?


WhenAreBandJoinsUsed?

•  Typicalusecase-MetricusedinwebanalyGcs– DiscoverifpeoplestareatapageforalongGme:

•  ifit’sanarGcle,thatmightbegood,thatmeansthey’rereadingthearGcle,•  ifit’sanavigaGonpageitmaybebad,itmeanstheycan’tquicklyfindwhatthey’relookingfor!

– Findalltheuserclicksmadewithinfivesecondsofsomepageload


BeterJoinProcessingforDWQueries:BandJoins

•  ImplementtruncaGnghashbandjoinalgorithm– Dividesthejoindomainintosmallparts– Each“small”partissize/rangeoftheband– Mapsallthejoinvaluesinsamebandintoonesinglevalue

• ChoosingofjoinmethodwillbebasedonopGmizercost– Ifqueryusesbandjoinprocessingit’sindicatedintheexplainplan

SoluGonforopGmizingbandjoins


Example:BandJoin

30

Queriesusingenhancedbandjoinprocessingshowed10Xbecerperformance

SELECT e1.ename, e2.ename!FROM emp e1, emp e2!WHERE e1.sal between e2.sal - 500 AND e2.sal + 500!AND e1.empno < e2.empno;

#1

#2

#3

#4

ENAME SALARYENAME SALARY


Example:BandJoinwithinexplainplan

. . .!

Predicate Information (identified by operation id):!

---------------------------------------------------!

. . .!

. . .!

5 - access(“E1".”SAL"<=“E2".”SAL”+500)!

filter(“E1".”SAL">=“E2".”SAL”-500 AND!

“E1".”SAL"<=“E2".”SAL”+500)!

. . . !

ConfidenGal–OracleInternal/Restricted/HighlyRestricted 31

DetailsofbandjoinoperaGonshownagainstthejoinoperaGoninexplainplan


Agenda





32

7 8 9 10


ExploringToday’sBigDataLakes

Businessesneedtoanswerslotsofdifferent“Howmany…”typequesGons– Howmanyuniquesessionstoday– Howmanyuniquecustomersloggedon– Howmanyuniqueeventsoccurred

Pre12-2SoluGon-useSQL’sCOUNT(DISTINCT...)funcGon

– ReturnstheexactnumberofrowsthatcontaindisGnctvaluesofspecifiedexpression– BUTcanberesourceintensivebecauserequiressorGng

33

SupporGngQueriesThatAsk:“HowMany…”


CountDisGnctQueryProcessing

34

ResourceintensiveSORT-8GBofmemory(PGA)+164GBoftemp


Ge{ngApproximateCounts• Most“HowMany…”queriesdon’tneedexactanswers,approximaGonsaregoodenough

• Approximateanswerscanbereturnedsignificantlyfaster• NewSQLfuncGonAPPROX_COUNT_DISTINCT (expr)

– processeslargeamountsofdatasignificantlyfaster– usesHyperLogLogalgorithm– negligibledeviaGonfromexactresult

•  ignoresrowscontainingnullvalues– supportsanyscalardatatype

•  DoesnotsupportBFILE,BLOB,CLOB,LONG,LONGRAW,orNCLOB

35


Resultsforaccuracy

• Accuracythatistypically97%

with95%confidencePerformanceResults

•  5-50ximprovement

• Notes:– thisapproachdoesnotusesampling,itusesahash-basedapproach

– ignoresrowsthatcontainanullvalueforspecifiedexpression

– SupportsanyscalardatatypeotherthanBFILE,BLOB,CLOB,LONG,LONGRAW,orNCLOB

AccuracyandPerformance

36


CountDisGnctQueryProcessing

37

ResourceintensiveSORTremoved


ApproximateStaGsGcs

•  Issue:PERCENTILE_CONT, PERCENTILE_DISC, MEDIAN funcGonsrequiresorGngandcanconsumelargeamountsofresources!

SoluGon:• NewapproximateSQLfuncGonsusefewerresources:!

APPROX_PERCENTILE!APPROX_MEDIAN!

– Uselessmemory,nosorGng,nouseoftemp

38


HowtogetmoreinformaGonaboutresultset

•  Eachapprox.funcGoncanusedifferentalgorithms,reporterrorrates,confidencelevels:

1. DETERMINISTIC/NONDETERMINISTIC [default]!– Non-determinisGcisfasterbutresultsmayvary,goodforpersonaldatadiscoveries– DeterminisGc,slightlyslower;beterwhereresultsaresharedwithotherusers

2. ERROR_RATE!– Returnsthemarginoferrorassociatedwithresult

3. CONFIDENCE– Returnedasapercentagethatindicatesthelevelofconfidence

AddiGonalkeywords

39


Resultsforaccuracy• Realworldcustomerusecase

•  Errorrangearound0.1-1.0%– Ingeneralaccuracywillnotbeamajorconcern

PerformanceResults

•  6-13ximprovement

• Notethatmajorsavingscomingfrom:– Useofboundedmemoryregardlessoftheinputsizepergroupbykey

– ReducGoninchanceofspilltodisk

AccuracyandPerformance

40


Usingapproximateprocessingwithzerocodechanges!

• UsingfollowingparameterstoconvertexisGngqueries:

– approx_for_count_distinct = TRUE !•  ConvertexisGngCOUNT(DISTINCT …)funcGonstouseapproximateprocessing

– approx_for_percentile = TRUE!•  AddiGonalparametertocontroldeterminisGcresults:approx_percentile_deterministiccanbeTRUE/FALSE!

• Canbesetatsessionanddatabaselevel

ConverGngExisGngQueriesToReturnApproximateAnswers

41


NewFuncGonsforBuildingApproximateAggregates

Issue: creating reusable approximate results to support other query shapes

Solution:

•  AllowsingletabletobeusedforqueriesthatrequireapproximateanswersforaggregaGonssliced-and-dicedoverdifferentdimensions

•  AllowapproximateaggregaGonstobematerializedandconvertedbacktoapproximatecountdisGnctandapproximatepercenGles.

•  Added3newfuncGonstosupportcreaGonofaggregatedapproximaGons

42


NewFuncGonsForBuildingApproximateAggregatesAPPROX_xxxxxx_DETAIL(expr[DETERMINISTIC])

APPROX_xxxxxx_AGG(expr) TO_APPROX_xxxxxx(expr,pct)

COUNTRY STATE PRODUCT …

US CA A

US CA B

...

US IL A

US IL C

US IL D

…

US TX A

…

COUNTRY STATEAPPROX.CNTDISTPRODUCT(INTERNAL)

US CA 000111011001

US IL 100010010010

US TX 111001001000

US CO 001001001000

US NY 011010010010

COUNTRYAPPROX.CNTDISTPRODUCT(INTERNAL)

US 000111011001

UK 100010010010

… 111001001000

COUNTRY STATEAPPROX.CNTDISTPRODUCT

US CA 2

43


Step1CreateaggregaGonresultinbinaryformat

APPROX_xxxxxx_DETAIL(expr[DETERMINISTIC])• BuildssummarytablecontainingresultsforalldimensionsinGROUPBYclause

• Datareturnedinbinaryformat/BLOBobject• BinaryformatcontainstuplesforallcombinaGonsofdimensions• Resultscanbestoredintable/materializedview COUNTRY STATE

APPROX.CNTDISTPRODUCT(INTERNAL)

US CA 000111011001

US IL 100010010010

US TX 111001001000

US CO 001001001000

US NY 011010010010

44


Step2(OpGonal)CreatehigherlevelaggregaGonfrombinaryformat

APPROX_xxxxxx_AGG(expr)• ExprisabinarysourcegeneratedbyAPPROX_xxxxxx_DETAIL• ObjecGveistobuildahigherlevelsummarytablecontainingresultsforalldimensionsinGROUPBYclausewithoutrescanningsourcedata– Speedsupprocessingforhigherlevelsummariesfeedingdatatodashboards

• Datareturnedinbinaryformat/BLOBobject• BinaryformatcontainstuplesforallcombinaGonsofdimensions• Resultscanbestoredintable/materializedview COUNTRY

APPROX.CNTDISTPRODUCT(INTERNAL)

US 000111011001

UK 100010010010

… 111001001000


Step3ConvertbinaryresultsintopresentaEonformat

TO_APPROX_xxxxxx(expr,pct)• ExprisabinarysourcegeneratedbyAPPROX_xxxxxx_DETAILorAPPROX_xxxxxx_AGG

• PctisthepercenGlevaluetobeextracted(median=50)• Datareturnedinnumericformatforreports/dashboards

COUNTRY STATEAPPROX.CNTDISTPRODUCT

US CA 2

COUNTRY STATEAPPROX.CNTDISTPRODUCT(INTERNAL)

US CA 000111011001

US IL 100010010010

US TX 111001001000

US CO 001001001000

US NY 011010010010

46


UsingapproxMVswithzerocodechanges

• ParameterstoconvertexisGngqueries:– Setapprox_for_aggregationtoTRUE

• EnablesautomaGcconversionfromexactaggregatetoapproximateaggregate

But…– IndividualmaterializedviewsmusthaveENABLEQUERYREWRITEclause

– SessionparameterQUERY_REWRITE_ENABLEDmustbesettoTRUE(thedefault)orFORCE

ConverGngexisGngqueriestouseMVsthatcontainapproximateresults

47


SQLPaternMatching• Recognize patterns in sequences of events using SQL

– Sequence is a stream of rows

– Event equals a row in a stream

• New SQL construct MATCH_RECOGNIZE – Logically partition and order the data

•  ORDER BY mandatory (optional PARTITION BY)

– Pattern defined using regular expression using variables

– Regular expression is matched against a sequence of rows

– Each pattern variable is defined using conditions on rows and aggregates

48


MATCH_RECOGNIZEUsessimilarsyntaxtoanalyGcfuncGons

3 8 12 16 Days

Stock price

SELECT symbol, first_x, last_z, price, match_no, var

FROM ticker MATCH_RECOGNIZE ( PARTITION BY symbol ORDER BY tstamp MEASURES LAST(z.tstamp) AS bottom_tstamp, FIRST(x.tstamp) AS end_tstamp, MATCH_NUMBER AS match_no CLASSIFIER AS var ALL ROWS PER MATCH PATTERN (X+ Y+ W+ Z+) DEFINE X AS (price < PREV(price)), Y AS (price > PREV(price)), W AS (price < PREV(price)), Z AS (price > PREV(price)));

49


Summary

Managinglistsandlanguages1.  EnhancementstoLISTAGG

2.  NewcolumnlevelCOLLATIONfeatures


3.  EnhancementstoCAST

4.  NewVALIDATE_CONVERSIONfuncGon


5.  In-MemoryCursorDuraGonTemptables

51

Top10datawarehousefeaturesforDBAsandDevelopers

6.  EnhancementstojoinprocessingAnalyGcsfordatalakes

7.  APPROX_COUNT_DISTINCT

8.  APPROX_PERCENTILE/MEDIAN

9.  ApproximateaggregaGons

10.  SQLpaternmatching


SafeHarborStatementTheprecedingisintendedtooutlineourgeneralproductdirecGon.ItisintendedforinformaGonpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfuncGonality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andGmingofanyfeaturesorfuncGonalitydescribedforOracle’sproductsremainsatthesolediscreGonofOracle.

52

top 10 data warehouse features for dbas and ... - oracle · oracle database 12c release 2: top 10...

Documents