achieving memory level performance: secrets beyond shared flash - oracle › technetwork ›...

Copyright©2017, Oracleand/oritsaffiliates.Allrightsreserved.|

AchievingMemoryLevelPerformance:SecretsBeyondSharedFlash

Kothanda (Kodi)UmamageswaranVicePresident,ExadataDevelopment

GurmeetGoindiExadataProductManagement


SafeHarborStatementThefollowingisintendedtooutlineourgeneralproductdirection.Itisintendedforinformationpurposesonly,andmaynotbeincorporatedintoanycontract.Itisnotacommitmenttodeliveranymaterial,code,orfunctionality,andshouldnotberelieduponinmakingpurchasingdecisions.Thedevelopment,release,andtimingofanyfeaturesorfunctionalitydescribedforOracle’sproductsremainsatthesolediscretionofOracle.

2


DidYouMisstheStorageRevolution?

• Incumbentstoragevendorshavedecadesoldinvestmentinlegacyprotocolskeepingthemfromadoptingnewtechnologies

• PCIe FlashwithNVMe interfaceisanewinterfacethatrealizesfullflashpotential

• PCIe/NVMe storagearchitecturesareordersofmagnitudefasterthanwhatyouprobablyusetoday

• AvailablenowwithOracleExadataandSuperCluster

GoodChanceYourStorageVendorDidToo

2014ConventionalStorageEra

ModernFlashEra

StoragePe

rforman

ce

SCSISAS

StorageVendors

PCIeNVMe


SolidStateMediaisVeryDifferentThanSpinningDisk• ComparedtoSpinningDisk,Flash– Ismanyordersofmagnitudefaster– Hasmanyordersofmagnitudehigherbandwidth– Hasextremelylowlatency– Haswearingissuesasitages,buttechnologyiscatchingup– Isexpensive,butthepricegapisshrinking

• EverystoragevendorhassomeflashbasedsolutionforyourDatabase

4

Q:Willmydatabaserealizethefullbenefitofflashtechnology?A:Itwilldependonhowfastyoucanmovethedatafromtheflashtothedatabase


SCSIAccessModel• SCSIwasdesignedfortapesandHDDs• HDDsaresequential whereasFlashdevicesaremassivelyparallel• TraditionalIOstackisoptimizedforspinningmedia– 512Byteblocksizetransfers– Flashanddatabasesdo4KB/8KBIOs

• UsinglegacyinterfaceslikeSCSIfundamentallybottlenecksflashdrives

5

CPU HBA SCSI

8KBIO

512B

4 KBIO512B

512B

SCSI


PCIExpressVs SASConnectivity• PCIExpressisordersofmagnitudefasterthanSAS,andisgettingfaster• PCIExpresshasthesamecharacteristicsasFlash– HighThroughput– LowLatency

• UsinglegacyinterconnectslikeSASfundamentallybottlenecksflashdrives

6

0.61.2

4

8

SAS6Gbps SAS12Gbps PCIe3.0x4 PCIe3.0x8

ThrougputGB/s

PCIe has13xthroughputofSAS


PCIExpressFlashwithNVMe Interface• NonVolatileMemoryExpressisabrandnewgroundsupinterfacedesignedforflash• NVMe isinherentlyparallel• NVMe providesnativeatomicIOsizeaffinity fordatabases• NVMe IOstackmassivelyreduces CPUutilizationandlatency

7

PCIExpressFlashwithNVMe InterfaceistherightchoiceforyourDatabase

CPUPCIe NVMe

2.2xNVMe is2.2x

FasterthanSCSI


ExadataisLeadingNVMe Adoption

8

ThousandsofExadataandSuperCluster systemsshippedwithNVMe Flashsince2014

2014 2016

1st NVMe DrivebySamsung

2015

1st NVMe DrivebyIntel

ExadataX5-2Industry’sFristEnterpriseSystem

withNVMe

EMCAnnouncesDSSDD5withNVMeExadataCloud

ServiceusesNVMe inPublic

Cloud

ExadataX6-22ndGeneration

NVMe with3DVNAND

FacebooklaunchesLightningbasedon

NVMe

2017

EMCClosesDSSDdivision

ExadataX73rdGeneration

NVMe


SharedStorageHasManyAdvantagesoverLocalStorage

• Muchbetterspaceutilization• Muchbettersecurity,management,reliability• EnablesDBconsolidation,DBhighavailability,RACscale-out• Sharesstorageperformance– Aggregateperformanceofsharedstoragecanbedynamicallyusedbyanyserverthatneedsit

9

Servers

SharedStorage

SAN/LAN


NVMe PCI-eFlashDisruptstheStorageArrayModel

|OracleConfidential– HighlyRestricted 10

LatestPCIeFlash5.5GB/sec

SANLink=40Gb5GB/sec

Lessthan1Flashcard

LeadingAllFlashArray38GB/sec

Lessthan5Flashcard

Newimprovementsarecausing100Xbottlenecksacrosssharedstoragestack

ArrayHeads

CPU

All-FlashStorageArrayIOPath: manysteps,eachaddslatency andcreatesbottlenecks

SAS/SATA PCIeFlashChips

Switches

SAN/LAN

SSDCtrl

HostHBA

SAN/LAN


ExadataAchievesMemoryPerformancewithSharedFlash

• ExadataX7delivers350GB/secflashbandwidthtoany server– Approaches800GB/secaggregateDRAMbandwidthofDBservers

• Mustmovecomputetodatatoachievefullflashpotential– Requiresowningfullstack,can’tbesolvedinstoragealone

• Fundamentally,StorageArrayscanshareflashcapacity butnotflashperformance– Evenwithnextgenscale-out,PCIe networks,orNVMe overfabric

• Sharedstoragewithmemorylevelbandwidthisaparadigmchangeintheindustry– GetnearDRAMthroughput,withthecapacityofsharedflash

11

ExadataDBServers

ExadataSmartStorage

InfiniBand

CPUPCIe NVMeFlashChips

QueryOffload


GettingMemoryperformancewithSharedFlashusingSmartSoftware

12


Oracle’sInfrastructureInnovationsinFlash

• OracleExadata V2:Firsttobringflashstoragetothedatabasemarket

• OracleExadata X3:Doubledflashcapacity

• OracleExadata X4:100GB/sthroughputscansinasinglerack

• OracleExadata X5:LowestlatencyNVMe andincreasesscansto263GB/s

• OracleExadata X5:Hot-pluggableNVMe serverforthedatabase

• OracleLinux:FirstLinuxvendorwithproductionNVMe drivers

• OracleExadataX6:Highestthroughputover300GB/s, over5MillionIOPs

• OracleExadataX7:Highestthroughputover350GB/s,nearly6MillionIOPs

13


Oracle’sSoftwareInnovationsinFlash

• ExadataSmartFlashCache• ExadataSmartFlashLog• ExadataSmartFlashCacheScanAwareness• ExadataSmartFileInitialization• ExadataColumnarFlashCache• ExadataFlashCacheSpaceResourceManagement• SmartwriteburstandtempIOinFlashCache

14


ExadataSmartFlashCache– CompletelyAutomatic• UnderstandsdifferenttypesofI/Os fromdatabase– SkipscachingI/Os tobackups,datapumpI/O,archivelogs,tablespace formatting– CachesControlFileReadsandWrites,fileheaders,dataandindexblocks– Enablesmorespaceforrelevantuserdata

• Immediatelyadaptstochangingworkloads

• Write-backflashcache– Cacheswritesfromthedatabasenotjustreads

• Doesn’tneedtomirrorinflashforreadintensiveworkloads– Flasharraysstorebothmirrorcopiesalwaysinflashincreasingyourcost

• SmartScanscanrunatthethroughputofflashdrives– FlasharraysneedlotsofserverswithlotsofprocessesandstillcannotmatchSmartScan

throughputofsinglequery

• Providesperformanceofflashatcostofdisk

15

1.7PBDISK

360TBPCIFLASH

12TBDRAM

ColdData

HottestData

ActiveData


Exadata SmartFlashLog • OutliersinlogIOslowdownlotsofclients

• Outliersfromanyonecopyofmirrorslowdownalltheforegrounds– Databasewaittimegoesupby#foregrounds*Stalltime– Backlogdoesn’tclearimmediatelylikeanaccidentonthefreewayandincreases“logfilesync”waits

• Performancecriticalalgorithmslikespacemanagementandindexsplitsaresensitivetologwritelatency

• LegacystorageIOcannotdifferentiateredologIOfromothers

• UPSprotectedcacheintraditionalstorageseemstoworkinitiallyuntilthecacheisoverwhelmedbyotherwrites– Measurelogfilelatencywithfullbackuporadataloadrunning

16

LogBuffer

client

foreground

client

foreground

client

foreground

LogWriter


ExadataSmartFlashLog– CompletelyAutomatic

• SmartFlashLogusesflashasaparallelwritecachetodiskcontrollercache

• Whicheverwritecompletesfirstwins(diskorflash)

• Reducesresponsetimeandoutliers– “logfileparallelwrite”histogramimproves– Greatlyimproves“logfilesync”

• Usesalmostnoflashcapacity(<0.1%)

• NetworkresourcemanagementprovidespriorityforredologI/Os acrossthenetwork

• OLTPworkloadstransparentlyacceleratedandprovidepredictableresponsetimes

17

SmartLogging- Off SmartLogging- On

NoOutliers


ExadataColumnarFlashCache– CompletelyAutomatic

• HybridColumnarCompressionbalancesneedforOLTPandAnalytics• AsCPUsgetfasterwantevenfasterscans• SmartFlashCacheautomaticallytransformsblocksfromhybridcolumnartopurecolumnarforanalyticsduringflashcache population• Dualformatrepresentationforsinglerowlookups• Onlyselectedcolumnsreadfromflashduringaquery• Upto 5x queryspeedup

selectcolumnA fromtablewhere…

CompressionUnits Columns

FlashCachePopulation

18


FlashCacheSpaceResourceManagement• FlashCacheisasharedresource

• DatabaseasaServicecreatesneedforefficientresourcesharing

• Specifyminimum(flashCacheMin)andmaximum(flashCacheLimit)sizes,orfixedallocations(flashCacheSize),adatabasecanuseintheflashcache

ALTER IORMPLAN -

dbplan=((name=sales, flashCacheSize=100G), -

(name=finance,flashCacheLimit=100G, flashCacheMin=20G), -

(name=schain, flashCacheSize=200G))

• Containerdatabaseresourcespecifiedatthestorage

• Pluggabledatabasecontainerresourcelimitsexpressedaspercentagesinthecontainerdatabase

• DatabaseandPluggabledatabaseI/OresourcemanagementisuniquetoExadata

• Predictableperformancefordatabasequeries– nomorenoisyneighbor

FINANCE

SUPPLYCHAIN

SALES

19


WriteburstsandtempIOinflashcache– CompletelyAutomatic• Writethroughputoffourflashcardshasbecomegreaterthanthewritethroughputof12-disks

• Whendatabasewritethroughputexceedsthethroughputofdisks,smartflashcacheintelligentlycacheswrites– Schemachangesduringapplicationupgradesrewriteentiretablesinsome

packagedapplications– Largedatabaseconsolidationscanhavewriteburstsatthesametime

• WhenquerieswritealotoftempIOanditisbottleneckedondisk,smartflashcacheintelligentlycachestempIO– Writestoflashfortempspillreduceselapsedtime– Readsfromflashfortempreduceselapsedtimefurther

• SmarttoprioritizeOLTPdataanddoesnotremovehotOLTPlinesfromthecache

• Smartflashwearmanagementforlargewrites

• Muchfasterscansanddiskwrites

20

WriteBurts andTempIOin

FlashCache


Exadata SmartFlashBenefits• AutomaticDatabaseAwareFlashCache• SmartFlashLoggingavoidsredologoutliersautomatically• SmartFlashCacheScanprovidessubsetscanningandistablescanresistant• SmartFileInitializationcreatesafilebywritingmeta-datatoflashcache• SmartColumnarFlashCacheextendscolumnarbenefittostorageautomatically• SmartFlashCacheSpaceResourceManagementprovidesgranularcontrol• SmartwriteburstandtempIOinFlashCache

21


TheNextBigThing:

In-MemoryPerformanceinStorage

22


Analytics:ExadataBringsIn-MemoryAnalyticstoStorage

In-MemoryColumnarscans

In-FlashColumnarscans

• WithExadata Flashthroughputapproachingmemorythroughput,SQLbottleneckmovesfromI/OtoCPU

• ExadataautomaticallytransformstabledataintoIn-MemoryDBcolumnarformatsinExadataFlashcache– Enablesfastvectorprocessingforstorageserverqueries

• Uniquely optimizesnextgenerationFlashasmemory– WorksforbothrowformatOLTPdatabases,andHybridColumnarCompressedAnalyticsdatabases

23


12.8TBFlash

In-MemoryColumnarFormatsinFlashCache(12.2.1.1.0)

24

3- 4xOverallAnalyticsPerformanceImprovement

Upto1.5TBDRAM

SGA

IMC

25.6TBFlashx3 =76.8TB(ormore)IMC(In-MemoryColumnar)data

DatabaseServer

In-MemoryColumnarscans

In-FlashColumnarscans

HCC/OLTPcompressed/UncompressedData

StorageServer


OLTP: ExadataBringsIn-MemoryOLTPtoStorage

• ExadataStorageServersaddamemorycacheinfrontofFlashmemory– SimilartocurrentFlashcacheinfrontofdisk

• Cacheisadditive withcacheatDatabaseServer– OnlypossiblebecauseoftightintegrationwithDatabase

• 2.5xLowerlatencyforOLTPIO– 100usec

• Upto21TBofDRAMforOLTPaccelerationwithMemoryUpgradeKit– Compareto5TBofflashinV2Exadata

25

ComputeServer

StorageServer

Hot

Warm

Cold

Flash

DRAM

Disk


In-MemoryOLTPAcceleration– JourneyofaDatabaseBlock

DBBufferCache

In-MemoryOLTPCache

FlashCache

HardDiskDrive

OracleConfidential– Internal 26

DBBufferCache

In-MemoryOLTPCache

FlashCache

HardDiskDrive

1.DBreadsablock

ExadataServestheBlockfromStorageDatainitiallyresidesonharddisk

DatabaseServer

StorageServer



DBBufferCache

In-MemoryOLTPCache

FlashCache

HardDiskDrive


DBBufferCache

In-MemoryOLTPCache

FlashCache

HardDiskDrive

2.FlashCacheGetsPopulated

DatabaseServer

StorageServer



DBBufferCache

In-MemoryOLTPCache

FlashCache

HardDiskDrive


DBBufferCache

In-MemoryOLTPCache

FlashCache

HardDiskDrive

3.Databaseevictstheblock

ExadataCachestheblockinIn-MemoryOLTPCache

DatabaseServer

StorageServer



DBBufferCache

In-MemoryOLTPCache

FlashCache

HardDiskDrive


DBBufferCache

In-MemoryOLTPCache

FlashCache

HardDiskDrive

4.Databasereadsthesameblockagain

ExadataservestheblockfromIn-MemoryOLTPCachewith100uslatency

DatabaseServer

StorageServer


In-MemoryOLTPAcceleration

DBBufferCache

In-MemoryOLTPCache

FlashCache

HardDiskDrive


DBBufferCache

In-MemoryOLTPCache

FlashCache

HardDiskDrive

DataisneverinDBBufferCacheorInMemoryOLTPCacheatthesametime

DatabaseServer

StorageServer


CacheisAdditive withCacheatDatabaseServer• BlocksinbuffercachewillnotbecachedinStorageServerIn-MemoryOLTPCache– ClientreadhitsinStorageServerIn-MemoryOLTPCachewillevicttheblocksfromthecache– ClientreadmissesinStorageServerIn-MemoryOLTPCachewillpopulateflashcache,butnotStorageServerIn-MemoryOLTPCache

• BlocksevictedfrombuffercachegloballywillbepopulatedintoStorageServerIn-MemoryOLTPCache– StorageServerwillreadtheblocksfromflashcacheandpopulateintoIn-MemoryOLTPCache

• EliminationofContextSwitchesFurtherReducesLatency– 100usecreadlatency



IdentifyingWorkloadsforIn-MemoryOLTPAcceleration


Copyright©2015, Oracleand/oritsaffiliates.Allrightsreserved.| 33

AChoiceofExadata DeploymentModels

PublicCloudServiceCloudatCustomer

X7-2 X7-8

On-Premises

CustomerDataCenterPurchased

CustomerManaged

CustomerDataCenterSubscription

OracleManaged

OracleCloudSubscription

OracleManaged


ExadataAdvantagesIncreaseEveryYear

34

• SmartScan• InfiniBandScale-Out

• DatabaseAwareFlashCache• StorageIndexes• ColumnarCompression

• IOPriorities• DataMiningOffload•OffloadDecryptonScans

• In-MemoryFaultTolerance• Direct-to-wireProtocol• JSONandXMLoffload• Instantfailuredetection

•NetworkResourceManagement•MultitenantAwareResourceMgmt• PrioritizedFileRecovery

•UnifiedInfiniBand

• Scale-OutServers

• Scale-OutStorage• DBProcessorsinStorage

• PCIeNVMe Flash• TieredDisk/Flash

• Software-in-Silicon

• 3DV-NANDFlash

• In-MemoryColumnarinFlash• ExadataCloudService• SmartFusionBlockTransfer

• ExadataCloudatCustomer• In-MemoryOLTPAcceleration

DramaticallyBetterPerformanceandCost • HotSwappable

Flash• 25GigEClientNetwork

achieving memory level performance: secrets beyond shared flash - oracle › technetwork ›...

Documents