faster transactions & analytics with the new sql2016 in-memory technologies

43
E102 Faster transactions & analytics with the new SQL2016 In-memory technologies [email protected] m Data Platform Solution Architect www.henkvanderValk.com

Upload: henk-van-der-valk

Post on 13-Apr-2017

35 views

Category:

Software


0 download

TRANSCRIPT

PowerPoint Presentation

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242VMSQL2016EE\hevandvMicrosoft Worldwide Partner Conference 2016 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.16/10/16 22:121

E102Faster transactions & analytics with the new SQL2016 In-memory [email protected] Platform Solution Architect

www.henkvanderValk.com

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Session objective:Understand the SQL Server In-Memory technology & how to benefit from it!

Key takeaways:

SQL Server 2016 In-Memory OLTP, Columnstore & In-Database R support provide serious performance gains for your transactional, data warehouse & ad-hoc Analytics query workloads!

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242

16/10/16 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.3Tech Ready 15

AgendaIntroIn-Memory OLTP & enhancementsClustered Columnstore & enhancementsDemos!In-memory OLTPData loading from flatfileMemory ManagementCopy staging data to destinationR!(Direct Query with Power BI)- Wrap-up

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242SQL Server 2016 as fraud detection scoring engine

https://blogs.technet.microsoft.com/machinelearning/2016/09/22/predictions-at-the-speed-of-data/

HTAP (Hybrid Transactional Analytical Processing)

8 socket, 192 cores16 TB RAM

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242SQL 2016: World Record Breaking Performance (TPC-H)

http://www.tpc.org/tpch/results/tpch_perf_results.asp?resulttype=noncluster

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242

Hardware

Lenovo System x3850 X66TB DDR4 4x Intel Xeon E7-4800/8800 v4 series, up to 3.2GHz, up to 1866MHz DDR4 memory access, 24 cores per processor

Lenovo System x3950 X612TB DDR4 4x Intel Xeon E7-4800/8800 v4 series, up to 3.2GHz, up to 1866MHz DDR4 memory access, 24 cores per processor

DL580 Gen9 / 6TB DDR4

HDX 24TB DDR4

Cisco UCS C460 M46TB DDR4

Dell PowerEdge R930 Rack Server6TB DDR4

Bull Bullion S16 Server

288 cores24TB

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Microsoft Azure Cloud - SQL PAAS & VMPay for Use & Capacity on DemandAzure Virtual Machines GS5: 448 GB & up Elastic-Cloud data warehousing service: SQLDW (MPP)

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Demos

SQL Server with 24TB RAM

& In-Memory OLTP

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Starting SQL2016 on a server with 24 TB RAM

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242

Microsoft Worldwide Partner Conference 2016 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.16/10/16 22:1210

Question

Which data provider will be fastest for OLTP?

https://github.com/Microsoft/sql-server-samples/releases/tag/in-memory-oltp-demo-v1.0

SqlClient / OLEDB / ODBC ?

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Post session Update : In case Step 2) Migrate SP results in an error like I encountered, please run the command : EXEC sp_changedbowner "sa -HenkMicrosoft Worldwide Partner Conference 2016 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.16/10/16 22:1211

Data loading for StagingQuestion: whats the absolute fastest way to load 7 GB of data from a single flatfile into a table?

Transport vehicles: BCP/BulkInsert/SSIS/Polybase/Powershell/External Script?

Destination:HEAP/In-memory table/CCI?

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Demo

Data loading into

In-Memory table

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242

(demo)Bulk Insert into non-durable In-mem tableSELECT pool_id , Name , min_memory_percent , max_memory_percent , max_memory_kb/1024 AS max_memory_mb , used_memory_kb/1024 AS used_memory_mb , target_memory_kb/1024 AS target_memory_mb FROM sys.dm_resource_governor_resource_pools

BULK INSERT [LINEITEM_MO_final]FROM 'F:\TPCH\lineitem.tbl'

WITH ( FIELDTERMINATOR = '|', ROWTERMINATOR = '\n' );go2x faster than Heap!

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Memory mgmtCREATE RESOURCE POOL [PoolHk] WITH(min_cpu_percent=0, max_cpu_percent=100, min_memory_percent=0, max_memory_percent=70, AFFINITY SCHEDULER = AUTO)

-- Dynamically change the value of MAX_MEMORY_PERCENT ALTER RESOURCE POOL PoolHk WITH (MAX_MEMORY_PERCENT = 90 ) -- RECONFIGURE enables resource governor ALTER RESOURCE GOVERNOR RECONFIGURE ALTER RESOURCE GOVERNOR RESET STATISTICS

--- BIND the Database to the Poolexec sys.sp_xtp_bind_db_resource_pool N'TicketReservations', N'PoolHk';-- Check memory assignments:SELECT pool_id , Name , min_memory_percent , max_memory_percent , max_memory_kb/1024 AS max_memory_mb , used_memory_kb/1024 AS used_memory_mb , target_memory_kb/1024 AS target_memory_mb FROM sys.dm_resource_governor_resource_pools -- take DB offline:ALTER DATABASE [TicketReservations] SET OFFLINE WITH ROLLBACK IMMEDIATE

-- Bring the Database back Online:ALTER DATABASE [TicketReservations] SET ONLINE

-- Deleting the Resource Pool:-- sys.sp_xtp_unbind_db_resource_pool N'TicketReservations'-- DROP RESOURCE POOL [PoolHk]

A binding has been created. Take database 'TicketReservations' offline and then bring it back online to begin using resource pool 'PoolHk'.

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Extended Event tracing for in-memory OLTP:Do not use SQL Profiler for in-memory profiling

CREATE EVENT SESSION [NCsp_completed] ON SERVER ADD EVENT sqlserver.sp_statement_completed( ACTION(sqlserver.session_id,sqlserver.session_nt_username,sqlserver.sql_text,sqlserver.tsql_stack) WHERE ([sqlserver].[database_id]=(7)))

ADD TARGET package0.ring_bufferWITH ( MAX_DISPATCH_LATENCY = 1 SECONDS )GO

ALTER EVENT SESSION [NCsp_completed] ON SERVER STATE = START/STOP

DROP EVENT SESSION [NCSP_completed] ON SERVER

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242SQL 2016 Columnstore

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242

Data Loading into Columnstore: Bulk ImportKey PointsEach Bulk Import thread takes X lock on the RG. If batchsize < 100k, the rows are inserted into a delta RG otherwise a new compressed RG is generatedMinimal Logging (only when imported into compressed rowgroup)Dont specify TABLOCK when importing unlike rowstore where it is required

Compressed RGsPage compressed Delta RGsBulk Import (batchsize < 102400 )

X rowgroup lockBulk Import (batchsize >= 102400 )

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Bulk Insert into CCI

BULK INSERT [TicketReservations].[dbo].[LINEITEM_CCI]FROM 'F:\TPCH\lineitem.tbl'WITH ( --TABLOCK, FIELDTERMINATOR = '|',ROWTERMINATOR = '\n' ,BATCHSIZE = 204800

);8min 9sec123K Rows/sec!

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242SSIS load into a Clustered Columnstore table7.2GB in 9min 51sec ==100K Rows/sec

Default loads into Delta Store!

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242SSIS 2016

Set the DefaultBuffer to 10x larger , like 104857600 for direct load into RowstoreNEW! Leverage SSIS 2016 AutoAdjustbufferSize = True (For wide tables)

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242(max buffer size is 100 MB per session)

Microsoft Worldwide Partner Conference 2016 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.16/10/16 22:1221

SSIS ADO.NetDefaults into deltastore

Description of the persistent state of the row group:

INVISIBLE A hidden compressed segment in the process of being built from data in a delta store. Read actions will use the delta store until the invisible compressed segment is completed. Then the new segment is made visible, and the source delta store is removed.

OPEN A read/write row group that is accepting new records. An open row group is still in rowstore format and has not been compressed to columnstore format.

CLOSED A row group that has been filled, but not yet compressed by the tuple mover process.

COMPRESSED A row group that has filled and compressed.

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242BCP bcp [TicketReservations].[dbo].[LINEITEM_CCI] in F:\TPCH\lineitem.tbl -S VMSQL2016EE\SQL2016RTM -c -T -t"|"

Default is okay - into CCI (ignore the 1000 rows message on screen)Adding a larger batchsize:-b990000 gave best throughput of 102K Rows/sec

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242R

INSERT INTO [dbo].[LINEITEM_MO_final] EXEC sp_execute_external_script @language = N'R', @script = N'OutputDataSet 1 in production1M requests/sec in testing

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242In-Memory OLTP featuresMemory-Optimized TablesData storage in memory-optimized structuresCompiled into DLLs for efficient data accessMemory-Optimized Table TypesMemory-optimized table variables live in memory, in user DB (no tempdb, no IO)Can be used for TVPs and table variables in all stored procs (not just native)Natively Compiled Stored ProceduresOptimize performance of OLTP-style operationsNot suitable for reporting/analytics style queriesNatively Compiled Scalar User-Defined FunctionsDo not require memory-optimized tables if there is no data access

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242

TechReady 23 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.16/10/16 22:1234

Create Table DDLCREATE TABLE [Customer]( [CustomerID] INT NOT NULL PRIMARY KEY NONCLUSTERED, [Name] NVARCHAR(250) NOT NULL, [CustomerSince] DATETIME2 NULL,INDEX [ICustomerSince] NONCLUSTERED (CustomerID, CustomerSince)

)WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA);

This table is durable (default).Non-durable tables:DURABILITY=SCHEMA_ONLY

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242

35

Create Table Type DDLCREATE TYPE [Sales].[SalesOrderDetailType_inmem] AS TABLE([OrderQty] [smallint] NOT NULL,[ProductID] [int] NOT NULL,[SpecialOfferID] [int] NOT NULL,

INDEX [IX_ProductID] NONCLUSTERED HASH ([ProductID]) WITH ( BUCKET_COUNT = 8)

) WITH ( MEMORY_OPTIMIZED = ON )

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Create Procedure DDLCREATE PROCEDURE [dbo].[InsertOrder] @id INT, @date DATETIME2 WITH NATIVE_COMPILATION, SCHEMABINDING

AS BEGIN ATOMIC WITH (TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE = NDutch')

-- insert T-SQL hereEND

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242

37

Create Function DDLCREATE FUNCTION [dbo].[ufnGetAccountingEndDate_native]()RETURNS [datetime] WITH NATIVE_COMPILATION, SCHEMABINDINGAS BEGIN ATOMIC WITH (TRANSACTION ISOLATION LEVEL=SNAPSHOT, LANGUAGE=N'us_english')

RETURN DATEADD(millisecond, -2, CONVERT(datetime, '20160701', 112));

END

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242

38

Durability optionsDurability levelConfigurationPros/ConsScenariosFull durabilityDefaultDURABILITY=SCHEMA_AND_DATAPro:Every committed change is guaranteed to survive failureCon: Latency impact: every commit requires log IODefault: most scenarios need full durabilityDelayed durabilityTransaction commit timeCOMMIT WITH (DELAYED_DURABILITY=ON)Atomic block of native procedureBEGIN ATOMIC WITH (DELAYED_DURABILITY=ON, )Database levelALTER DATABASE CURRENT SET DELAYED_DURABILITY=FORCEDPro:Low latency due to no log IO in transaction execution pathEfficient log IO due to batchingCon:Limited data loss on failure (usually ~60K or ~1ms worth)Low latency requirements

Can accept some data lossORCopy of recent data exists elsewhere in case of failure

AlwaysOn auto-failover (sync replicas) with low latencyNon-durable tablesTable creationDURABILITY=SCHEMA_ONLYPro:No IO at allCon:Lose data on failureTransient data such as session stateCachingETL (staging tables)

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Allow 32-bit operating systems to access large amounts of memory!Introduced in SQL2000 Adress Windowing Extensions (AWE)

sp_configure 'awe enabled', 1Reconfigurego

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242

Analysis Services In-MemoryIn-Memory Analytics ColumnStore (NCCI)PowerPivot In-Memory for Excel

The Journey to Complete In-Memory

PowerPivot In-Memory for Excel

2012SQL Server

2008 R2SQL Server

+In-Memory Analytics ColumnStore (CCI)In-Memory OLTPAnalysis Services In-MemoryPowerPivot In-Memory for Excel

2014SQL Server41

+Real Time Operational AnalyticsIn-Memory OLTPAnalysis Services In-Memory (10x faster)PowerPivot In-Memory for Excel

Azure SQL DB and Azure SQL DW

2016SQL ServerAPS- introductionIn-Memory Analytics ColumnStore (CCI)

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242Lets start off by taking a look at how in-memory technologies have evolved in SQL Server. This may be a surprise to many of you, but our in-memory journey actually started way back in SQL Server 2008 R2, when our engineering team made a key design decision to build in-memory technology into the core data platform, rather than acquire and stitch in an in-memory solution to run in parallel to the core database.

We believe our customers want to utilize other key capabilities that SQL Server and the broader Microsoft Data Platform have to offer in conjunction with in-memory. They dont want to use a different tool set for in-memory database than they do for disk database, they still wan to use T-SQL and SQL Server management when enabling in-memory. This is the unique design approach we took back when we first started improving analytics by building in-memory into PowerPivot for billions for rows of data analysis in Excel.

Then in SQL Server 2012, we expanded our in-memory footprint with the same built-in approach by adding in-memory to Analysis Services so IT could build data models much faster, and introduced an in-memory column store that could improve query speeds by 100x.

With SQL Server 2014, we are covering the final workload by introducing an in-memory OLTP solutionor in-memory rowstoreto significantly speed transactional performance. We also enhanced the in-memory columnstore with faster performance and significantly higher data compression so memory utilization can be optimized.

SQL Server 2014 is our third release of in-memory solutions across the data platform, and in a single sku, no options or add ons.

2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.16/10/1641

Performance gains using Fast PathOpportunistic evaluation of aggregation in storage layerEfficient Aggregation on compressed/encoded data in cache-friendly execution (column at a time) Leverages SIMD Datatype = 100kIX rowgroup lockRows < 100kKey Points:Similar to Bulk Import but with single BATCHResidual rows < 100k are inserted into delta storeSQL 2016: Parallel Insert into Columnstore index

Delete bitmapInsert into select * from

Using tablock!

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242

Thank you!

Hope to See you again @ SQL Saturday!Oct 15 2016 - Universiteit Utrecht!

http://www.sqlsaturday.com/551/eventhome.aspxwww.henkvandervalk.com

2016 Microsoft Corporation. All rights reserved.

PurpleR:0 G:45 B:145Dark PurpleR:50 G:20 B:90Dark BlueR:0 G:32 B:80MagentaR:180 G:0 B:158Dark GrayR:80 G:80 B:80BlueR:0 G:120 B:215Main colorsSecondary colors (use only when necessary)Mid BlueR:0 G:24 B:143GreenR:16 G:124 B:16Light BlueR:0 G:188 B:242