sql server 2016 ctp technical deep dive
TRANSCRIPT
SQL Server 2016New innovations
There’s an opportunity to drive smarter decisions with data
The explosion of data sources...
…drives an explosion of data
…which drives businesses to learn more
2013-2020 CAGR = 41%
25B
4.0B
1.3B2010 2013 2020
Source: Forecast: Internet of Things, Endpoints and Associated Services, Worldwide, 2014. Gartner. Oct 20 2014Source: IDC “Digital Universe”, Dec. 2012
$ 1.6TData dividendEmployee
productivityOperations improvementProduct innovations
Higher sales
More people
New analytics
Diverse data
Speed
Source: IDC & Microsoft, April 2014
Data differentiates today's leading companies
Operational Database Management Systems
Data Warehouse Database Management Systems
Business Intelligence and Analytics Platforms
x86 Server Virtualization
Cloud Infrastructure as a Service
Enterprise Application Platform as a Service
Public Cloud Storage
Leader in 2014 for Gartner Magic QuadrantsMicrosoft platform leads the way on-premises and cloud
Hyperscale cloud
Deeper insights across data
Do more. Achieve more.
Performance Security Availability Scalability
Operational analyticsInsights on operational data; Works with in-memory OLTP and disk-based OLTP
In-memory OLTP enhancementsGreater T-SQL surface area, terabytes of memory supported, and greater number of parallel CPUs
Query data store Monitor and optimize query plans
Native JSON Expanded support for JSON data
Temporal database supportQuery data as points in time
Always encryptedSensitive data remains encrypted at all times with ability to query
Row-level securityApply fine-grained access control to table rows
Dynamic data maskingReal-time obfuscation of data to prevent unauthorized access
Other enhancementsAudit success/failure of database operations
TDE support for storage of in-memory OLTP tables
Enhanced auditing for OLTP with ability to track history of record changes
Enhanced AlwaysOnThree synchronous replicas for auto failover across domains
Round robin load balancing of replicas
Automatic failover based on database health
DTC for transactional integrity across database instances with AlwaysOn
Support for SSIS with AlwaysOn
Enhanced database caching Cache data with automatic, multiple TempDB files per instance in multi-core environments
Mission-critical performance
Performance Security Availability Scalability
Operational analyticsInsights on operational data; Works with in-memory OLTP and disk-based OLTP
In-memory OLTP enhancementsGreater T-SQL surface area, terabytes of memory supported, and greater number of parallel CPUs
Query data store Monitor and optimize query plans
Native JSON Expanded support for JSON data
Temporal database supportQuery data as points in time
Always encryptedSensitive data remains encrypted at all times with ability to query
Row-level securityApply fine-grained access control to table rows
Dynamic data maskingReal-time obfuscation of data to prevent unauthorized access
Other enhancementsAudit success/failure of database operations
TDE support for storage of in-memory OLTP tables
Enhanced auditing for OLTP with ability to track history of record changes
Enhanced AlwaysOnThree synchronous replicas for auto failover across domains
Round robin load balancing of replicas
Automatic failover based on database health
DTC for transactional integrity across database instances with AlwaysOn
Support for SSIS with AlwaysOn
Enhanced database caching Cache data with automatic, multiple TempDB files per instance in multi-core environments
Mission-critical performance
Operational AnalyticsWhat is operational analytics and what does it mean to you?
Operational analytics with disk-based tables
Operational analytics with In-Memory OLTP
BI and analytics
Add analytics-specific indexes
SQL ServerAnalysis Server
SQL Server
Database
Application tier
Presentation layer
IIS Server
BI analysts
Traditional operational/analytics architecture Key issues
Complex implementation
Requires two servers (capital expenditures and operational expenditures)
Data latency in analytics
More businesses demand; requires real-time analytics
BI and analytics
SQL Server
Database
Application tier
Presentation layer
IIS Server
ETL
Hourly, Daily, Weekly Database
SQL ServerRelational DW
SQL ServerAnalysis Server
BI analysts
Minimizing data latency for analytics BenefitsNo data latency
No ETL
No separate data warehouse
ChallengesAnalytics queries are resource intensive and can cause blocking
Minimizing impact on operational workloads
Sub-optimal execution of analytics on relational schema
BI and analytics
Add analytics-specific indexes
SQL ServerAnalysis Server
SQL Server
Database
Application tier
Presentation layer
IIS Server
BI analysts
Operational analytics forin-memory tables
Operational analytics: Columnstore on in-
memory tables SQL Server 2016 – CTP2 limitationYou can create columnstore index on empty tableAll columns must be included in the columnstore
No explicit delta rowgroupRows (tail) not in columnstore stay in in-memory OLTP tableNo columnstore index overhead when operating on tailBackground task migrates rows from tail to columnstore in chunks of 1 million rows not changed in last 1 hour.
Deleted Rows Table (DRT) – Tracks deleted rows
Columnstore data fully resident in memory Persisted together with operational data
No application changes required.
In-Memory OLTP table
Updateable CCI
TailDRT
Range Index
Hash index
Hot
Like Delta rowgroup
Operational Analytics: Columnstore Overhead
Operation Hash or Range Index HK-CCI
Insert Insert row into HK Insert row into HK
Delete (a)Seek row(s) to be deleted
(b)Delete the row
(a)Seek row(s) to be deleted(b)Delete the row in HK(c) If row in TAIL then return else insert <colstore-RID> into DRT
Update (a)Seek the row(s) (b)Update (delete/insert)
(a)Seek the row(s) (b)Update (delete/insert) in HK(c) If row in TAIL then return else insert <colstore-RID> into DRT
DML operations on In-Memory OLTP
Operational Analytics: Minimizing Columnstore
overhead DML OperationsKeep hot data only in-memory tablesExample – data stays hot for 1 day, 1 week…
CTP2: Work-AroundUse TF – 9975 to disable auto-compressionForce compression using a spec-proc“sp_memory_optimized_cs_migration”
Analytics queriesOffload analytics to AlwaysOn Readable secondary
In-Memory OLTP table
Updateable CCI
TailDRT
Range Index
Hash index
Hot
Like Delta rowgroup
Summary: Operational analyticsData Warehouse queries can be run on in-memory OLTP workload with no application changes.
These operations have minimal impact on OLTP workload.
Best performance and scalability available
Offloading analytics workload to Readable Secondary
Minimizing impact on OLTP Example: for HOT/WARM (Predicate)
Minimizing impact on OLTP Example: for HOT/WARM (Time Based)
Operational analytics disk-based tables
Columnstore feature summaryFunctionality:Fast equality search and short-range scans
PK/FK enforcement
Coarser concurrencyLocking granularity: Impact MERGE load or UPDATES
No support of SI/RCSI
Full support of MARS (no updates)
Improved supportability
Performance and high availabilityOnline index maintenance
Batch operations: SORT
Window Aggregates
Single
Threaded Execution
String predicate push down
Single Instruction Multiple Destination (SIMD) support
Aggregate Pushdown
Parallel Insert
Loading data is very primitive
Query offload to AlwaysOn Readable Secondaries
Operational analytics with columnstore index Key points
Create an updateable NCCI for analytics queries
Drop all other indexes that were created for analytics
No application changes
Columnstore index is maintained just like any other index
Query Optimizer will choose columnstore index where needed
Row Group 1
Row Group 2
Row Group 3
Columns A B C
Dele
te B
itm
ap
Delta Table Row
Oriented
Modifiacations of Rows Clusters Column Store
Online Merge
Insert of Rows
Operational analytics: Columnstore index
overhead
Operation B-tree (NCI) Non-clustered columnstore index (NCCI)
Insert Insert row into b-tree. Insert row into b-tree (delta store).
Delete (a)Seek row(s) to be deleted.
(b)Delete the row.
(a)Seek row in delta stores. (There can be multiple rows.(b)If found, delete row.(c) If not found, insert key into delete row
buffer.
Update (a)Seek the row(s) .(b)Update.
(a)Delete row (steps same as above).(b)Insert updated row into delta store
Operational analytics with columnstore indexDML operations on OLTP workload
--Create the tableCREATE TABLE t_account ( accountkey int PRIMARY KEY, accountdescription nvarchar (50), accounttype nvarchar(50), unitsold int);
--Create the columnstore index with a filtered conditionCREATE NONCLUSTERED COLUMNSTORE INDEX account_NCCI ON T_account (accountkey, accountdescription, unitsold) WHERE accountkey > 0;
--For this SELECT statement, the query optimizer will consider using the --columnstore index for the query plan, in which case it will also retrieve --all rows with accountkey = 0 from the rowstore table.SELECT SUM( unitsold ) FROM t_account WHERE accountkey >=0 AND accountkey < 1000;
Access hot data from btree index, warm data from columnstore index
This example uses a filtered condition (accountkey > 0) to establish which rows will be in the columnstore index
When the query optimizer chooses the filtered columnstore index, the query will return the correct results even when the results contain rows that do not meet the filtered condition. This is different from a regular nonclustered filtered index which can be used only in queries that restrict themselves to the rows present in the index.
Minimizing columnstore overhead for DML
Using Availability Groups instead of data warehouse Key points
Mission Critical Operational Workloads typically configured for High Availability using AlwaysOn Availability Groups
You can offload analytics to readable secondary replica
Secondary Replica
Secondary Replica
Secondary Replica
Primary Replica
Always on Availability Group
Summary of improvementsImprovements SQL Server 2014 SQL Server 2016
clustered columnstore index
Master copy of the data (10x compression)Only index supported; simplified analyticsNo PK/FK constraintsUniqueness can be enforced through Materialized ViewsLocking granularity for UPDATE/DELETE at rowgroup levelDDL: ALTER, REBUILD, REORGANIZE
Master copy of the data (10x compression)
Additional b-tree indexes for efficient equality, short-range searches,
and PK/FK constraints.
Locking granularity at row level using NCI index path
DDL: ALTER, REBUILD, REORGANIZE
updateable non-clustered index
Introduced in SQL Server 2012NCCI is read-only: no delete bitmap or delta storeOptimizer will choose between NCCI and NCI(s)/CI or heap-based on the cost-based modelPartitioning supported
Updateable
Ability to mix OLTP and analytics workload
Ability to create filtered NCCI
Partitioning supported
equality and short-range queries
Rowgroup elimination (when possible)
Partition-level scan (somewhat expensive)
Full index scan (expensive)
Optimizer can choose NCI on column C1; index points directly to
rowgroup
No full index scan
Covering NCI index
string predicate pushdown
Retrieve 10 million rows by converting dictionary encoded value to string
Apply string predicate on 10 million rows
Apply filter on dictionary entriesFind rows that refer to dictionary entries that qualify (R1)Find rows not eligible for this optimization (R2)Scan returns (R1 + R2) rowsFilter node applies string predicate on (R2)Row returned by Filter node = (R1 + R2’)
Support for Index MaintenanceOperation SQL Server 2014 SQL Server 2016
Removing deleted rows Requires index REBUILD Index REORGANIZERemove deleted rows from single compressed RGMerge one or more compressed RGs with deleted rowsDone ONLINE
Smaller RG size resulting fromSmaller BATCHSIZEMemory pressureIndex build residual
Index REBUILD Index REORGANIZE
Ordering rows Create clustered indexCreate columnstore index by dropping clustered index
No changes in CTP2
Query Rowgrou GranularityNo Support of RCSI or SIRecommendation: Use Read Uncommitted
Support of SI and RCSI (non blocking)
Insert Lock at row level (trickle insert)Rowgroup level for set of rows
No changes
Delete Lock at Rowgroup level Row level lock in conjunction with NCI
Update Lock at Rowgroup level Implemented as Delete/Insert
Row level lock in conjunction with NCI
AlwaysOn Failover Clustering (FCI) Fully supported Fully supported
AlwaysON Availability Groups Fully supported except readable secondary Fully supported with readable secondary
Index create/rebuild Offline Offline
In-memory OLTP enhancements
In-memory OLTP enhancements
ALTER TABLE Sales.SalesOrderDetail ALTER INDEX PK_SalesOrderID REBUILD WITH (BUCKET_COUNT=100000000)
T-SQL surface area: New
{LEFT|RIGHT} OUTER JOINDisjunction (OR, NOT)UNION [ALL]SELECT DISTINCTSubqueries (EXISTS, IN, scalar)
ALTER support
Full schema change support: add/alter/drop column/constraint
Add/drop index supported
Surface area improvements
Almost full T-SQL coverage including scaler user-defined functions
Improved scaling
Increased size allowed for durable tables; more sockets
Other improvements
MARS support
Lightweight migration reports
| ADD { <column_definition> | <table_constraint> | <table_index> } [ ,...n ]
| DROP { [ CONSTRAINT ] { constraint_name } [ ,...n ] | COLUMN { column_name } [ ,...n ] | INDEX { index_name } [ ,...n ] } [ ,...n ]
| ALTER INDEX index_name { REBUILD (WITH <rebuild_index_option>) }}
The ALTER TABLE syntax is used for making changes to the table schema, as well as for adding, deleting, and rebuilding indexes
Indexes are considered part of the table definition
Key advantage is the ability to change the BUCKET_COUNT with an ALTER INDEX statement
Altering memory-optimized tables
CREATE PROCEDURE [dbo].[usp_1]WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNERAS BEGIN ATOMIC WITH( TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE = N'us_english') SELECT c1, c2 from dbo.T1ENDGO
ALTER PROCEDURE [dbo].[usp_1]WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNERAS BEGIN ATOMIC WITH( TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE = N'us_english') SELECT c1 from dbo.T1ENDGO
You can now perform ALTER operations on natively compiled stored procedures using the ALTER PROCEDURE statement
Use sp_recompile to recompile stored procedures on the next execution
Altering natively compiled stored procedures
• CREATE PROCEDURE (Transact-SQL) • DROP PROCEDURE (Transact-SQL) • ALTER PROCEDURE (Transact-SQL) • SELECT (Transact-SQL) and INSERT SELECT statements• SCHEMABINDING and BEGIN ATOMIC (required for natively compiled stored procedures)• NATIVE_COMPILATION• Parameters and variables can be declared as NOT NULL • Table-valued parameters.• EXECUTE AS OWNER, SELF, and user.• GRANT and DENY permissions on tables and procedures.• Nesting natively compiled stored procedures• RIGHT OUTER JOIN, LEFT OUTER JOIN, INNER JOIN, and CROSS JOIN in SELECT statements• NOT, OR, and IN operators in SELECT, UPDATE and DELETE statement• UNION ALL and UNION• SELECT DISTINCT• GROUP BY clause with no aggregate functions in the SELECT clause (<select> list).• COLUMNSTORE• COLLATE
Greater Transact-SQL coverage
Improved scaling
In-memory OLTP engine has been enhanced to scale linearly on servers up to 4 sockets
Other enhancements include:• Multiple threads to persist memory
optimized tables• Multi-threaded recovery• MERGE operation• Dynamic management view
improvements to sys.dm_db_xtp_checkpoint_stats and sys.dm_db_xtp_checkpoint_files
Data Source=MSSQL; Initial Catalog=AdventureWorks; Integrated Security=SSPI; MultipleActiveResultSets=True
Setup MARS connection for memory optimized tables using the MultipleActiveResultsSets=True in your connection string
Using multiple active result sets (MARS)
In SQL Server 2016 CTP2, the storage for memory-optimized tables will be encrypted as part of enabling TDE on the database
Simply follow the same steps as you would for a disk-based database
Support for Transparent Data Encryption
Windows Operating SystemLevel Data Protection
SQL ServerInstance Level
User DatabaseLevel Database Encryption
Key
Service Master Key
Transparent Database Encryption Architecture
DPAPI encrypts the Service Master Key
MasterDatabase Level Database Encryption
Key
Service Master Key Encrypts the Database master Key for the master Database
Database Master Key of the master Database creates a certificate in the master database
The certificate encrypts the database Encryption Key in the user database
The entire user database is secured by the Datbase Encryption Key (DEK) of the user database by using transparent database encryption
Created at a time of SQL Server setup
Statement:CREAT MASTER KEY…
Statement:CREATE CERTIFICATE…
Statement:CREATE DATABASE ENCRYPTION KEY…
Statement:ALTER DATABSE… SET ENCRYPTION
New Transaction Performance Analysis
Overview report New report replaces the need to use the Management Data Warehouse to analyze which tables and stored procedures are candidates for in-memory optimization
Query StoreYour flight data recorder for your database
Have You Ever…?…had your system down/slowed down and everyone waiting for you to magically fix the problem ASAP?
…upgraded an application to the latest SQL Server version and had an issue with a plan change slowing your application down?
…had a problem with your Azure SQL Database and been unable to determine what was going wrong?
With Query Store…
I CAN get full history of query execution
I CAN quickly pinpoint the most expensive queries
I CAN get all queries that regressed
I CAN easily force better plan from history with a single line of
T-SQL
I CAN safely do server restart or upgrade
Compile MSG
Execute MSG
Query Store
Durability latency controlled by DB option
DATA_FLUSH_INTERNAL_SECONDS
Async Write-Back
Compile
Execute
SQL
Plan Store
Runtime Stats
Query Store
Schema
Query data storeCollects query texts (+ all relevant properties)
Stores all plan choices and performance metrics
Works across restarts / upgrades / recompiles
Dramatically lowers the bar for perf. Troubleshooting
New Views
Intuitive and easy plan forcing
Keeping stability while upgrading to SQL
Sever 2016
Upgrade to SQL vNextKeep
110/120 CompatLev
elFreeze plans
(optional)
Run Query Store
(establish perf.
baseline)
Move to 130
CompatLevel and unfreeze
plans
Monitor perf. and
fix regressions with
plan forcing
Monitoring Performance By Using the
Query Store The query store feature provides DBAs with insight on query plan choice and performance
Working with Query Store
/* (1) Turn ON Query Store */
ALTER DATABASE MyDB SET QUERY_STORE = ON;
/* (2) Review current Query Store parameters */SELECT * FROM sys.database_query_store_options
/* (3) Set new parameter values */ALTER DATABASE MyDBSET QUERY_STORE ( OPERATION_MODE = READ_WRITE, CLEANUP_POLICY = ( STALE_QUERY_THRESHOLD_DAYS = 30 ), DATA_FLUSH_INTERVAL_SECONDS = 3000, MAX_SIZE_MB = 500, INTERVAL_LENGTH_MINUTES = 15);
/* (4) Clear all Query Store data */ALTER DATABASE MyDB SET QUERY_STORE CLEAR;
/* (5) Turn OFF Query Store */ALTER DATABASE MyDB SET QUERY_STORE = OFF;
/* (6) Performance analysis using Query Store views*/SELECT q.query_id, qt.query_text_id, qt.query_sql_text, SUM(rs.count_executions) AS total_execution_countFROMsys.query_store_query_text qt JOIN sys.query_store_query q ON qt.query_text_id = q.query_text_id JOINsys.query_store_plan p ON q.query_id = p.query_id JOINsys.query_store_runtime_stats rs ON p.plan_id = rs.plan_idGROUP BY q.query_id, qt.query_text_id, qt.query_sql_textORDER BY total_execution_count DESC
/* (7) Force plan for a given query */exec sp_query_store_force_plan 12 /*@query_id*/, 14 /*@plan_id*/
DB-level feature exposed through T-SQL extensions
ALTER DATABASE
Catalog views (settings, compile & runtime stats)
Stored Procs (plan forcing, query/plan/stats cleanup)
Live query statisticsCollect actual metrics about query while running
View CPU/memory usage, execution time, query progress, etc.
Enables rapid identification of potential bottlenecks for troubleshooting query performance issues.
Allows drill down to live operator level statistics:
Number of generated rows
Elapsed time
Operator progress
Live warnings, etc.
Java Script Object Notation (JSON)
JSON became ubiquitous
Compact and simple data exchange format
The choice on the web
Recommended scenario
I CAN accept JSON, easily parse and store it as relational
I CAN export relational easily as JSON
I CAN correlate relational and non-relational
Motivation
[ { "Number":"SO43659", "Date":"2011-05-31T00:00:00" "AccountNumber":"AW29825", "Price":59.99, "Quantity":1 }, { "Number":"SO43661", "Date":"2011-06-01T00:00:00“ "AccountNumber":"AW73565“, "Price":24.99, "Quantity":3 }]
Number DateCustomer Price
Quantity
SO43659 2011-05-31T00:00:00 AW29825 59.99 1
SO43661 2011-06-01T00:00:00 AW73565 24.99 3
SELECT * FROM myTable
FOR JSON AUTO
SELECT * FROM OPENJSON(@json)
Data exchange with JSON
CREATE TABLE SalesOrderRecord ( Id int PRIMARY KEY IDENTITY, OrderNumber NVARCHAR(25) NOT NULL, OrderDate DATETIME NOT NULL,
JSalesOrderDetails NVARCHAR(4000) CONSTRAINT SalesOrderDetails_IS_JSON
CHECK ( ISJSON(JSalesOrderDetails)>0 ), Quantity AS CAST(JSON_VALUE(JSalesOrderDetails, '$.Order.Qty') AS int) )GO
CREATE INDEX idxJson ON SalesOrderRecord(Quantity) INCLUDE (Price);
JSON and relational
JSON is plain textISJSON guarantees
consistency
Optimize further with computed column and
INDEX
SELECT t.Id, t.OrderNumber, t.OrderDate,JSON_VALUE(t.JSalesOrderDetails,
'$.Order.ShipDate') AS ShipDateFROM SalesOrderRecord AS tWHERE ISJSON(t.JSalesOrderDetails) > 0AND t.Price > 1000
Plans for CTP3
Ability to load JSON text in tables
Extract values from JSON text
Index properties in JSON text stored in columns
-- and more
Querying JSON (CTP3)
No new data type
If you need to store it raw, store it as NVARCHAR
What is new:
Easy export: FOR JSON
Easy import: OPENJSON
Easy handling: ISJSON, JSON_VALUE
How to handle JSON?
SELECT OrderNumber AS
'Order.Number',OrderDate AS 'Order.Date'
FROM SalesOrder
FOR JSON PATH
For JSON path
[ { "Order":{ "Number":"SO43659", "Date":"2011-05 31T00:00:00“ }, }, { "Order":{ "Number":"SO43660", "Date":"2011-06-01T00:00:00“ }, }]
Query Result (JSON array)
SELECT SalesOrderNumber, OrderDate, UnitPrice, OrderQtyFROM Sales.SalesOrderHeader H INNER JOIN Sales.SalesOrderDetail D ON H.SalesOrderID = D.SalesOrderIDFOR JSON AUTO
For JSON AUTO
[ { "SalesOrderNumber":"SO43659", "OrderDate":"2011-05-31T00:00:00", "D":[ {"UnitPrice":24.99, "OrderQty":1 } ] }, { "SalesOrderNumber":"SO43659" , "D":[ { "UnitPrice":34.40 }, { "UnitPrice":134.24, "OrderQty":5 } ] }]
Query Result
CREATE TABLE OrderRecord ( Id int PRIMARY KEY IDENTITY, OrderDetails NVARCHAR(25)
CONSTRAINT OrderDetails_IS_JSON CHECK (ISJSON(OrderDetails)>1),
Quantity AS CAST(JSON_VALUE(OrderDetails,'$. Order.Qty') AS int))
JSON functions (CTP3)
CREATE INDEX idxJson ON OrderRecord(Quantity);
ISJSON (json_text) JSON_VALUE (json_text,json_path)
OPENJSON
OPENJSON (@json, N'$.Orders.OrdersArray')WITH (
Number varchar(200) N'$.Order.Number',
Date datetime N'$.Order.Date',Customer varchar(200)
N'$.AccountNumber',Quantity int
N'$.Item.Quantity')
{"Orders": { "OrdersArray": [ { "Order": { "Number":"SO43659", "Date":"2011-05-31T00:00:00“ }, "AccountNumber":"AW29825“, "Item": { "Price":2024.9940, "Quantity":1 } }, { "Order":{ “Number":"SO43661", "Date":"2011-06-01T00:00:00“ }, "AccountNumber":"AW73565“, "Item": { "Price":2024.9940, "Quantity":3 } }]} }
Number Date Customer Quantity
SO43659 2011-05-31T00:00:00
AW29825 1
SO43661 2011-06-01T00:00:00
AW73565 3
SQL Server and Azure DocumentDB
Schema-free NoSQL document store
Scalable transactional processing for rapidly
changing apps
Premium relational DB capable to
exchange data with modern apps & services
Derives unified insights from structured/unstructured data
SQL Server 2016
JSON JSJSJSON
Key takeaways
Temporal: data audit and time-based analysis
Query Store: performance troubleshooting and tuning
JSON: interop with modern services and applications
Available in the cloud and on-prem
QUERY STORE
JSON
TemporalQuery back in time
Real data sources are dynamic Historical data may be critical to business success
Traditional databases fail to provide required insights
Workarounds are…Complex, expensive, limited, inflexible, inefficient
SQL Server 2016 makes life easyNo change in programming model
New Insights
Why Temporal
Time Travel Data Audit
Slowly Changing Dimensions
Repair record-level corruptions
No change in programming model
New Insights
INSERT / BULK INSERT
UPDATE
DELETE
MERGE
DML SELECT * FROM temporal
Querying
How to start with temporal
CREATE temporal TABLE PERIOD FOR SYSTEM_TIME…
ALTER regular_table TABLE ADD PERIOD…
DDL
FOR SYSTEM_TIMEAS OF FROM..TOBETWEEN..ANDCONTAINED IN
Temporal Querying
Temporal table (actual data)
Insert / Bulk Insert
* Old versions
Update */ Delete *
How system-time works?
History Table
Temporal table (actual data)
Temporal Queries * (Time travel,etc.)
How system-time works?
History Table
Regular queries (current data)
* Include Historical Version
DepNumDepName
MngrID From To
A001 Marketing 5 2005 2008
A002 Sales 2 2005 2007
A003 Consulting 6 2005 2006
A003 Consulting 10 2009 2012
DepNum DepName
MngrID
A001 Marketing 5
A001 Marketing 6
A002 Sales 2
A002 Sales 5
A003 Consulting 6
A003 Consulting 10
DepNumDepName
MngrID From To
A001 Marketing 6 2008 ∞
A002 Sales 5 2007 ∞A001
A002
A003
period of validity current time
∞
∞
Department (current)
Department (history)
Department (current + history)
2005 2015
SELECT * FROM DepartmentSELECT * FROM Department FOR SYSTEM_TIMEBETWEEN '2006.01.01' AND '2007.01.01'
SELECT * FROM Department FOR SYSTEM_TIMECONTAINED IN ('2007.01.01', '2009.01.01')SELECT * FROM Department FOR SYSTEM_TIME AS OF '2006.01.01'
Getting insights from temporal – AS OF
A001
A002
A003
“Get actual row versions”AS OFBETWEEN..ANDCONTAINED IN
SELECT * FROM Department FOR SYSTEM_TIME AS OF '2010.01.01'
Facts:
1. History is much bigger than actual data
2. Retained between 3 and 10 years
3. “Warm”: up to a few weeks/months
4. “Cold”: rarely queried
Solution:
History as a stretch table:
PeriodEnd < “Now - 6 months”
Azure SQL Database
Provides correct information about stored facts at any point in time, or between 2 points in time.
There are two orthogonal sets of scenarios with regards to temporal data:
System(transaction)-time
Application-time
SELECT * FROM Person.BusinessEntityContactFOR SYSTEM_TIME BETWEEN @Start AND @EndWHERE ContactTypeID = 17
Performance
Temporal database support - BETWEEN
Performance Security Availability Scalability
Operational analyticsInsights on operational data; Works with in-memory OLTP and disk-based OLTP
In-memory OLTP enhancementsGreater T-SQL surface area, terabytes of memory supported, and greater number of parallel CPUs
Query data store Monitor and optimize query plans
Native JSON Expanded support for JSON data
Temporal database supportQuery data as points in time
Always encryptedSensitive data remains encrypted at all times with ability to query
Row-level securityApply fine-grained access control to table rows
Dynamic data maskingReal-time obfuscation of data to prevent unauthorized access
Other enhancementsAudit success/failure of database operations
TDE support for storage of in-memory OLTP tables
Enhanced auditing for OLTP with ability to track history of record changes
Enhanced AlwaysOnThree synchronous replicas for auto failover across domains
Round robin load balancing of replicas
Automatic failover based on database health
DTC for transactional integrity across database instances with AlwaysOn
Support for SSIS with AlwaysOn
Enhanced database caching Cache data with automatic, multiple TempDB files per instance in multi-core environments
Mission-critical performance
Always Encrypted
Prevents Data Disclosure
Client-side encryption of sensitive data using keys that are never given to the database system.
Queries on Encrypted Data
Support for equality comparison, incl. join, group by and distinct operators.
Application Transparency
Minimal application changes via server and client library enhancements.
Allows customers to securely store sensitive data outside of their trust boundary.Data remains protected from high-privileged, yet unauthorized users.
Benefits of Always Encrypted
dbo.Patients
Jane Doe
Name
243-24-9812
SSN
USA
Country
Jim Gray 198-33-0987
USA
John Smith 123-82-1095
USA
dbo.Patients
Jane Doe
Name
1x7fg655se2e
SSN
USA
Jim Gray 0x7ff654ae6d
USA
John Smith 0y8fj754ea2c
USA
Country
Result Set
Jim Gray
Name
Jane Doe
Name
1x7fg655se2e
SSN
USA
Country
Jim Gray 0x7ff654ae6d
USA
John Smith 0y8fj754ea2c
USA
dbo.Patients
SQL Server
Query
Trusted
Apps
SELECT Name FROM Patients WHERE SSN=@SSN@SSN='198-33-0987'
Result Set
Jim Gray
Name
SELECT Name FROM Patients WHERE SSN=@SSN
@SSN=0x7ff654ae6d
Column Encryption
Key
EnhancedADO.NET
Library
ColumnMasterKey
Client side
Always EncryptedHelp protect data at rest and in motion, on-premises & cloud
ciphertext
Randomized encryptionEncrypt('123-45-6789') = 0x17cfd50aRepeat: Encrypt('123-45-6789') = 0x9b1fcf32Allows for transparent retrieval of encrypted data but NO operationsMore secure
Deterministic encryptionEncrypt('123-45-6789') = 0x85a55d3fRepeat: Encrypt('123-45-6789') = 0x85a55d3fAllows for transparent retrieval of encrypted data AND equality comparison
E.g. in WHERE clauses and joins, distinct, group by
Two types of encryption available
Randomized encryption uses a method that encrypts data in a less predictable manner
Deterministic encryption uses a method which always generates the same encrypted value for any given plain text value
Types of Encryption for Always Encrypted
Security Officer
1. Generate CEKs and Master Key
2. Encrypt CEK
3. Store Master Key Securely
4. Upload Encrypted CEK to DB
CMK Store:Certificate Store
HSMAzure Key Vault
…
EncryptedCEK
Column Encryption Key(CEK)
ColumnMaster Key(CMK)
Key Provisioning
CMK
DatabaseEncrypted CEK
Param
Encryption Type/ Algorithm
Encrypted CEK Value
CMK Store Provider Name CMK Path
@Name
Non-DET/ AES 256
CERTIFICATE_STORE
Current User/ My/f2260…
EXEC sp_execute_sql N'SELECT * FROM Customers WHERE SSN = @SSN', @params = N'@SSN VARCHAR(11)', @SSN=0x7ff654ae6d
Param
Encryption Type/ Algorithm
Encrypted CEK Value
CMK Store Provider Name CMK Path
@SSN DET/ AES 256
CERTIFICATE_STORE
Current User/ My/f2260…
Enhanced ADO.NET
Plaintext CEKCache
exec sp_describe_parameter_encryption @params = N'@SSN VARCHAR(11)', @tsql = N'SELECT * FROM Customers WHERE SSN = @SSN'
Result set (ciphertext)
Name
Jim Gray
Result set (plaintext)
using (SqlCommand cmd = new SqlCommand("SELECT Name FROM Customers WHERE SSN = @SSN“ , conn)){ cmd.Parameters.Add(new SqlParameter( "@SSN", SqlDbType.VarChar, 11).Value = "111-22-3333"); SqlDataReader reader = cmd.ExecuteReader();
Client - Trusted SQL Server - Untrusted
En
crypti
on
meta
data
Name
0x19ca706fbd9
En
crypti
on
meta
data
CMK Store
Example
Select columns to be encrypted Analyze
schema and application queries to detect conflicts (build time)
Set up the keys: master & CEK
Static schema analysis tool (SSDT only)
UI for selecting columns (no automated data classification)
Key setup tool to automate selecting CMK, generating and encrypting CEK and uploading key metadata to the database
Setup (SSMS or SSDT)User Experience: SSMS or SSDT (Visual Studio)
Existing App – SetupUser Experience: SSMS or SSDT (Visual Studio)
UI for selecting columns (no automated data classification)
Select candidate columns to be encrypted
Analyze schema and application queries to detect conflicts and identify optimal encryption settings
Set up the keys
Encrypt selected columns while migrating the database to a target server (e.g. in Azure SQL Database
Key Setup tool to streamline selecting CMK, generating and encrypting CEK and uploading key metadata to the database
Encryption tool creating new (encrypted) columns, copying data from old (plain text) columns, swapping columns and re-creating dependencies
Select desired encryption settings for selected columns
UI for configuring encryption settings on selected columns (accepting/editing recommendations from the analysis tool)
Schema/workload analysis tool analyzing the schema and profiler logs
Data remains encrypted during query
Summary: Always encryptedProtect data at rest and in motion, on-premises & cloud Capability
ADO.Net client library provides transparent client-side encryption, while SQL Server executes T-SQL queries on encrypted data
Benefits
Sensitive data remains encrypted and queryable at all times on-premises & cloud.
Unauthorized users never have access to data or keys
No application changes
Apps TCE-enabledADO .NET library
SQL ServerEncrypte
d query
Columnarkey
No app changes
Master key
Row-Level Security.SQL Server 2016SQL Database
Fine-grained access control over specific rows in a database table
Help prevent unauthorized access when multiple users share the same tables, or to implement connection filtering in multitenant applications
Administer via SQL Server Management Studio or SQL Server Data Tools
Enforcement logic inside the database and schema bound to the table.
Protect data privacy by ensuring the right access across rows
Status: Public preview
SQL Database
Customer 1
Customer 2
Customer 3
Row-level security
Fine-grained access control
Keeping multi-tenant databases secure by limiting access by other users who share the same tables.
Application transparency
RLS works transparently at query time, no app changes needed.
Compatible with RLS in other leading products.
Centralized security logic
Enforcement logic resides inside database and is schema-bound to the table it protects providing greater security. Reduced application maintenance and complexity.
Store data intended for many consumers in a single database/table while at the same time restricting row-level read & write access based on users’ execution context.
Benefits of row-level security
CREATE SECURITY POLICY mySecurityPolicyADD FILTER PREDICATE dbo.fn_securitypredicate(wing, startTime,
endTime) ON dbo.patients
Predicate functionUser-defined inline table-valued function (iTVF) implementing security logicCan be arbitrarily complicated, containing joins with other tables
Security predicateApplies a predicate function to a particular table (SEMIJOIN APPLY)Two types: filter predicates and blocking predicates
Security policyCollection of security predicates for managing security across multiple tables
RLS Concepts
Security
CREATE FUNCTION dbo.fn_securitypredicate(@wing int)
RETURNS TABLE WITH SCHEMABINDING AS
return SELECT 1 as [fn_securitypredicate_result] FROM
StaffDuties d INNER JOIN Employees e
ON (d.EmpId = e.EmpId)
WHERE e.UserSID = SUSER_SID()
AND @wing = d.Wing;
CREATE SECURITY POLICY dbo.SecPol
ADD FILTER PREDICATE dbo.fn_securitypredicate(Wing)
ON Patients
WITH (STATE = ON)
Fine-grained access control over rows in a table based on one or more pre-defined filtering criteria, e.g., user’s role or clearance level in organization.
Concepts:Predicate function
Security policy
Example
Two
App user (e.g., nurse) selects from Patients tableThree
Security Policy transparently rewrites query to apply filter predicate
Database Policy Manager
CREATE FUNCTION dbo.fn_securitypredicate(@wing int) RETURNS TABLE WITH SCHEMABINDING AS return SELECT 1 as [fn_securitypredicate_result] FROM StaffDuties d INNER JOIN Employees e ON (d.EmpId = e.EmpId) WHERE e.UserSID = SUSER_SID() AND @wing = d.Wing;
CREATE SECURITY POLICY dbo.SecPol ADD FILTER PREDICATE dbo.fn_securitypredicate(Wing) ON Patients WITH (STATE = ON)
FilterPredicate:
INNER JOIN…
SecurityPolicy
Application
Patients
One
Policy manager creates filter predicate and security policy in T-SQL, binding the predicate to the Patients table
Nurse
SELECT * FROM Patients
SELECT * FROM Patients SEMIJOIN APPLY dbo.fn_securitypredicate(patients.Wing);
SELECT Patients.* FROM Patients, StaffDuties d INNER JOIN Employees e ON (d.EmpId = e.EmpId) WHERE e.UserSID = SUSER_SID() AND Patients.wing = d.Wing;
RLS in Three Steps
-- The following syntax creates a security policy with a filter predicate for the Customer table, and leaves the security policy disabledCREATE SECURITY POLICY [FederatedSecurityPolicy]
ADD FILTER PREDICATE [rls].[fn_securitypredicate]([CustomerId])
ON [dbo].[Customer];
-- Create a new schema and predicate function, which will use the application user ID stored in CONTEXT_INFO to filter rows.CREATE FUNCTION rls.fn_securitypredicate (@AppUserId int)
RETURNS TABLE WITH SCHEMABINDING
AS RETURN (SELECT 1 AS fn_securitypredicate_result WHERE
DATABASE_PRINCIPAL_ID() = DATABASE_PRINCIPAL_ID('dbo') -- application context
AND CONTEXT_INFO() = CONVERT(VARBINARY(128), @AppUserId); GO
Creates a security policy for row level security.
The following examples demonstrate the use of the CREATE SECURITY POLICY syntax.
For an example of a complete security policy scenario, see Row Level Security.
Create Security Policy
Dynamic Data MaskingSQL Server 2016SQL Database
Configuration made easy in the new Azure portal
Policy-driven at the table and column level, for a defined set of users
Data masking applied in real-time to query results based on policy
Multiple masking functions available (e.g. full, partial) for various sensitive data categories (e.g. Credit Card Numbers, SSN, etc.)
SQL DatabaseSQL Server 2016 CTP2
Table.CreditCardNo
4465-6571-7868-5796
4468-7746-3848-1978
4484-5434-6858-6550
Real-time data masking; partial masking
Status: Public preview
Dynamic Data MaskingPrevent the abuse of sensitive data by hiding it from users
Audit success/failure of database operations
Enhanced auditing for OLTP with ability to track history of record changes
Transparent Data Encryption support for storage of In-memory OLTP Tables
Backup encryption now supported with compression
Other security enhancements
MSDN Documentation https://msdn.microsoft.com/en-us/library/dn765131.aspx
Security Center for SQL Server Database & SQL Database\https://msdn.microsoft.com/en-us/bb510589.aspx
SQL Server Security BlogAdditional examples, useful tips and tricks
http://blogs.msdn.com/b/sqlsecurity/
SQL Server Label Security ToolkitUpdated version to take advantage of RLS coming later in CY15
http://sqlserverlst.codeplex.com/
Security resources
Performance Security Availability Scalability
Operational analyticsInsights on operational data; Works with in-memory OLTP and disk-based OLTP
In-memory OLTP enhancementsGreater T-SQL surface area, terabytes of memory supported, and greater number of parallel CPUs
Query data store Monitor and optimize query plans
Native JSON Expanded support for JSON data
Temporal database supportQuery data as points in time
Always encryptedSensitive data remains encrypted at all times with ability to query
Row-level securityApply fine-grained access control to table rows
Dynamic Data MaskingReal-time obfuscation of data to prevent unauthorized access
Other enhancementsAudit success/failure of database operations
TDE support for storage of in-memory OLTP tables
Enhanced auditing for OLTP with ability to track history of record changes
Enhanced AlwaysOnThree synchronous replicas for auto failover across domains
Round robin load balancing of replicas
Automatic failover based on database health
DTC for transactional integrity across database instances with AlwaysOn
Support for SSIS with AlwaysOn
Enhanced database caching Cache data with automatic, multiple TempDB files per instance in multi-core environments
Mission-critical performance
Enhanced AlwaysOn
Greater scalability:Load balancing readable secondaries
Increased number of auto-failover targets
Log transport performance
Improved manageability:DTC support
Database-level health monitoring
Group managed service account
AG_Listener
New York(Primary)
Asynchronous data Movement
Synchronous data Movement
Unified HA Solution
Enhanced AlwaysOn Availability Groups
AG
Hong Kong
(Secondary)
AG
New Jersey
(Secondary)
AG
DR Site Computer2
Computer3
Computer4
Computer5
READ_ONLY_ROUTING_LIST=(('COMPUTER2', 'COMPUTER3', 'COMPUTER4'), 'COMPUTER5')
Primary Site
Computer1 (Primary)
Readable Secondary load balancing
Ability to rebuild online indexes in a single partition provides partition-level control for users who need continual access to the database.
Allows database administrators to specify whether or not to terminate processes that block their ability to lock tables.
SQL Server 2016 CTP2 now provides 100% uptime with enhanced online database operations when conducting ALTER or TRUNCATE operations on tables.
Enhanced online operations
Performance Security Availability Scalability
Operational analyticsInsights on operational data; Works with in-memory OLTP and disk-based OLTP
In-memory OLTP enhancementsGreater T-SQL surface area, terabytes of memory supported, and greater number of parallel CPUs
Query data store Monitor and optimize query plans
Native JSON Expanded support for JSON data
Temporal database supportquery data as points in time
Always encryptedSensitive data remains encrypted at all times with ability to query
Row-level securityApply fine-grained access control to table rows
Dynamic Data MaskingReal-time obfuscation of data to prevent unauthorized access
Other enhancementsAudit success/failure of database operations
TDE support for storage of in-memory OLTP tables
Enhanced auditing for OLTP with ability to track history of record changes
Enhanced AlwaysOnThree synchronous replicas for auto failover across domains
Round robin load balancing of replicas
Automatic failover based on database health
DTC for transactional integrity across database instances with AlwaysOn
Support for SSIS with AlwaysOn
Enhanced database caching Cache data with automatic, multiple TempDB files per instance in multi-core environments
Mission-critical performance
Scalability improvements
Supports caching data with automatic, multiple TempDB files per instance in multi-core environments
Reduces metadata and allocation contention for TempDB workloads, improving performance and scalability
Enhanced database caching
Enhanced support for Windows ServerHardware Acceleration for TDE Encryption/Decryption
Implements Microsoft’s next gen cryptography
Takes advantage of specialized microprocessor instructions
Can improve performance 3x to 10x
Parallelizing the Decryption Built-in Function to Improve Read Performance
Decryption now supported as parallelizable (used to be sequential only)
Results in dramatically better response times for queries with encrypted data columns
Switches
Networking
Compute
Rack 2
Edge
Storage
Networking
Compute
Rack 1
Edge
Storage
Management
Switches
Networking
Compute
Rack 3
Edge
Storage
Switches
Networking
Compute
Rack 4
Edge
Storage
Cloud Platform System advantagesPartnership with Dell.
Combination of optimized hardware/software.
Designed specifically to reduce the complexity and risk of implementing a self-service cloud.
CPS can go from delivery to live within days—not months—and lets service providers and enterprises move up the stack to focus on delivering services to users
Access any data Scale and manage
Powerful Insights
Advanced analytics
PolyBase Insights from data across SQL Server and Hadoop with simplicity of T-SQL
Enhanced SSISDesigner support for previous SSIS versions
Support for Power Query
Enterprise-grade Analysis Services Enhanced performance and scalability for analysis services
Single SSDT in Visual Studio 2015 (CTP3)Build richer analytics solutions as part of your development projects in Visual Studio
Enhanced MDS Excel add-in 15x faster; more granular security roles; archival options for transaction logs; and reuse entities across models
Mobile BIBusiness insights for your on-premises data through rich visualization on mobile devices with native apps for Windows, iOS and Android
Enhanced Reporting ServicesNew modern reports with rich visualizations
R integration (CTP3)Bringing predictive analytic capabilities to your relational database
Analytics libraries (CTP3)Expand your “R” script library with Microsoft Azure Marketplace
Deeper insights across data
Access any data Scale and manage
Powerful Insights
Advanced analytics
PolyBase Insights from data across SQL Server and Hadoop with simplicity of T-SQL
Enhanced SSISDesigner support for previous SSIS versions
Support for Power Query
Enterprise-grade Analysis Services Enhanced performance and scalability for analysis services
Single SSDT in Visual Studio 2015 Build richer analytics solutions as part of your development projects in Visual Studio
Enhanced MDS Excel add-in 15x faster; more granular security roles; archival options for transaction logs; and reuse entities across models
Mobile BIBusiness insights for your on-premises data through rich visualization on mobile devices with native apps for Windows, iOS and Android
Enhanced Reporting ServicesNew modern reports with rich visualizations
R integration (CTP3)Bringing predictive analytic capabilities to your relational database
Analytics libraries (CTP3)Expand your “R” script library with Microsoft Azure Marketplace
Deeper insights across data
PolyBase for SQL Server 2016
Query relational and non-relational data, on-premises
and in Azure
Apps
T-SQL query
SQL Server
Hadoop
Capability
T-SQL for querying relational and non-relational data across SQL Server and Hadoop
Benefits
New business insights across your data lake
Leverage existing skillsets and BI tools
Faster time to insights and simplified ETL process
PolyBaseQuery relational and non-relational data with T-SQL
Prerequisites for installing PolyBase
64-bit SQL Server Evaluation edition
Microsoft .NET Framework 4.0.
Oracle Java SE RunTime Environment (JRE) version 7.51 or higher
NOTE: Java JRE version 8 does not work.
Minimum memory: 4GB
Minimum hard disk space: 2GB
Using the installation wizard for PolyBase
Run SQL Server Installation Center. (Insert SQL Server installation media and double-click Setup.exe.)
Click Installation, then click New Standalone SQL Server installation or add features
On the feature selection page, select PolyBase Query Service for External Data.
On the Server Configuration Page, configure the PolyBase Engine Service and PolyBase Data Movement Service to run under the same account.
-- Run sp_configure ‘hadoop connectivity’ -- and set an appropriate valuesp_configure
@configname = 'hadoop connectivity', @configvalue = 7;
GORECONFIGUREGO
-- List the configuration settings for -- one configuration namesp_configure @configname='hadoop connectivity';GO
Option values
0: Disable Hadoop connectivity
1: Hortonworks HDP 1.3 on Windows Server Azure blob storage (WASB[S])
2: Hortonworks HDP 1.3 on Linux
3: Cloudera CDH 4.3 on Linux
4: Hortonworks HDP 2.0 on Windows Server Azure blob storage (WASB[S])
5: Hortonworks HDP 2.0 on Linux
6: Cloudera 5.1 on Linux
7: Hortonworks 2.1 and 2.2 on Linux Hortonworks 2.2 on Windows Server Azure blob storage (WASB[S])
Choose Hadoop data source with sp_configure
Start the PolyBase services
After running for sp_configure, you must stop and restart the SQL Server engine service
Run services.msc
Find the services shown below and stop each one
Restart the services
-- Using credentials on database requires enabling-- traceflag DBCC TRACEON(4631,-1)
-- Create a master keyCREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo';
CREATE CREDENTIAL WASBSecret ON DATABASE WITH IDENTITY = 'pdw_user', Secret = 'mykey==';
-- Create an external data source (Azure Blob Storage)-- with the credentialCREATE EXTERNAL DATA SOURCE Azure_Storage WITH ( TYPE = HADOOP, LOCATION ='wasb[s]://[email protected]/path’,
CREDENTIAL = WASBSecret)
Type methods for providing credentials
Core-site.xml in installation path of SQL Server - <SqlBinRoot>\Polybase\Hadoop\Conf
Credential object in SQL Server for higher security
NOTE: The syntax for a database-scoped credential (CREATE CREDENTIAL … ON DATABASE) is temporary and will change in the next release. This new feature is documented only in the examples in the CTP2 content, and will be fully documented in the next release.
Configure PolyBase for Azure blob storage
-- Create an external data source (Hadoop)CREATE EXTERNAL DATA SOURCE hdp2 with (TYPE = HADOOP, LOCATION ='hdfs://10.xxx.xx.xxx:xxxx', RESOURCE_MANAGER_LOCATION='10.xxx.xx.xxx:xxxx')
CTP2 supports the following Hadoop distributions
Hortonworks HDP 1.3, 2.0, 2.1, 2.2 for both Windows and Linux
Cloudera CDH 4.3, 5.1 on Linux
Create a reference to a Hadoop cluster
-- Create an external file format -- (delimited text file)CREATE EXTERNAL FILE FORMAT ff2 WITH (FORMAT_TYPE = DELIMITEDTEXT, FORMAT_OPTIONS (FIELD_TERMINATOR ='|', USE_TYPE_DEFAULT = TRUE))
CTP2 supports the following file formats
Delimited text
Hive RCFile
Hive ORC
Define the external file format
-- Create an external table pointing to file stored in HadoopCREATE EXTERNAL TABLE [dbo].[CarSensor_Data] ( [SensorKey] int NOT NULL, [CustomerKey] int NOT NULL, [GeographyKey] int NULL, [Speed] float NOT NULL, [YearMeasured] int NOT NULL)WITH (LOCATION='/Demo/car_sensordata.tbl', DATA_SOURCE = hdp2, FILE_FORMAT = ff2, REJECT_TYPE = VALUE, REJECT_VALUE = 0
The external table provides a T-SQL reference to the data source used to:
Query Hadoop or Azure blob storage data with Transact-SQL statements
Import and store data from Hadoop or Azure blob storage into your SQL Server database
Create an external table to the data source
-- Create statistics on an external table. CREATE STATISTICS StatsForSensors ON CarSensor_Data(CustomerKey, Speed)
In CTP2, you can optimize query execution against the external tables using statistics
Optimize queries by adding statistics
SELECT DISTINCT C.FirstName, C.LastName, C.MaritalStatusFROM Insurance_Customer_SQL INNER JOIN (
SELECT * FROM SensorData_ExternalHDP WHERE Speed > 35UNION ALLSELECT * FROM SensorData_ExternalHDP2 WHERE Speed > 35) AS SensorDON C.CustomerKey = SensorD.CustomerKey
External tables referring to data in 2 HDP Hadoop clusters
SQL Server table
Query Capabilities (1)Joining relational and external data
SELECT DISTINCT C.FirstName, C.LastName, C.MaritalStatusFROM Insurance_Customer_SQL -- table in SQL Server…OPTION (FORCE EXTERNALPUSHDOWN) – push-down computationCREATE EXTERNAL DATA SOURCES ds-hdp WITH .( TYPE = Hadoop, LOCATION = “hdfs://10.193.27.52:8020”, Resources_Manager_Location = ‘10.193.27.52:8032’);
Pushing ComputeEither on data source level or Per-query basis using new query hints
Query Capabilities (2)Push-Down Computation
SSIS improvements
SSIS improvements for SQL Server 2016
CTP2
AlwaysOn support
Incremental deployment of packages
Improved project upgrade support
CTP3
Designer improvements
One designer multi-version support
OData V4 support
Power Query as a data source
AlwaysOn Availability
Groups Secondary for
SSISDBNew York(Primary)
New Jersey
(Secondary)
SSIS DB
SSISDB
SQL Server 2012SSIS Project XSQL Server 2016SSIS Project X
Improved project upgrade
AlwaysOn for SSIS Catalog
Steps to configure AlwaysOn for SSIS catalog
Create Integrations Services catalog
Add SSISDB to an AlwaysOn Availability Group
Enable SSIS support for AlwaysOn
private static void Main(string[] args) { // Connection string to SSISDB var connectionString = "Data Source=.;Initial Catalog=SSISDB;Integrated Security=True;MultipleActiveResultSets=false";
using (var sqlConnection = new SqlConnection(connectionString)) { sqlConnection.Open();
var sqlCommand = new SqlCommand { Connection = sqlConnection, CommandType = CommandType.StoredProcedure, CommandText = "[catalog].[deploy_packages]" };
var packageData = Encoding.UTF8.GetBytes(File.ReadAllText(@"C:\Test\Package.dtsx"));
// DataTable: name is the package name without extension and package_data is byte array of package. var packageTable = new DataTable(); packageTable.Columns.Add("name", typeof(string)); packageTable.Columns.Add("package_data", typeof(byte[])); packageTable.Rows.Add("Package", packageData);
// Set the destination project and folder which is named Folder and Project. sqlCommand.Parameters.Add(new SqlParameter("@folder_name", SqlDbType.NVarChar, ParameterDirection.Input, "Folder", -1)); sqlCommand.Parameters.Add(new SqlParameter("@project_name", SqlDbType.NVarChar, ParameterDirection.Input, "Project", -1)); sqlCommand.Parameters.Add(new SqlParameter("@packages_table", SqlDbType.Structured, ParameterDirection.Input, packageTable, -1));
var result = sqlCommand.Parameters.Add("RetVal", SqlDbType.Int); result.Direction = ParameterDirection.ReturnValue;
sqlCommand.ExecuteNonQuery(); } }
Deployment options in CTP2
Integration Services Deployment Wizard
SQL Server Management Studio
deploy_packages stored procedure
Object model API
Deployment options in CTP3
SQL Server Data Tools for Business Intelligence
Deploy packages to Integration Services server
Access any data Scale and manage
Powerful Insights
Advanced analytics
PolyBase Insights from data across SQL Server and Hadoop with simplicity of T-SQL
Enhanced SSISDesigner support for previous SSIS versions
Support for Power Query
Enterprise-grade Analysis Services Enhanced performance and scalability for analysis services
Single SSDT in Visual Studio 2015 (CTP3)Build richer analytics solutions as part of your development projects in Visual Studio
Enhanced MDS Excel add-in 15x faster; more granular security roles; archival options for transaction logs; and reuse entities across models
Mobile BIBusiness insights for your on-premises data through rich visualization on mobile devices with native apps for Windows, iOS and Android
Enhanced Reporting ServicesNew modern reports with rich visualizations
R integration (CTP3)Bringing predictive analytic capabilities to your relational database
Analytics libraries (CTP3)Expand your “R” script library with Microsoft Azure Marketplace
Deeper insights across data
Enterprise grade Analysis Services
Scale your tabular models with support for high end servers
More memory
More cores
Enhanced Analysis ServicesDeliver high performance and scalability for your BI solutions
High-end server hardware
Capability
Parallel partition processing
NUMA optimization for tabular models
On-demand loading and paging
Tabular and MOLAP modeling enhancements
Benefits
Better performance and higher scale
Lower TCO – more efficient utilization of existing hardware capabilities
No application changes
SSAS Enterprise Readiness: MOLAP
Netezza as a data source
Performance improvements (unnatural hierarchies, ROLAP distinct count, drill-through, semi-additive measures)
Detect MOLAP index corruption using DBCC
High-end Server Hardware
SSAS Enterprise Readiness: Tabular
Processing optimizations
Parallel partition processing
Support for Power Query in AS engine
Support for BI directional cross filtering (M:M)
Calculated tables, Date Table
Translations
Tabular scripting language
New DAX functions (DATEDIFF, GEOMEAN, PERCENTILE, PRODUCT, XIRR, XNPV, many more..)
Query engine optimizations
New DAX functions for high perf. reporting
Direct Query perf. optimizations
DimDepartmentGr…DimDepartmentGroupKey…ParentDepartmentGr…
DepartmentGroupNa…
DimAccount
AccountKey
ParenAccountKey
AccountCodeAlter…
ParenAccountCod…
AccountDescription
FactFinance
FinanceKey
DateKey
OrganizationKey
DepartmentGroup…
ScenarioKey
DimOrganization
OrganizationKey
ParnentOrganization
PercentageOfOwn…
OrganizationName
CurrencyKey
DimAccount
ScenarioKey
ScenarioName
*
*
1
*
1
*1
*1
SSAS Rich Modeling Platform: Tabular
Unified Visual Studio Data Tools (CTP3)
Capability
SSDT-BI and SSMS for Visual Studio 2015
Rich data modeling enhancements
New DAX analytical functions
MDX Query Plan tool for performance optimizations and troubleshooting
Benefits
Lower BI solution development costs and faster delivery cycles
Support for most recent version of Visual Studio to create BI solutions
Lower DAX expression complexity through new DAX and SuperDAX functions
{JSON}
Powerful Insights
Rich BI Application PlatformDevelop and deliver BI solutions faster and at lower costs
Master Data Services improvements
Performance and Scale
Overall performance and scale improvements
Target of 15x performance improvement for Excel add-in
Increased performance for bulk entity based staging operations
Security Improvements
New security roles for more granular permissions around read, write and delete
Multiple system administrators
Manageability and ModelingArchival of Transaction Logs
Reuse of entities across models
Support for compound keys
Allow the Name and Code attributes to be renamed
MDS Improvements
Access any data Scale and manage
Powerful Insights
Advanced analytics
PolyBase Insights from data across SQL Server and Hadoop with simplicity of T-SQL
Enhanced SSISDesigner support for previous SSIS versions
Support for Power Query
Enterprise-grade Analysis Services Enhanced performance and scalability for analysis services
Single SSDT in Visual Studio 2015 Build richer analytics solutions as part of your development projects in Visual Studio
Enhanced MDS Excel add-in 15x faster; more granular security roles; archival options for transaction logs; and reuse entities across models
Mobile BIBusiness insights for your on-premises data through rich visualization on mobile devices with native apps for Windows, iOS and Android
Enhanced Reporting ServicesNew modern reports with rich visualizations
R integration (CTP3)Bringing predictive analytic capabilities to your relational database
Analytics libraries (CTP3)Expand your “R” script library with Microsoft Azure Marketplace
Deeper insights across data
Mobile BI improvements
Mobile BI apps for SQL Server – formally
Datazen For on-premises implementations - optimized for SQL Server.
Rich, interactive data visualization on all major mobile platforms
No additional cost for SQL Server Enterprise Edition customers 2008 or later and Software Assurance
Data visualization and publishing
Create beautiful visualizationsand KPIs with a touch-based designer
Connect to the Mobile BI for SQL Server server toaccess SQL Server data
Publish for access by others
Datazen architecture overviewServer, Publisher and Mobile apps
SQL Server
Analysis Services
File
Data sources
Active directory controller
Authentication
Internet boundary
Viewer
Apps
Publisher
App
Web
browser
Datazen Enterprise Server
Core Service
Data Acquisition Service
Web applications
Enterprise environment
Mobile BI for SQL Server summary
Beautiful visualizations
KPI repository
Create once - publish to any device
Native apps for all platforms
Perfect scaling to any form factor
Custom branding
Collaborate on the go
Enhanced Reporting Services
Modern reports with SQL Server Reporting
Services Report consumption from modern browsers
Improved parameters
Modern themes
New chart types
Access any data Scale and manage
Powerful Insights
Advanced analytics
PolyBase Insights from data across SQL Server and Hadoop with simplicity of T-SQL
Enhanced SSISDesigner support for previous SSIS versions
Support for Power Query
Enterprise-grade Analysis Services Enhanced performance and scalability for analysis services
Single SSDT in Visual Studio 2015 Build richer analytics solutions as part of your development projects in Visual Studio
Enhanced MDS Excel add-in 15x faster; more granular security roles; archival options for transaction logs; and reuse entities across models
Mobile BIBusiness insights for your on-premises data through rich visualization on mobile devices with native apps for Windows, iOS and Android
Enhanced Reporting ServicesNew modern reports with rich visualizations
R integration (CTP3)Bringing predictive analytic capabilities to your relational database
Analytics libraries (CTP3)Expand your “R” script library with Microsoft Azure Marketplace
Deeper insights across data (CTP3)
R integration with database engine (CTP3)
Example SolutionsFraud detection
Sales forecasting
Warehouse efficiency
Predictive maintenance
Extensibility
Microsoft AzureMachine Learning Marketplace
New R scripts
010010
100100
010101
010010
100100
010101
010010
100100
010101
010010
100100
010101
Built-in advanced analytics (CTP3)In-database analytics
Built-in to SQL Server
Analytic Library
T-SQL Interface
Relational Data
010010
100100
010101010010
100100
010101
Data ScientistInteract directlywith data
Data Developer/DBAManage data and analytics together
?R
R Integration
Capability
Extensible In-database analytics, integrated with R, exposed through T-SQL
Centralize enterprise library for analytic models
Benefits
No data movement, resulting in faster time
to insights
Real-time analytics on transactional data
Integrate with existing workflows
Unified governance across analytics & storage
SQL Server
Analytical EnginesFull R IntegrationFully Extensible
Data Management Layer
Relational Data
T-SQL InterfaceStream Data In-Memory
Analytics LibraryShare and CollaborateManage and Deploy
R +
Data Scientists
Business Analysts
Publish algorithms, and interact directly with Data
Analysis through TSQL, tools, and vetted
algorithms
DBAsManage storage and analytics together
Advanced Analytics (CTP3)Faster analytics with In-database processing
Hybrid solutions Simplicity Consistency
Stretch DatabaseStretch operational tables in a secure manner into Azure for cost effective historic data availability works with Always Encrypted and Row Level Security
Power BI with on-premises dataNew interactive query with Analysis Services. Customer data stays behind your firewall
Hybrid Scenarios with SSISAzure Data Factory integration with SSIS, package lineage and impact analysis and connect SSIS to cloud data source
Enhanced Backup to AzureFaster restore times and 50% reduction in storage, support larger DBs with Block blobs and custom backup schedule with local staging
Easy migration of on-premises SQL ServerSimple point and click migration to Azure
Simplified Add Azure Replica Wizard Automatic listener configuration for AlwaysOn in Azure VMs
Common development, management and identity toolsIncluding Active Directory, Visual Studio, Hyper-V and System Center
Consistent Experience from SQL Server on-premises to Microsoft Azure IaaS and PaaS
Deeper insights across data
Hybrid solutions Simplicity Consistency
Stretch DatabaseStretch operational tables in a secure manner into Azure for cost effective historic data availability works with Always Encrypted and Row Level Security
Power BI with on-premises dataNew interactive query with Analysis Services. Customer data stays behind your firewall
Hybrid Scenarios with SSISAzure Data Factory integration with SSIS, package lineage and impact analysis and connect SSIS to cloud data source
Enhanced Backup to AzureFaster restore times and 50% reduction in storage, support larger DBs with Block blobs and custom backup schedule with local staging
Easy migration of on-premises SQL ServerSimple point and click migration to Azure
Simplified Add Azure Replica Wizard Automatic listener configuration for AlwaysOn in Azure VMs
Common development, management and identity toolsIncluding Active Directory, Visual Studio, Hyper-V and System Center
Consistent Experience from SQL Server on-premises to Microsoft Azure IaaS and PaaS
Deeper insights across data
Stretch Database
Capability
Stretch large operational tables from on-premises to Azure with the ability to query
Benefits
Cost effective historical data availability for operational DBs
Entire table is online and queryable
Transparent to application
Supports new TCE transparent encryption and Row Level Security
Apps continue working without change or disruption
BI integration for on-
premises and cloud
Cold/closed data
OrdersIn-memory OLTP table
Hot/active data
Order historyStretched table
Trickle data movement and
remote query processing
On-premises
Azure
Stretch SQL Server into AzureSecurely stretch cold tables to Azure with remote query processing
Stretch database architecture
How it works
Creates a secure linked server definition in the on-premises SQL Server
Linked server definition has the remote endpoint as the target
Provisions remote resources and begins to migrate eligible data, if migration is enabled
Queries against tables run against both the local database and the remote endpoint
Remote Endpoint
Remote Data
On-Premise Instance
Azure
Inte
rnet B
oundary
Local Database
Local Data
Eligible Data
-- Enable local serverEXEC sp_configure 'remote data archive' , '1';RECONFIGURE;
-- Provide administrator credential to connect to-- Azure SQL DatabaseCREATE CREDENTIAL <server_address> WITH
IDENTITY = <administrator_user_name>, SECRET = <administrator_password>
-- Alter database for remote data archiveALTER DATABASE <database name>
SET REMOTE_DATA_ARCHIVE = ON (SERVER = server name); GO
-- Alter table for remote data archiveALTER TABLE <table name> ENABLE REMOTE_DATA_ARCHIVE
WITH ( MIGRATION_STATE = ON );GO;
High level steps
Configure local server for remote data archive
Create a credential with administrator permission
Alter specific database for remote data archive
Alter table for remote data archive
Typical workflow to enable Stretch Database
On-premises integration with Power BI
Live dashboards
and exploration
Analysis Services on-
premisesTabular model
Interactive query
AS Connector
Capability
Publish on-premises Analysis Services data models for consumption in Power BI
Benefits
Faster time to solution without having to move your data to the cloud.
Optimized query performance for sub-second response times.
Manage and secure data residing on-premises in SQL Server Analysis Services.
SQL Server vNext
Connect live to on-premises Analysis Services data
SSIS for Azure
SSIS Improvements for Azure services (CTP3)
Capability
Stretch large operational tables from on-premises to Azure with the ability to query
Benefits
Cost effective historical data availability for operational DBs
Entire table is online and queryable
Transparent to application
Supports new TCE transparent encryption and Row Level Security
SQL
Enhanced backup
Managed backupGranular control of the backup schedule;
Local staging support for faster recovery and resilient to transient network issues;
Support for system databases;
Support simple recovery mode.
Backup to Azure block blobsCost savings on storage;
Significantly improved restore performance; and
More granular control over Azure storage.
Azure Storage snapshot backupFastest method for creating backups and running restores
Uses SQL Server db files on Azure Blob Storage
Enhanced backup to Azure
Hybrid solutions Simplicity Consistency
Stretch DatabaseStretch operational tables in a secure manner into Azure for cost effective historic data availability works with Always Encrypted and Row Level Security
PowerBI with on-premises dataNew interactive query with Analysis Services. Customer data stays behind your firewall
Hybrid Scenarios with SSISAzure Data Factory integration with SSIS, package lineage and impact analysis and connect SSIS to cloud data source
Enhanced Backup to AzureFaster restore times and 50% reduction in storage, support larger DBs with Block blobs and custom backup schedule with local staging
Easy migration of on-premises SQL ServerSimple point and click migration to Azure
Simplified Add Azure Replica Wizard Automatic listener configuration for AlwaysOn in Azure VMs
Common development, management and identity toolsIncluding Active Directory, Visual Studio, Hyper-V and System Center
Consistent Experience from SQL Server on-premises to Microsoft Azure IaaS and PaaS
Deeper insights across data
Migrate databases to Azure
User DB System Objects SQL Settings
Schema
Data
Logins
Jobs
Certificates
…
Max server memory
Max worker threads
Default language
…
Traceflags
Migration Wizard
On Premise
EZ Button – Migrate to AzureSimple Single Click Migration Experience
Capability
Publish on-premises Analysis Services data models for consumption in Power BI
Benefits
Faster time to solution without having to move your data to the cloud.
Optimized query performance for sub-second response times.
Manage and secure data residing on-premises in SQL Server Analysis Services.
Method 1 Method 2 Method 3
TargetDB
Azure SQL Database
SQL Server
SSMS
2. Import
1. DeployOR
.bacpac
1. Export
Source DB
TargetDB
Azure SQL Database
SQL Server
SQL Azure Migration Wizard
2. Execute
1. Generate
T-SQL
Source DB
TargetDB
Azure SQL Database
SQL Server
SSMS
6. Export/Importor Deploy
SQLAzureMigrationWizard
4. Copy
CopyDB
Source DB
5. Publish
1. ImportVisual Studio
Database Project
2. Transform
3. Edit, Build & Test
*.sql
4. Publish (schema only)
Migrate a compatible database using SSMS
Migrate a near-compatible database using SAMW
Update the database schema offline using Visual Studio and SAMW, and then deploy it with SSMS
Migration methodologies
Migration Cookbook
Migrate an on-premises SQL Server database to Azure SQL Database (v12).
The Migration Cookbook describes various approaches you can use to migrate an on-premises SQL Server database to the latest Azure SQL Database Update (v12).
Download: http://aka.ms/azuresqlmigration
Simplified AlwaysOn with replicas on Azure
Today this requires manually configuring the Listener
SQL Server 2016 CTP2
Allows configuring the Availability Group Listener in Azure
Clients can connect to the Azure replica after failover using the Listener name
Simplified Add Azure Replica WizardAutomatic Listener Configuration
Hybrid solutions Simplicity Consistency
Stretch DatabaseStretch operational tables in a secure manner into Azure for cost effective historic data availability works with Always Encrypted and Row Level Security
PowerBI with on-premises dataNew interactive query with Analysis Services. Customer data stays behind your firewall
Hybrid Scenarios with SSISAzure Data Factory integration with SSIS, package lineage and impact analysis and connect SSIS to cloud data source
Enhanced Backup to AzureFaster restore times and 50% reduction in storage, support larger DBs with Block blobs and custom backup schedule with local staging
Easy migration of on-premises SQL ServerSimple point and click migration to Azure
Simplified Add Azure Replica Wizard Automatic listener configuration for AlwaysOn in Azure VMs
Common development, management and identity toolsIncluding Active Directory, Visual Studio, Hyper-V and System Center
Consistent Experience from SQL Server on-premises to Microsoft Azure IaaS and PaaS
Deeper insights across data
Consistent tools and experience
Consistent ToolsConsistency across:
On-premises, Private Cloud, Public Cloud
SQL Server local, VM, Azure SQL Database
Scalability, availability, security, identity, backup and restore, and replication
Plethora of data sources
Reporting, integration, processing, and analytics
All of this support Hybrid Cloud
Options:
SQL Server on physical machines
SQL Server in on-premises VM (private cloud)
SQL Server in Azure VM (public cloud)
Azure SQL Database (public cloud)
SQL Server, Azure VMs, Azure SQL DB
Shared Lower Cost
Dedicated Higher Cost
Higher Administration Lower Administration
Off Premises
Hybrid Cloud
Physical
SQL ServerPhysical Machines(raw Iron)SQ
L
Virtual
SQL Server Private CloudVirtualized Machines + Appliances
Infrastructure as a service
SQL Server in Azure VMVirtualized machinesSQ
L
Platform as a service
Software as a services
Azure SQL DatabaseVirtualized Databases
SQL
On Premises
The Microsoft data platform
Internal & external
DashboardsReports Ask Mobile
Information managementOrchestration
Extract, transform, load Prediction
RelationalNon-
relational Analytical
Apps
Streaming
VISUALIZE + DECIDE
TRANSFORM + ANALYZE
COLLECT + MANAGE
© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.