Download - U-SQL Query Execution and Performance Tuning
Michael RysPrincipal Program ManagerMicrosoft Big Data@MikeDoesBigData, [email protected]
U-SQL Query Execution and Performance Tuning
Session Objectives And TakeawaysSession Objective(s): • Understand the U-SQL Query execution• Be able to understand and improve the U-SQL query plan• Know how to write more efficient U-SQL scripts
Key Takeaways:• U-SQL is designed for scale-out• U-SQL provides scalable execution of user code• U-SQL has a tool set that can help you analyze and improve your scalability and performance
Agenda• Job Execution Experience and Investigations
Query ExecutionStage GraphDryad crash courseJob MetricsResource Planning
• Partitioning AnalysisAnalyze the critical pathHeat MapCritical PathData Skew
• Tuning / OptimizationsData PartitioningPartition EliminationPredicate PushingColumn PruningSome Data HintsUDOs can be evilINSERT optimizations
U-SQL Query Execution and Performance Tuning
Job Execution Experience and Investigations
12Expression-flow Programming Style
Automatic "in-lining" of U-SQL expressions – whole script leads to a single execution model.
Execution plan that is optimized out-of-the-box and w/o user intervention.
Per job and user driven level of parallelization.
Detail visibility into execution steps, for debugging.
Heatmap like functionality to identify performance bottlenecks.
Job Scheduling: Job States
Preparing
Queued
Running
Finalizing
Ended(Succeeded, Failed,
Cancelled)
NewCompiling
QueuedSchedulin
gStarting
Running
Ended
What you see in the UX
Underlying Job State
The script is being compiled by the Compiler Service
All jobs enter the queue.
Are there enough ADLAUs to start the job?
If yes, then allocate those ADLAUs for the job
The U-SQL runtime is now executing the code on 1 or more ADLAUs or finalizing the outputs
The job has concluded.
The USQL Compilation
U-SQL C# user code
C++ system code
Algebra
other files(system files, deployed
resources)
managed dll
Unmanaged dll
Input script
Compilation output (in job folder)
Compiler & Optimizer
Files
Meta Data
Service
Deployed to vertices
Distributing Work into VertexesSome fixed amount of
work
Each square is called a “vertex”
Each vertex represents a fraction of the work
U-SQL Query Execution Physical plans vs. Dryad stage graph…
Super Vertex Stage Details252 Pieces of work
AVG Vertex execution time
4.3 Billion rows
Data Read & Written
U-SQL Query ExecutionDryad as an art form…
13
U-SQL Query ExecutionRedefinition of big-data…
14
U-SQL Query ExecutionRedefinition of big-data…
Performance
Analyzing the Critical Path
16
U-SQL Performance AnalysisAnalyze the critical path, heat maps, playback, and runtime metrics on every vertex…
Partitioning during Processing
Analyzing Data Skew
What is Data Skew? Some data points
are much more common than others
data may be distributed such that all rows that match a certain key go to a single vertex
imbalanced execution, vertex time out.
Californ
iaFlo
rida
Ohio
North C
arolina
Washing
ton
Indian
a
Marylan
d
Colorad
o
Louisia
na
Oklaho
ma
Mississ
ippi
Utah
Nebras
kaHaw
aii
Rhode
Islan
d
South
Dakota
District
of Colu
mbia0
5,000,000
10,000,000
15,000,000
20,000,000
25,000,000
30,000,000
35,000,000
40,000,000Population by State
Low Distinctiveness Keys Keys with small
selectivity can lead to large vertices even without skew
Solutions: Find more distinctive keys Implement
aggregation/reducer recursively if possible
@rows = SELECT Gender, AGG<MyAgg>(…) AS Result
FROM @HugeInputGROUP BY Gender;
Gender==Male Gender==Female
@HugeInput
Vertex 0 Vertex 1
Vertex 4
Vertex 3 Vertex 2 Vertex 1
Vertex 1
Non-Recursive vs Recursive SUM20
1 2 3 4 5 6 7 8 36
1 2 3 4 5 6 7 8
6 15 15
36
U-SQL Partitioning during Processing Data Skew
U-SQL PartitioningData Skew – Recursive Reducer
// Metrics per domain@Metric = REDUCE @Impressions ON UrlDomain USING new Bing.TopNReducer(count:10) ;
// …
Inherent Data Skew
[SqlUserDefinedReducer(IsRecursive = true)]public class TopNReducer : IReducer{ public override IEnumerable<IRow> Reduce(IRowset input, IUpdatableRow output) { // Compute TOP(N) per group // … }}
Recursive• Allow multi-stage aggregation trees• Requires same schema (input => output) • Requires associativity:• R(x, y) = R( R(x), R(y) )
• Default = non-recursive• User code has to honor recursive
semantics
www.bing.combrought to a single vertex
U-SQL Partitioning during ProcessingPartitioning – Combiner Modes
// Existing Watson hits@DeDup = COMBINE @Exceptions AS L WITH @WatsonBuckets AS R ON L.AppId WITH R.AppId USING new Windows.WatsonDedupCombiner() ;
// …
[SqlUserDefinedCombiner(Mode = CombinerMode.Right)]public class WatsonDedupCombiner : ICombiner{ public override IEnumerable<IRow> Combine(IRowset left, IRowset right, IUpdatableRow output) { // DeDup against existing Call Stacks // … }}
CombinerMode•Allow parallelism even within a partition public enum CombinerMode { Inner, /// Inner join - both row level Left, /// Left Group Join - left row level Right, /// Right Group Join - right row level Full /// Full Group Join - none row level
• Default = Full Group Join• User code has to honor row level semantics
Row Level Combiner
X
IDother columns
XQ
QFQ
XXZ
FFX
LEFT
X
ID other columns
Q
QF
Q
X
Z
FFX
RIGHT
F
Z
M1
M2
M3
M4
Enables Broadcast JOIN
Data Storage Files Tables
Unstructured Data in files Row-oriented files are split into 250MB
extents Should be uploaded with row-boundary alignment
(CR/LF as boundary) Allows parallel extraction
Binary files (incl XML/JSON) will still be in extents Have to be processed in single vertex extractor with
atomicFileProcessing=true. Use File Sets to provide semantic partition
pruning Tables
Clustered Index (row-oriented) storage Vertical and horizontal partitioning Statistics for the optimizer (CREATE
STATISTICS) Native scalar value serialization
Data PartitioningFilesTables
Partitioning of unstructured data Use File Sets to provide semantic partition pruning
Table Partitioning Fine grained (horizontal) partitioning
Partitions within a file (together with clustering) to keep same data values close
Choose for: Join alignment, partition size, filter selectivity
Coarse grained (vertical) partitioning Partition is addressable in language Query predicates will allow partition pruning
Partition Scheme
When to use?
HASH(keys)DIRECT HASH Exact control of hash bucketRANGE(keys) Keeps ranges togetherROUND ROBIN To get equal distribution (if others give skew)
Optimizations
Data Partitioning and the optimizer
// Unstructured Files (24 hours daily log impressions)@Impressions = EXTRACT ClientId int, Market string, OS string, ... FROM @"wasb://ads@wcentralus/2015/10/30/{*}.nif" FROM @"wasb://ads@wcentralus/2015/10/30/{Market:*}_{*}.nif" ;
// …
// Filter to by Market@US = SELECT * FROM @Impressions WHERE Market == "en" ;
U-SQL OptimizationsPartition Elimination – Unstructured Files
Partition Elimination• Even with unstructured files!• Leverage Virtual Columns (Named)• Avoid unnamed {*}
• WHERE predicates on named virtual columns• That binds the PE range during compilation time• Named virtual columns without predicate = error
• Design directories/files with PE in mind• Design for elimination early in the tree, not in the leaves
Extracts all files in the folder
Post filter = pay I/O cost to drop most data
PE pushes this predicate to the EXTRACT
EXTRACT now only reads “en” files!
en_10.0.nif
en_8.1.nif
de_10.0.nif
jp_7.0.nif
de_8.1.nif
../2015/10/30/
…
// TABLE(s) - Structured Files (24 hours daily log impressions)CREATE TABLE Impressions (Day DateTime, Market string, ClientId int, ... INDEX IX CLUSTERED(Market, ClientId) PARTITIONED BY BUCKETS (Day) HASH(Market, ClientId) INTO 100 );
DECLARE @today DateTime = DateTime.Parse("2015/10/30");
// Market = Vertical PartitioningALTER TABLE Impressions ADD PARTITION (@today);
// …
// Daily INSERT(s)INSERT INTO Impressions(Market, ClientId) PARTITION(@today) SELECT * FROM @Q ;
// …
// Both levels are elimination (H+V)@Impressions = SELECT * FROM dbo.Impressions WHERE Market == "en" AND Day == @today ;
U-SQL OptimizationsPartition Elimination – TABLE(s) Partition Elimination
• Horizontal and vertical partitioning• Horizontal is traditional within file (range, hash, robin)• Vertical is across files (bucketing)
• Immutable file system• Design according to your access patterns
Enumerate all partitions filtering for today
30.ss
30.1.ss
29.ss28.ss
29.1.ss
Impressions
…
deen
jp
de
PE across files + within each file
@Inpressions = SELECT * FROM searchDM.SML.PageView(@start, @end) AS PageView OPTION(LOWDISTINCTNESS=Query) ;
// Q1(A,B)@Sessions = SELECT ClientId, Query, SUM(PageClicks) AS Clicks FROM @Impressions GROUP BY Query, ClientId ;
// Q2(B)@Display = SELECT * FROM @Sessions INNER JOIN @Campaigns ON @Sessions.Query == @Campaigns.Query ;
U-SQL OptimizationsPartitioning – Minimize (re)partitions
Input must be partitioned on: (Query)
Input must be partitioned on:(Query) or (ClientId) or (Query,
ClientId)
Optimizer wants to partition only onceBut Query could be skewed
Data Partitioning• Re-Partitioning is very expensive• Many U-SQL operators can handle multiple partitioning
choices• Optimizer bases decision upon estimations
Wrong statistics may result in worse query performance
// Unstructured (24 hours daily log impressions)@Huge = EXTRACT ClientId int, ... FROM @"wasb://ads@wcentralus/2015/10/30/{*}.nif" ;
// Small subset (ie: ForgetMe opt out)@Small = SELECT * FROM @Huge WHERE Bing.ForgetMe(x,y,z) OPTION(ROWCOUNT=500) ;
// Result (not enough info to determine simple Broadcast join)@Remove = SELECT * FROM Bing.Sessions INNER JOIN @Small ON Sessions.Client == @Small.Client ;
U-SQL OptimizationsPartitioning - Cardinality
Broadcast JOIN right?
Broadcast is now a candidate.
Wrong statistics may result in worse query performance
Optimizer has no stats this is small...
Optimizations
Predicate pushing and user code
// Bing impressions@Impressions = SELECT * FROM searchDM.SML.PageView(@start, @end) AS PageView ;
// Compute sessions@Sessions = REDUCE @Impressions ON Client, Market READONLY Market USING new Bing.SessionReducer(range : 30) ;
// Users metrics@Metrics = SELECT * FROM @Sessions WHERE Market == "en-us" ;
// …
Microsoft Confidential
U-SQL OptimizationsPredicate pushing – UDO pass-through columns
// Bing impressions@Impressions = SELECT * FROM searchDM.SML.PageView(@start, @end) AS PageView ;
// Compute page views@Impressions = PROCESS @Impressions READONLY Market PRODUCE Client, Market, Header string USING new Bing.HtmlProcessor() ;
@Sessions = REDUCE @Impressions ON Client, Market READONLY Market USING new Bing.SessionReducer(range : 30) ;
// Users metrics@Metrics = SELECT * FROM @Sessions WHERE Market == "en-us" ;
Microsoft Confidential
U-SQL OptimizationsPredicate pushing – UDO row level processors
public abstract class IProcessor : IUserDefinedOperator{ /// <summary/> public abstract IRow Process(IRow input, IUpdatableRow output);}
public abstract class IReducer : IUserDefinedOperator{ /// <summary/> public abstract IEnumerable<IRow> Reduce(IRowset input, IUpdatableRow output);}
// Bing impressions@Impressions = SELECT Client, Market, Html FROM searchDM.SML.PageView(@start, @end) AS PageView ;
// Compute page views@Impressions = PROCESS @Impressions PRODUCE Client, Market, Header string USING new Bing.HtmlProcessor() ;
// Users metrics@Metrics = SELECT * FROM @Sessions WHERE Market == "en-us" && Header.Contains("microsoft.com") AND Header.Contains("microsoft.com") ;
U-SQL OptimizationsPredicate pushing – relational vs. C# semantics
Optimizations
Column pruning
// Bing impressions@Impressions = SELECT * FROM searchDM.SML.PageView(@start, @end) AS PageView ;
// Compute page views@Impressions = PROCESS @Impressions PRODUCE * REQUIRED ClientId, HtmlContent(Header, Footer) USING new Bing.HtmlProcessor() ;
// Users metrics@Metrics = SELECT ClientId, Market, Header FROM @Sessions WHERE Market == "en-us" ;
U-SQL OptimizationsColumn Pruning and dependencies
C H M
C H M
C H M
Column Pruning• Minimize I/O (data shuffling)• Minimize CPU (complex processing, html)• Requires dependency knowledge:• R(D*) = Input ( Output )
• Default no pruning• User code has to honor reduced columns
A B C D E F G J KH I … M … 1000
General UDO Warning and other Tips • Use SELECT with UDFs instead of PROCESS• Use User-defined Aggregators instead of REDUCE• Hint Cardinality if you use CROSS APPLY and it
does chose the wrong plan• Avoid ORDER BY unless needed (OUTPUT, “Top N
Rows”)• Learn to use Windowing Functions (OVER
expression)• Use SQL.MAP and SQL.ARRAY instead of C#
Dictionary and array
INSERT Multiple INSERTs into same table
• Generates separate file per insert in physical storage:• Can lead to performance degradation
• Recommendations:• Try to avoid small inserts• Rebuild table after frequent insertions
with:ALTER TABLE T REBUILD;
In Review: Session Objectives & TakeawaysSession Objective(s): • Understand the U-SQL Query execution• Be able to understand and improve the U-SQL query plan• Know how to write more efficient U-SQL scripts
Key Takeaways:• U-SQL is designed for scale-out• U-SQL provides scalable execution of user code• U-SQL has a tool set that can help you analyze and improve your scalability and performance
Additional Resources
Blogs and community page: http://usql.io http://blogs.msdn.com/b/visualstudio/ http://azure.microsoft.com/en-us/blog/topics/big-data/ https://channel9.msdn.com/Search?term=U-SQL#ch9Search
Documentation and articles: http://aka.ms/usql_reference https://azure.microsoft.com/en-us/documentation/services/dat
a-lake-analytics/
https://msdn.microsoft.com/en-us/magazine/mt614251
ADL forums and feedback http://aka.ms/adlfeedback https://
social.msdn.microsoft.com/Forums/azure/en-US/home?forum=AzureDataLake
http://stackoverflow.com/questions/tagged/u-sql
© 2016 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.