SQL CLR Demystified

A look at the integration of the CLR into SQL Server

Matt WhitfieldAtlantis Interactive UK Ltd

What will be covered

• Overview of CLR integration

• SqlContext• Stored Procedures• Scalar & Table

Functions• CLR DML Triggers• CLR DDL Triggers• Aggregate Functions

• CLR Types• Visual Studio

Database Projects• A look at permission

sets• Examples:

– Data maniplation for performance

– Environment manipulation for maintenance

Overview

• CLR integration into SQL Server allows us to run CLR code that conforms to certain restrictions

• CLR integration means that we can use common business rule validation code in the middle tier and the database

• For some operations, CLR integration allows us to extract maximum performance

• CLR integration allows us to perform maintenance tasks that would have been very difficult or impossible before

CLR Disabled by Default

• CLR integration is disabled by default in SQL Server, and therefore must be enabled:

EXEC sp_configure ‘clr enabled’, 1;

RECONFIGURE WITH OVERRIDE;GO

• This can also be done using the Surface Area Configuration tool.

What Framework version?

• The CLR in SQL Server is always loaded as the 2.0 runtime

• Until .NET 4.0, a single CLR host process could only load a single version of the .NET runtime

• It is possible future versions of SQL Server may allow more up-to-date runtime versions to be loaded

• We can confirm the loaded version with:select * from sys.dm_clr_properties

• However, we can use types from .NET 3.0 and 3.5, as these releases did not update mscorlib

Some Things To Note

• When SQL Server loads assemblies, they are cached in memory. When the O/S signals memory pressure to SQL Server, explicit garbage collection may be run, and the assemblies may be unloaded. This can cause performance issues if it happens frequently.

• SQL CLR code cannot be costed accurately by the query optimiser, as it does not look at what the code actually does – this can affect execution plans.

• SQL CLR code can sometimes prevent parallelism, and SQL CLR procedures are usually single threaded. Sometimes this can hurt performance, other times you may find you achieve the same amount of work in the same time using 1 thread instead of 8.

SqlContext

• SqlContext is a class that gives us access to four members:– IsAvailable– Pipe– TriggerContext– WindowsIdentity

• IsAvailable indicates whether the other members can be used

SqlContext.Pipe

• Pipe is the main point of interaction with the SQL Server when returning result sets or messages

• Messages can be sent to the client using the Send(string) function:SqlContext.Pipe.Send(“Hello World”);

• Single row result sets can be sent using the Send(SqlDataRecord) function.

Multi-row Result Sets

• SendResultsStart is used to specify the schema of the result set– No data is sent by the

SendResultsStart call

• 0 or more calls to SendResultsRow are then sent with data– Empty result sets can be

sent by making no calls to SendResultsRow

• SendResultsEnd marks the end of the result set

SendResultsStart(SqlDataRecord)

SendResultsRow(SqlDataRecord)

SendResultsEnd()

SqlContext.WindowsIdentity

• This returns a standard System.Security.Principal.WindowsIdentity object

• The property can only be used under EXTERNAL_ACCESS or UNSAFE permission sets

• For 99% of CLR work, using this property is not required

SqlContext.TriggerContext

• TriggerContext gives you access to:– ColumnCount – number of columns in the target table– EventData – SqlXml that returns the same as the

EVENTDATA T-SQL function– TriggerAction – a TriggerAction enumeration value

that specifies which action caused the trigger to fire– IsUpdatedColumn – method that determines if a

column was updated – same as the T-SQL UPDATE() function

– …but what about the inserted and deleted virtual tables that are available in normal triggers?

Context Connection

• To access data from within CLR code, the special ‘context connection’ connection string is used…using (SqlConnection connection = new SqlConnection("context connection=true")){ connection.Open(); ...}

• We can then run any SQL statement as we would normally from .NET

• The inserted and deleted virtual tables are available through this method when in a DML trigger

Stored Procedures

• What is a stored procedure?– A procedure that contains a sequence of

operations, which may affect existing data, return result sets or both

– CLR Stored Procedures are no different– CLR Stored Procedures are implemented as

static members of a class– The containing class is irrelevant to the

execution of the procedure, and is an organisational unit only

Hello Worldpublic partial class StoredProcedures{ [Microsoft.SqlServer.Server.SqlProcedure] public static void HelloWorld() { SqlContext.Pipe.Send("Hello World!"); }}

• Procedure is just a method, decorated with the SqlProcedure attribute.• Procedure is registered in SQL Server as follows:

CREATE PROCEDURE [dbo].[HelloWorld]AS EXTERNAL NAME [SQLAssembly].[StoredProcedures].[HelloWorld]GO

Simple Data Access

• In part 1:– we will open a context connection– we will execute a command– we will read the results, and use those to populate a Dictionary

object

• In part 2:– We will create record metadata– We will create a record set– We will populate the record set from the Dictionary

• While a very mundane example, and not something you would usually use the CLR for, it shows the basic mechanics of moving data into and out of the CLR

Simple Data Access (1/2)public static void RowCounter(){ Dictionary<string, int> typeDictionary = new Dictionary<string, int>(); using (SqlConnection connection = new SqlConnection("context connection=true")) { connection.Open(); using (SqlCommand command = new SqlCommand("SELECT [type] from [sys].[objects];",

connection)) { using (SqlDataReader reader = command.ExecuteReader()) { while (reader.Read()) { string type = reader.GetString(0); if (typeDictionary.ContainsKey(type)) { typeDictionary[type] = typeDictionary[type] + 1; } else { typeDictionary.Add(type, 1); } } } } }...

Simple Data Access (2/2)... SqlDataRecord record = new SqlDataRecord(new SqlMetaData[] { new SqlMetaData("Type", SqlDbType.NChar,

2), new SqlMetaData("Count", SqlDbType.Int) }); SqlContext.Pipe.SendResultsStart(record); foreach (KeyValuePair<string, int> kvp in

typeDictionary) { record.SetString(0, kvp.Key); record.SetInt32(1, kvp.Value); SqlContext.Pipe.SendResultsRow(record); } SqlContext.Pipe.SendResultsEnd();}

Things to note

• That we did not create a new SqlDataRecord object for every row, we just populated the existing object

• That the name exposed to SQL Server does not have to match the name of the implementing method

• The return type of the stored procedure can be int to return a value, as with T-SQL stored procedures

Scalar Functions

• What is a scalar function?– A statement or set of statements that return a

single value of a specific type based on zero or more parameters

– CLR Scalar Functions are no different– CLR Scalar Functions are implemented much

the same as stored procedures – a single static method of a class

String Length Scalar Function

[Microsoft.SqlServer.Server.SqlFunction]public static long fn_LengthTestCLR(string input){ return input.Length;}

CREATE FUNCTION [dbo].[fn_LengthTestCLR] (@input nvarchar (MAX))RETURNS bigint AS EXTERNAL NAME [SQLAssembly].

[UserDefinedFunctions].[fn_LengthTestCLR]GO

Performance Comparison• Consider the equivalent T-SQL function, and the following SQL

Statement:

CREATE FUNCTION dbo.fn_LengthTest(@input varchar(MAX))

RETURNS [bigint]AS BEGIN RETURN LEN(@input)END

SELECT SUM(dbo.fn_LengthTest(o.[name])) FROM [sys].[objects] [o] CROSS JOIN [sys].[all_columns] [ac] CROSS JOIN [sys].[all_columns] [ac1]

CLR would be slower, right?

• No!

• On a dual core 2GHz CPU –– SQL Function – 9.036 seconds– SQL CLR Function – 3.918 seconds

• Overhead of calling a SQL CLR function is lower than that of calling a T-SQL function

Table Functions

• What is a table function?– A statement or set of statements that return a single

result set with a defined schema based on zero or more parameters

– CLR Table Functions can only be multi-statement equivalents, there is no concept of an ‘inline CLR table function’

– Implemented as a pair of methods – one that returns an IEnumerable, and one that sets outbound parameters for each object – both are static members

Regular Expression Function

• In Part 1:– We will implement an object that we can return to hold

the data required to fill each row• In Part 2:

– We will create a list of our storage object– We will run a regular expression match based on the

parameters– We will populate our list with the results of the regular

expression– We will return the list

• In Part 3:– We will set the values to be stored in each row

A Regular Expression Function (1/3)

private struct _match{ public readonly int MatchNumber; public readonly int GroupNumber; public readonly string CaptureValue; public _match(int matchNumber, int groupNumber, string captureValue) { MatchNumber = matchNumber; GroupNumber = groupNumber; CaptureValue = captureValue; }}

A Regular Expression Function (2/3)

[SqlFunction(FillRowMethodName = "FillRow", TableDefinition = "MatchNumber int, " + "GroupNumber int, " + "CaptureValue nvarchar(MAX)")]public static IEnumerable GetCaptureGroupValues(string input, string pattern){ List<_match> matchList = new List<_match>(); int matchIndex = 0; foreach (Match m in Regex.Matches(input, pattern)) { int groupIndex = 0; foreach (Group g in m.Groups) { matchList.Add(new _match(matchIndex, groupIndex++, g.Value)); } matchIndex++; } return matchList;}

A Regular Expression Function (3/3)

public static void FillRow(Object obj, out int MatchNumber, out int GroupNumber, out SqlString

CaptureValue){ _match match = (_match)obj; MatchNumber = match.MatchNumber; GroupNumber = match.GroupNumber; CaptureValue = match.CaptureValue;}

Creation SQLCREATE FUNCTION [dbo].[GetCaptureGroupValues] (@input [nvarchar] (MAX), @pattern [nvarchar] (MAX))RETURNS TABLE ([MatchNumber] [int] NULL, [GroupNumber] [int] NULL, [CaptureValue] [nvarchar] (MAX) NULL)AS EXTERNAL NAME [SQLAssembly].[UserDefinedFunctions].

[GetCaptureGroupValues]GO

• Again, the name exposed to SQL Server does not have to match the name of the main body function

• The SQL definition can specify different column names and parameter names to the main method and fill row method

Things to note

• That we used a struct for the storage object, rather than a class, because of reduced instantiation costs

• That our function was decorated with a SqlFunction attribute that named the fill row method and specified the shape of the output table

• That the fill row method took an object as it’s first parameter, and had an outbound parameter for each of the columns we wanted to fill

CLR Triggers

• Known bug in Visual Studio 2005 and 2008 that means SqlTrigger attribute does not allow you to specify the schema of the target table

• Fixed in Visual Studio 2010

• Workaround is to comment out the SqlTrigger attribute, and deploy manually

DML Triggers

• What is a DML Trigger?– A statement or set of statements run when

data in a target table is modified– CLR DML Triggers are no different– CLR DML Triggers still have access to the

inserted and deleted virtual tables– CLR DML Triggers are implemented as static

methods in a class

A Simple DML Trigger

• We will specify the event type and target object using the SqlTrigger attribute

• We will connect to the database using the context connection

• We will select the rows from the inserted virtual table, and count them

• We will send a message to the SqlContext pipe stating the number of rows

A Simple DML Trigger[Microsoft.SqlServer.Server.SqlTrigger(Target="CallLog", Event="FOR UPDATE")]public static void TestDMLTrigger(){ int i = 0; using (SqlConnection connection = new SqlConnection("context connection=true")) { connection.Open(); using (SqlCommand command = new SqlCommand("SELECT * from inserted;",

connection)) { using (SqlDataReader reader = command.ExecuteReader()) { while (reader.Read()) { i++; } } } } SqlContext.Pipe.Send(i.ToString() + " rows in inserted");}

DDL Triggers

• What is a DDL Trigger?– A statement or set of statements run when a

schema modification event occurs– CLR DDL Triggers are no different– CLR DDL Triggers have access to the event

data via SqlContext.TriggerContext– CLR DDL Triggers are implemented as static

methods in a class

A Simple DDL Trigger

• We will specify the target and the event type using the SqlTrigger attribute

• We will create an XmlDocument based on the TriggerContext object from the SqlContext object

• We will then use XPath to find the created object’s schema and name

• We will send a message to the SqlContext pipe stating the name of the created object

A Simple DDL Trigger[Microsoft.SqlServer.Server.SqlTrigger(Target="DATABASE", Event="FOR CREATE_TABLE")]public static void TestDDLTrigger(){ XmlDocument document = new XmlDocument(); document.LoadXml(SqlContext.TriggerContext.EventData.Value);

string objectName = document.SelectSingleNode("//SchemaName").InnerText +

"." +

document.SelectSingleNode("//ObjectName").InnerText;

SqlContext.Pipe.Send("Table " + objectName + " was created.");}

Aggregate Functions

• What is an aggregate function?– A function that takes a series of values and

accumulates them into a single, aggregated result– CLR Aggregate functions are no different– Implemented as a class with four methods:

• Init• Accumulate• Merge• Terminate

– Aggregate functions in SQL Server 2005 can take only one parameter, in SQL Server 2008 this restriction is lifted

Aggregate Function Methods

• Init method is called first, here we initialise our member fields

• Accumulate is called to add a value to the aggregate

• Merge is called to add together two aggregates (for example, as a result of parallelism)

• Terminate is called to return our value

A Simple Aggregate

• We will use the SqlUserDefinedAggregate attribute to specify the serialisation format of our aggregate

• We will track the minimum value in Accumulate, and also track that we have seen values

• In Merge, we will check if the other aggregate class has seen values - if it has, then we take the minimum value seen

• In Terminate we will return the minimum value we have seen if we have seen any values, or NULL if we have not seen any values

A Simple Aggregate (1/2)[Serializable][Microsoft.SqlServer.Server.SqlUserDefinedAggregate(Format.Native)]public struct MinValue{ int minValue; bool hasValues;

public void Init() { minValue = Int32.MaxValue; hasValues = false; }

public void Accumulate(SqlInt32 Value) { hasValues = true; if (Value.Value < minValue) { minValue = Value.Value; } }...

A Simple Aggregate (2/2)... public void Merge(MinValue Group) { if (Group.hasValues) { if (Group.minValue < minValue) { minValue = Group.minValue; } hasValues = true; } }

public SqlInt32 Terminate() { if (hasValues) { return minValue; } else { return SqlInt32.Null; } }}

SqlUserDefinedAggregate

• This attribute contains parameters which can affect the query optimiser, and can cause incorrect results if set wrong:– IsInvariantToDuplicates – should only be true if

aggregating the same value many times does not affect the result

– IsInvariantToNulls – should only be true if aggregating NULL values does not affect the result

– IsInvariantToOrder – should only be true if the result is not dependent on the order of the aggregated values

– IsNullIfEmpty – should only be true if the aggregate of 0 values is NULL

Native and UserDefined Serialisation

• If your aggregate contains reference types then you must implement custom serialisation, and specify Format.UserDefined in the SqlUserDefinedAggregate attribute

• When specifying UserDefined format, your aggregate must implement the IBinarySerialize interface, and specify it’s maximum size in the SqlUserDefinedAggregate attribute

• Maximum byte size can be 1 to 8000 in SQL Server 2005, or -1 for any value between 8001 bytes and 2 GB in SQL Server 2008.

• The same serialisation interface applies to both Aggregates and User Defined Types

CLR Types

• What is a user defined type?– A type based on a system type, which may be

bound to old-style rules and defaults– CLR Types are different – CLR Types can be complex, and can have

properties and methods– CLR Types can have static methods (and the

built in CLR Types in SQL Server 2008 have some)

A Simple CLR Type

• Part 1:– We will define the serialisation format of the type– We will provide the IsNull property and the Null static property to

return a NULL instance

• Part 2:– We will define methods to convert the type to and from strings

• Part 3:– We will define a property for the symbol, a method to change the

symbol and a static method to create a new currency with a defined symbol and value

• Part 4:– We will declare our member fields and provide implementation of

the IBinarySerialize interface to load and store our type

A Simple CLR Type (1/4)[Serializable][Microsoft.SqlServer.Server.SqlUserDefinedType(Format.UserDefined, MaxByteSize=24)]public struct Currency : INullable, IBinarySerialize{ public bool IsNull { get { return _null; } }

public static Currency Null { get { Currency h = new Currency(); h._null = true; return h; } }

A Simple CLR Type (2/4) public override string ToString() { return _symbol + _value.ToString(); }

public static Currency Parse(SqlString s) { if (s.IsNull || string.IsNullOrEmpty(s.Value) || s.Value.Length < 2 || string.Equals(s.Value, "NULL", StringComparison.OrdinalIgnoreCase)) { return Currency.Null; } Currency u = new Currency(); u._symbol = s.Value[0]; u._value = Decimal.Parse(s.Value.Substring(1)); return u; }

A Simple CLR Type (3/4) public char Symbol { get { return _symbol; } }

[SqlMethod(IsMutator=true)] public void ChangeSymbol(char newSymbol) { _symbol = newSymbol; }

public static Currency CreateCurrency(char symbol, Decimal value) { Currency c = new Currency(); c._symbol = symbol; c._value = value; return c; }

A Simple CLR Type (4/4) private char _symbol; private Decimal _value; private bool _null;

public void Read(BinaryReader r) { _null = r.ReadBoolean(); _symbol = r.ReadChar(); _value = r.ReadDecimal(); }

public void Write(BinaryWriter w) { w.Write(_null); w.Write(_symbol); w.Write(_value); }}

The CLR Type in use• Using our static method to create an instance:DECLARE @i [Currency]SET @i = [Currency]::[CreateCurrency]('$', 0.53)PRINT CONVERT([varchar], @i)

• Using our instance mutator method to change the symbol:DECLARE @i [Currency]SET @i = [Currency]::[CreateCurrency]('$', 0.53)SET @i.ChangeSymbol('£')PRINT CONVERT([varchar], @i)

• Retrieving the value of a property:DECLARE @i [Currency]SET @i = [Currency]::[CreateCurrency]('$', 0.53)PRINT @i.Symbol

Things to note

• We can call static methods and properties on CLR types with [type-name]::[member-name]

• IBinarySerialize must be used to serialise types that cannot be serialised natively

• A CLR type should always be able to parse it’s own string representation

• Methods that change the state of the type must be marked as mutator methods

• Although in .NET we can define operator overloads to specify how operators (e.g. +, -, *, /) are applied to our classes, SQL Server does not use these.

Permission Sets

• Each assembly in SQL Server has an associated ‘permission set’ – one of:– SAFE– EXTENAL_ACCESS– UNSAFE

• The permission set assigned to the assembly determines how limited the CLR functionality is

• This, in turn, determines how much damage we can do to the stability of the host process (i.e. SQL Server itself)

Allowed operationsSAFE EXTERNAL_ACCESS UNSAFE

Code access security permissions

Execute only Execute + access to external resources

Unrestricted

Programming model restrictions

Yes Yes No

Verifiability requirement

Yes Yes No

Local data access Yes Yes Yes

Ability to call native code

No No Yes

Programming model restrictions

• A few of the programming model restrictions for SAFE and EXTERNAL_ACCESS assemblies are as follows:– No static fields may be used to store information– PEVerify type safety checking test is passed– No synchronization may be used– Finalizer methods may not be used– No threading may be used– No self-affecting code may be used

• There are more requirements that must be met under the above two permission sets – the above is just some of the more common restrictions.

Visual Studio Database Projects

• Visual Studio offers us the ability to create a ‘CLR Database Project’ which enables simplified deployment and debugging

• Some issues present, though these are mostly solved in VS2010

• Database projects handle the deployment of the objects that we create simply, which can be really helpful during implementation

Performance Example

Phil Factor Speed Phreak Competition 4

The Log Parsing Problem• Producing aggregated results from an IIS log file• Made difficult by the need to determine

‘sessions’ – i.e. groups of rows from the same IP within a set time-span

• Requirement to produce three aggregated result sets – visitor summary by day, visitor summary and page summary by day

How did the CLR help?

• CLR methods can yield excellent results when doing forward-only navigation through result sets, performing aggregation and calculation along the way

• Iteration over the log allowed the data to be parsed into ‘visits’ by applying the session timeout

• Iteration over collection of visits allowed the data to be aggregated into country and page summaries

What was the result?

• Execution in 301ms, as opposed to the closest T-SQL approach which took 1779ms.

• Efficiency gains through the fact that the CLR procedure would only occupy 1 thread

• Maintenance benefits in readability and simplicity

Maintenance Example

• CLR Procedure to retrieve the free disk space on any specified server – by Tara Kizer

• Uses System.Diagnostics namespace to interface with WMI to retrieve results

• Requires UNSAFE assembly permission set because PerformanceCounterCategory is synchronised

• Simple use of framework classes to retrieve the information required

When should we use the CLR?

• To improve performance:– In situations where a forward-only scan over a result set requires

row by row processing to produce an aggregated result set– In situations where complex logic cannot simply be expressed in

T-SQL• To improve maintenance:

– In situations where information required can be simply obtained from any .NET process

– In situations where interaction with the O/S is required (will require elevated permission set)

• For functionality:– In situations where commonly available .NET Framework

classes provide desired functionality (classic example is Regular Expression handling

– To bring business logic from the middle tier into the database

And when should we not?

• When the functionality presented offers little over it’s T-SQL equivalent

• Instead of regular DML statements (i.e. a CLR procedure to insert a row is not the way to go!)

• To replicate row by row update logic from a cursor – conversion to set-based T-SQL yields far higher performance

• When the desired operation really does not belong in the database

• When the database has to be portable to other types of database server

Summary

• We learned:– How to implement each type of CLR object– How to get data out of and back into SQL Server– That the only SQL CLR object type which differs

significantly from it’s T-SQL equivalent is the CLR User Defined Type

– That the CLR can offer significant performance improvements, particularly for forward-only scans and scalar functions

– That we are restricted as to what we can achieve in a CLR method based on the permission set of the assembly