isug 113: file stream

File StreamMicrosoft SQL Server

2008 R2

Kobi Cohen

System DBA ManagerSpiral Solutions

©Spiral Solutions, Ltd. All rights reserved. 2

Objectives

Familiarization with FILESTREAM Overview

Storage

Security & Access

Implementation

Application Considerations

Combined with other SQL Server features

Cluster Configuration

RBS (Remote BLOB store)


File StreamOverview

For unstructured data docs, images, videos.

not new data type Storage attribute.

Varbinary (max) only Fully transactional.

BLOBs saved on storage (NTFS file system) Needs coordination between DBA and System Administrator.

Can be accessed by T-SQL (DMLs) or Win32 API (stream access).

Use NT system cache:

Much faster Avoid double writing (Buffer cache + Disk).

Reduce load from DB engine.

SQL Server buffer pool isn’t used more free memory.

BLOB size limited only by the volume size (not limited to 2GB)

even in Express Edition.


File StreamOverview

Store BLOBs in table? On separate files? or in FILESTREAM?

Data size

Larger than 1 MB NTFS FILESTREAM.

Smaller than 256 KB inside the database.

Between 256 KB and 1MB depend on read vs. write ratio + overwrite rate.

Performance Fast read access is important NTFS FILESTREAM.

Transactional Consistency BLOB + structured data FILESTREAM.

Security sensitive data FILESTREAM SQL security over ACLs.

Manageability integrated solution less complex FILESTREAM.

Data fragmentation frequent updates easier to handle on NTFS.

Client access Using DB access protocols (ODBC, OLEDB) NTFS

FILESTREAM.


File StreamOverview

BLOB Storage Methods Comparison

Comparison point

Storage solution

File server / file system SQL Server (using varbinary(max))

FILESTREAM

Maximum BLOB size NTFS volume size 2 GB – 1 bytes NTFS volume size

Streaming performance of large BLOBs

Excellent Poor Excellent

Security Manual ACLs Integrated Integrated + automatic ACLs

Cost per GB Low High Low

Manageability Difficult Integrated Integrated

Integration with structured data Difficult Data-level consistency Data-level consistency

Application development and deployment

More complex More simple More simple

Recovery from data fragmentation Excellent Poor Excellent

Performance of frequent small updates

Excellent Moderate Poor


File StreamOverview

Read Performance Comparison (Cold buffer pool)


File StreamOverview

Read Performance Comparison (Warm buffer pool)


File StreamOverview

Insert Performance Comparison (Cold buffer pool)


File StreamOverview

Update Performance Comparison (Cold buffer pool)


File StreamStorage

Preparing for FILESTREAM:

Set the proper Disk/RAID level resilience, performance, cost, load.

Use FSUTIL :

Turn off short (8.3) file names.

Disable last access time.

Format volume cluster size Use 64-KB NTFS clusters 4-KB for

compressed volumes.

Compress volume extra CPU.

Regularly defrag FILESTREAM computer systems.

Set antivirus scan of FILESTREAM volumes to avoid delete

infected files quarantine if it’s a must.


File StreamStorage

FILESTREAM stored in special file groups:

File system filter per each volume one for SQL Server

version Integrated security is a must.

File system directories (data containers)

instead of data files (mdf, ndf).

FILESTREAM file group = FILESTREAM data container.

Adding file to FILESTREAM file group = define data container:

Each logical file name is the root directory name.

Filestream.hdr FILESTREAM metadata describing data container.

$FSLOG FILESTREM equivalent to transaction log.

FILESTREAM operation mapped to LSN.

FILESTREAM garbage collection thread triggered by CHECKPOINT.

DBCC CHECKDB for validates link-level consistency.


File StreamStorage

File stream storage considerations:

Separate FILESTREAM data containers from:

OS volumes and paging file volumes.

Tempdb volumes.

Databases data & log files volumes.

Each data container on separate volume granular control.

Data containers cannot be nested.

All FILSTREAM columns data in a table must be stored in same

FILESTREAM file group FILESTREAM_ON better performance.

Each row must have non-null unique row id (uniqueidentifier

ROWGUID) and a unique/primary key index (clustered only for

non-random data to avoid index reordering on every insert/delete).

Can use partitioning on uniqueidentifier column for load balancing.


File StreamSecurity & Access

FILESTREAM is secured like other data:

GRANT permission on DB, schema, table or column.

NTFS permissions (ACLs) are ignored.

Only SQL Server service account is granted NTFS access to data

containers to avoid outside data editing.

Requires integrated security (Windows authentication) Windows

token pass client identity in file I/O operation.

Support snapshot isolation level (only in 2008 R2).

Remote access By Server Message Block (SMB) protocol.

Firewall considerations:

Enable both client & server to resolve DNS names.

Open file-sharing TCP ports: 139, 445 (for remote access).



Dual programing model to access BLOBs:

T-SQL for DMLs:

Insert can be used with null value, empty (‘’) value or short inline data.

Update if FILESTREAM field is set to null, the related BLOB is deleted.

Partial field update can’t be done with T-SQL (use UPDATE.Write()) since

update = create new copy of FILESTREAM data file + delete old file.

Partial update only by: FSCTL_SQL_FILESTREAM_FETCH_OLD_CONTENT

Delete or Truncate underlined BLOB data is deleted in file system.

File system streaming access (for read & write):

Statement model models T-SQL statements by using open & close

methods File handle is live in that scope open handle = open

transaction.

Storage namespace DB engine controls BLOB physical file new internal

function column_name.PathName(@option) used by API to provide the

logical UNC path of the BLOB to get the Win32 handle.



Transacted file system access New function provides token for current

session’s transaction GET_FILESTREAM_TRANSACTION_CONTEXT ()

binds file streaming operations with a started transaction.

Return NULL if there is no explicitly started transaction.

In C++/ODBC SQL Native Client 10 OpenSqlFilestream Win32 handle.

In C# .NET Framework 3.5 SP1 SqlFileStream Data Type path name +

transaction token handle can be passed to Win32 File API.

Write-through from remote client No write operation are cached on client

side better to make fewer writes with larger data size.


File Stream Demo

Demo


File StreamApplication Considerations

Use NULL instead of 0x for non-initialized FILESTREAM

column to avoid file creation.

Avoid insert & update in tables with NOT NULL FILESTREAM

columns decrease application performance over time.

In replication use NEWSEQUENTIALID() instead of NEWID()

for better performance.

Avoid T-SQL for update, append, prepend data to FILESTREAM

BLOB will be spooled into tempdb.

Avoid appending small BLOB updates Causes FILESTREAM

files to be copied Write BLOB into varbinary(max) parameter and

do single write operation.

Avoid getting lots of BLOB files length in application use

DATALENGTH() in T-SQL does not open the BLOB.


File StreamCombined with other features

Snapshots Only by excluding FILESTREAM file groups

FILESTREAM file groups will be marked as offline.

Replication Supported by both transactional & merge

with many considerations SQL 2008 or later on all subscribers.

Log Shipping supports FILESTREAM.

Mirroring Does not support FILESTREAM.

Clustering fully supported. Must enable FILESTREAM on all nodes

Backup & Restore Fully supported on all recovery models.

Encryption is not supported even when TDE is enabled.

Indexes cannot be a part of: index key / include column / statistics.

Constraints (PK,FK,UQ) cannot be created on FILESTREAM column.

Temp tables & TVP cannot be used in local/global temp tables & TVP.


File StreamCluster Configuration

Data containers must be on shared disk resource.

Separate LUN for FILESTREAM is recommended.

Steps for enabling FILESTREAM:

Enable file stream on service level on all nodes.

Windows Share Name must be the same on all nodes.

Allow remote clients access create a file-share cluster resource.

Enable file stream on instance level (facets) on all nodes.

Add dependency of FILESTREAM LUN to SQL Service.

Take SQL Service off & Bring SQL Service + dependencies on.

NetApp Snap Manager requirements:

Separate LUNs to data files, Log files & FILESTREAM files.


File StreamRBS

Complementary approach store BLOBs on remote storage.

Provided with SQL Server 2008 R2 Feature pack.

library API set moving BLOBs from MS SQL to external storage.

Set of SPs and SDK RBS provider for File stream.

Client-side DLL linked into a user application + stored procedures on

SQL Server.


File StreamUseful Links

FILESTREAM Storage in SQL Server 2008: http://msdn.microsoft.com/en-us/library/cc949109(v=sql.100).aspx

To BLOB or Not To BLOB - Large Object Storage in a Database or

a File system: http://research.microsoft.com/apps/pubs/default.aspx?id=64525

How to enable FILESTREAM from the command line: http://

sqlsrvengine.codeplex.com/wikipage?title=FileStreamEnable&referringTitle=Home&ProjectName=sqlsrvengine

FSUTIL syntax:http://technet.microsoft.com/en-us/library/cc785435(v=ws.10).aspx

SQL Server 2008 R2 Remote Blob Store (RBS) Download :http://www.microsoft.com/download/en/confirmation.aspx?id=16978

http://msdn.microsoft.com/en-us/library/cc949109(v=sql.100).aspx



http://research.microsoft.com/apps/pubs/default.aspx?id=64525



http://sqlsrvengine.codeplex.com/wikipage?title=FileStreamEnable&referringTitle=Home&ProjectName=sqlsrvengine




http://technet.microsoft.com/en-us/library/cc785435(v=ws.10).aspx

http://technet.microsoft.com/en-us/library/cc785435(v=ws.10).aspx

http://www.microsoft.com/download/en/confirmation.aspx?id=16978

http://www.microsoft.com/download/en/confirmation.aspx?id=16978


File StreamEnd

Thanks

isug 113: file stream

Technology