transaction log internals - meetupfiles.meetup.com/12803752/ssseug.20150203.[dba... · sydney sql...

Sydney SQL Server Enterprise User Group February 2015

Transaction LogInternals

Victor Isakov

[DBA]


Victor Isakov

Victor Isakov is a Database Architect / Trainer / Author who provides consulting and training services to various organizations in the public, private and NGO sectors globally. He participates in different capacities at various international events and conferences. Victor specializes in:

• Designing SQL Server infrastructure solutions and upgrade strategies

• Architecting OLTP, DSS and DW database solutions

• Performance troubleshooting and optimization

• Customized on premise training and mentoring

Email [email protected]

Blog www.victorisakov.com

Twitter @victorisakov

LinkedIn www.linkedin.com/in/victorisakov

Website www.sqlserversolutions.com.au


Session

SQL Server's transaction log is critical to performance, let alone data durability and transaction integrity. It is critical for a DBA to understand how it works, how to best manage it and optimize it's performance.

In this session Victor Isakov (MCA, MCM, MCT, MVP) will examine the "internals" of the transaction log, including logging/recovery, checkpoints and the WAL protocol. Along the way he will discuss best practices around performance optimization and management before looking at the internal structure of the log.


Agenda

• Database Architecture

• Recovery

• Write-Ahead Logging

• Recovery Models

• Transaction Log Architecture

• Writing and Reading

• Analysis

• Delayed Durability


Database Architecture

• Back in the days…• SELECT * FROM syslog

• Re-architected in SQL Server 7.0

• Databases are made up of 3 files• Primary Data File

• Contain system and user tables

• Secondary Data File• Can only contain user tables

• Transaction Log File• New binary format


Data Files

• Used to store user data (tables, indexes, BLOBs, etc)

• Formatted into extents (64KB)• Extents formatted in 8 x pages (8KB)

• Rows stored with page

• Can associate data file(s) with FILEGROUP logical container

• Random and sequential I/O pattern

• Database engine pre-fetches pages from files into Buffer Pool through Read-Ahead Manager

• Buffer Pool uses write-back strategy

• Data is never modified directly • Record modification in transaction log first


Log Files

• Capture state changes that are made within database• More than just DML statements

• Can be very verbose

• Changes are written and hardened to log before changes are attempted on data files

• Implement Atomicity and Durability of relational database’s ACID properties

• Sequential I/O pattern

• No performance benefit of having more than one log file


ACID Properties

• Atomicity• All changes happen or nothing happens

• Consistency• Data is transformed from one correct state to another

• Isolation• Concurrent updaters are prevented from interfering with one another

• Durability• Committed changes remain permanent despite failures


Log File Placement

• Separate log file from the data files

• Consider underlying RAID array• RAID 5 or RAID 6 typically a poor choice due to overhead of parity calculation

and write penalty

• RAID 1 pairs or RAID 10 a better choice

• Be careful of automatically putting onto you fastest storage• With sequential I/O the performance gap between HDDs and SSD is less

compared to random I/O

• IT ALL DEPENDS ON YOUR ENVIRONMENT!!!


Log File Best Practices

• AUTOSHRINK database option

• Autogrow database file option

• Zero out


Automatic Recovery

• Automatic recovery occurs whenever a database is “started” by the database engine• SQL Server starts up

• Failover occurs in a failover cluster instance

• Database brought online

• Database attached

• 3 Phases in Automatic Recovery


Automatic Recovery Phases

• Analysis Phase• Analyse transaction log• Read the transaction log and find where the last CHECKPOINT process occurred

• All operations after the last CHECKPOINT were never written to disk• Dirty Pages were lost in memory

• It does not matter if the transaction was committed

• Redo Phase• Replays all transactions after the last CHECKPOINT process

• Committed transactions are written to disk

• Database is now available in Enterprise Edition

• Undo Phase• Roll back any uncommitted transactions• May require multiple passes of reading the transaction log• Starts at oldest active transaction


Demo

• Automatic recovery• BEGIN TRAN

• Modify data

• Dirty Shutdown

• Start SQL Server

• Examine error log


Write-Ahead Logging

• Transaction log changes need to be written to the disk before they can be attempted on the data• Commonly referred to as “hardening” the log

• Enforces Atomicity• Transaction occurs in it’s entirety or not at all

• Ensures rollback operations can be performed

• Enforces Durability• Changes “hardened” first to stable media

• Ensure that transaction can be replayed if required


CHECKPOINT

• Writes all dirty pages to the data file• Transaction state does not matter

• Effectively the “write signature” of a database

• Affects the Automatic Recovery process’ REDO Phase• Remember, we never loose data due to the WAL Protocol

• Can be very I/O intensive

• It does not remove pages from the Buffer Pool• That is the responsibility of the Lazy Writer


CHECKPOINT Process

• Writes a record to the log file, marking the start of the checkpoint.

• Stores information recorded for the checkpoint in a chain of checkpoint log records.

• One piece of information recorded in the checkpoint is the log sequence number (LSN) of the first log record that must be present for a successful database-wide rollback. This LSN is called the Minimum Recovery LSN (MinLSN). The MinLSN is the minimum of the: • LSN of the start of the checkpoint• LSN of the start of the oldest active transaction• LSN of the start of the oldest replication transaction that has not yet been delivered to the

distribution database

• The checkpoint records also contain a list of all the active transactions that have modified the database.

• If the database uses the simple recovery model, marks for reuse the space that precedes the MinLSN.

• Writes all dirty log and data pages to disk.

• Writes a record marking the end of the checkpoint to the log file.

• Writes the LSN of the start of this chain to the database boot page.


Controlling CHECKPOINT Process

• sp_configure ‘recovery interval’ • Default of 0

• SQL Server 2012 introduced TARGET_RECOVERY_TIME database option

• SQL Server 2014 introduced concept of DELAYED_DURABILITY• Trade performance for data loss

• Log blocks are only flushed to disk when they reach their maximum size of 60KB

• Transaction commits do not cause log blocks to be fushed by themselves


Delayed Durability

• Disabled: This is the default setting and very similar to full transaction durability.

• Allowed: This option will allow each transaction to decide the delayed durability. Once this enables each transactions’s durability will be based on the transaction level level settings which will see later in this post.

• Forced: This option will force each transaction to follow Delayed Durability.


Delayed Durability


Recovery Models

FULL • All operations are fully logged• Log truncated with LOG BACKUP

SIMPLE • Log truncated with CHECKPOINT

BULK_LOGGED • Some operations are minimally logged• Trade-off of transaction log overhead versus LOG BACKUP

• When do you want to pay the penalty?• POTENTIAL FOR DATA LOSS


Minimally Logged

• Certain operations can be minimally logged.

• List of operations that are minimally logged under BULK_LOGGED or SIMPLE recovery models:• Bulk import operations ( BCP, BULK INSERT and INSERT..SELECT )• SELECT INTO operations• TRUNCATE• Partial updates to large value data types, using the .WRITE clause.• CREATE INDEX,ALTER INDEX REBUILD• DROP TABLE• Partition Switch• Merge (If 610 Trace flag is enabled)• Starting in SQL Server 2008 , INSERT SELECT statement can also be handled

with minimal logging


Minimally Logged

• Prerequisites for Minimal Logging : • Table should not be replicated

• TABLOCK should be used

• You can specify TABLOCK with the command or turn on ‘table lock on bulk load’ table option .


Trace Flag 610

• SQL Server 2008 introduces trace flag 610, which controls minimally logged inserts into indexed tables

• As with minimal logging for heap, the database must be set to BULK_LOGGED or SIMPLE recovery model

• The input data must be sorted in the index key order

• Read the The Data Loading Performance Guide whitepaper• https://technet.microsoft.com/en-us/library/dd425070(v=sql.100).aspx

https://technet.microsoft.com/en-us/library/dd425070(v=sql.100).aspx


Clear as mud…


Log Backups

• Archives the log records that are no longer needed for REDO / UNDO• Reclaims portions of the log that are no longer needed

• Affected by open, long running transaction

• Not available in SIMPLE recovery model

• With BULK_LOGGED recovery model the log backup includes the extents that have been modified by the minimally logged operations

• Does not shrink the log file

• FYI: Full Database Backup includes Log Backup


Transaction Log Architecture

• The transaction log is a wrap-around file

• Each physical log file is divided internally into a number of virtual log files (VLFs)

• New log records are added at the end of the logical log and expand toward the end of the physical log

• Log truncation frees any virtual logs whose records all appear in front of the minimum recovery log sequence number (MinLSN)

• The MinLSN is the log sequence number of the oldest log record that is required for a successful database-wide rollback


Log Architecture


Virtual Log File

• A VLF is a logical unit of truncation within transaction log files

• Generated at log creation or log growth

• Number of VLFs can impact database solution

Log File Size Number of VLFs

<= 1MB 2

>=1MB and < 64MB 4

>=64MB and < 1GB 8

1GB and larger 16


VLF Implications

• Too many VLFs will cause database engine prbems with reading the log and restoring the database• Affects availability

• Too few causes more growth as it takes too long to truncate log

• SQL Server 2012 alerts you via ERROGLOG if you have too many

• Watch out for autogrowth of log files!

Error 9017 Database dbName has more than n virtual log files which is excessive. Too many virtual log files can cause long startup and backup times. Consider shrinking the log and using a different growth increment to reduce the number of virtual log files.


DBCC LOGINFO

• DBCC LOGINFO• Parity = (0, 64, 128)

• Status = (0, 2)


DBCC LOGINFO

FileID • The FileID number as found in sysfiles

FileSize • The size of the VLF in bytes

StartOffset • The start of the VLF in bytes, from the front of the transaction log

FSeqNo • Indicates the order in which transactions have been written to the different VLF files. The VLF with the highest number is the VLF to which log records are currently being written.

Status • Identifies whether or not a VLF contains part of the active log. • A value of 2 indicates an active VLF that can't be overwritten.

Parity • The Parity Value• Which can be 0, 64 or 128

CreateLSN • Identifies the LSN when the VLF was created. A value of zero indicates that the VLF was created when the database was created. If two VLFs have the same number then they were created at the same time, via an auto-grow event.


DEMO

• DBCC LOGINFO

• Show circular nature

• This helps with shrinking log file


Log Blocks

• The unit of physical commit to the transaction log file• 512 bytes – 60 KB

• Stored in the Buffer Pool• Log Buffer

• Write-through

• Obviously more efficient to have larger transactions (60KB)

• Frequent smaller transactions cause frequent log flushing• Remember Delayed Durability…

• Can control via sp_flush_log system stored procedure


Log Record

• Atomic modification within database

• Uniquely identified by Log Sequence Number (LSN)

• Contains information to redo and undo a transaction

• [tempdb]’s transaction log works differently• No need for REDO information, for example


Log Sequence Number (LSN)

• Monotonically increasing identifier

• Uniquely identifies log record

• (VLF Number : Log Block Number : Log Record Number)


Demo

• DBCC LOGINFO

• fn_dblog

• Relationship between FSeqNo and LSN


Why is my log so large? And growing

• Never been backed up

• Last VLF is still active• Remember we shrink on VLFs boundaries from the end

• Long running transaction

• Large transaction

• VLF can only be reused if:• No active tranasctions are contain in VLF or previous VLFs• No unreplicated transactions

• Remember that Replication can cause problems with SIMPLE recovery model

• Remediation• Determine cause• Create “overflow” log file


SELECT log_reuse_wait_desc FROM sys.databases

• Use to help determine why log is not being reused

• States:• NOTHING • CHECKPOINT • LOG_BACKUP • ACTIVE_BACKUP_OR_RESTORE • ACTIVE_TRANSACTION • DATABASE_MIRRORING • REPLICATION • DATABASE_SNAPSHOT_CREATION • LOG_SCAN • OTHER_TRANSIENT


Reducing VLFs

• Shrink log to 1MB

• Use DBCC SHRINKFILE

• Resize log to decent amount

• Change autogrowth to avoid problem• Don’t use percentages…

• Use decent size


Log Analysis

• DBCC LOG

• SELECT * FROM fn_dblog(NULL,NULL)


Demo

• DBCC LOG

• SELECT * FROM fn_dblog(NULL,NULL)

• Excel


Summary

• Questions?

transaction log internals - meetupfiles.meetup.com/12803752/ssseug.20150203.[dba... · sydney sql...

Documents