dbmirroring tips use shift enter to open the menu (new window)

April 2006 1SEAS06 SQL Server 2005 HA & DR

TopicsTopics Positioning of various technologiesPositioning of various technologies Backup/Restore considerations Backup/Restore considerations Windows Volume Shadow ServiceWindows Volume Shadow Service DB Mirroring Performance & DB Mirroring Performance &

DeploymentDeployment (Geo) Clustering(Geo) Clustering Peer-to-Peer ReplicationPeer-to-Peer Replication Third party technologiesThird party technologies Miscellaneous topicsMiscellaneous topics


Modes and FailoverModes and FailoverMirroringMirroring SynchronousSynchronous AsynchronouAsynchronou

ss

SafetySafety FullFull OffOff

WitnessWitness SetSet NoNo NoNo

ModeMode High High AvailabilitAvailabilityy

High High ProtectionProtection

High High PerformancePerformance

FailFail--oveoverr

AutoAuto SupportedSupported NoNo NoNo

ManuManualal

SupportedSupported SupporteSupportedd

ForceForce


Database Mirroring Database Mirroring Authentication Authentication

An Authentication ExampleAn Authentication Example On REMUSR03 create the endpoint & grant REMUSR04 permission to On REMUSR03 create the endpoint & grant REMUSR04 permission to

connect:connect:

CREATE ENDPOINT DbmEndpoint CREATE ENDPOINT DbmEndpoint STATE = STARTED STATE = STARTED AS TCP (LISTENER_PORT = 7022) AS TCP (LISTENER_PORT = 7022) FOR DATABASE_MIRRORING (AUTHENTICATION = WINDOWS, ROLE = ALL);FOR DATABASE_MIRRORING (AUTHENTICATION = WINDOWS, ROLE = ALL);

CREATE LOGIN [REDMOND\REMUSR04$] FROM WINDOWS;CREATE LOGIN [REDMOND\REMUSR04$] FROM WINDOWS;GRANT CONNECT ON ENDPOINT::DbmEndpoint TO [REDMOND\REMUSR04$];GRANT CONNECT ON ENDPOINT::DbmEndpoint TO [REDMOND\REMUSR04$];

On REMUSR04 create the endpoint & grant REMUSR03 permission to On REMUSR04 create the endpoint & grant REMUSR03 permission to connect:connect:

CREATE ENDPOINT DbmEndpoint CREATE ENDPOINT DbmEndpoint STATE = STARTED STATE = STARTED AS TCP (LISTENER_PORT = 7022) AS TCP (LISTENER_PORT = 7022) FOR DATABASE_MIRRORING (AUTHENTICATION = WINDOWS, ROLE = ALL);FOR DATABASE_MIRRORING (AUTHENTICATION = WINDOWS, ROLE = ALL);

CREATE LOGIN [REDMOND\REMUSR03$] FROM WINDOWS;CREATE LOGIN [REDMOND\REMUSR03$] FROM WINDOWS;GRANT CONNECT ON ENDPOINT::DbmEndpoint TO [REDMOND\REMUSR03$];GRANT CONNECT ON ENDPOINT::DbmEndpoint TO [REDMOND\REMUSR03$];


Database Mirroring Database Mirroring Authentication Authentication

AuthenticationAuthentication Use certificates if the NTLM/KERBEROS authentication not Use certificates if the NTLM/KERBEROS authentication not

possible. Mainly there are two cases for which this applies:possible. Mainly there are two cases for which this applies: The instances run on machines that are member of two completely The instances run on machines that are member of two completely

unrelated domainsunrelated domains One or two of the instances run as an account that cannot be One or two of the instances run as an account that cannot be

authenticated over the network (e.g. BUILTIN\LOCAL SERVICE or a authenticated over the network (e.g. BUILTIN\LOCAL SERVICE or a local account like MACHINENAME\UserName)local account like MACHINENAME\UserName)

Most commonly used accounts (LocalSytem, NETWORK Most commonly used accounts (LocalSytem, NETWORK SERVICE) authenticate over the network using the machine SERVICE) authenticate over the network using the machine account (DOMAIN\MachineName$)account (DOMAIN\MachineName$)

However, this account normally will not be granted login permission However, this account normally will not be granted login permission on any SQL instance nor connect permission on the DBM endpoint. on any SQL instance nor connect permission on the DBM endpoint.

The administrator would have to explicitly grant these permissionsThe administrator would have to explicitly grant these permissions Optionally create a group on the domain: e.g. 'REDMOND\Optionally create a group on the domain: e.g. 'REDMOND\

MirorringAllowedMachines' MirorringAllowedMachines' And grant connect permission on the endpoint to this accountAnd grant connect permission on the endpoint to this account Then add machine to this group for it to establish mirroring sessionsThen add machine to this group for it to establish mirroring sessions


Database Mirroring Database Mirroring Deployment Deployment 1 1

Useful setup tipsUseful setup tips Make sure the database has the same nameMake sure the database has the same name Check that the PORT numbers are correctCheck that the PORT numbers are correct

PORT column in sys.tcp_endpoints and column PORT column in sys.tcp_endpoints and column MIRRORING_PARTNER_NAME in sys.database_mirroring MIRRORING_PARTNER_NAME in sys.database_mirroring

Check that the ROLE information is correctCheck that the ROLE information is correct Columns TYPE_DESC and ROLE_DESC in Columns TYPE_DESC and ROLE_DESC in

sys.database_mirroring_endpointssys.database_mirroring_endpoints Make sure that the endpoints are startedMake sure that the endpoints are started

Check cCheck column STATE_DESC in olumn STATE_DESC in sys.database_mirroring_endpointssys.database_mirroring_endpoints

All the endpoints have compatible encryption settingsAll the endpoints have compatible encryption settings Column IS_ENCRYPTION_ENABLED in Column IS_ENCRYPTION_ENABLED in

sys.database_mirroring_endpoints sys.database_mirroring_endpoints Make sure that the login from the other server has Make sure that the login from the other server has

CONNECT permission.CONNECT permission. On the mirror, make sure that you do the restore with On the mirror, make sure that you do the restore with

NORECOVERYNORECOVERY


Database Mirroring Database Mirroring Deployment Deployment 22

SQL Server Agent JobsSQL Server Agent Jobs Need to be defined on both Principal & Need to be defined on both Principal &

Mirror Mirror User developed scripts needed for kicking User developed scripts needed for kicking

on the appropriate serveron the appropriate server Distributed TransactionsDistributed Transactions

For operational ease suggest avoiding For operational ease suggest avoiding distributed transactions if practicaldistributed transactions if practical

Principal and Mirror SQL Server instances Principal and Mirror SQL Server instances are two distinct resource managersare two distinct resource managers Manual resolution required to address the DTC Manual resolution required to address the DTC

transaction depending on status (in-flight, in-transaction depending on status (in-flight, in-doubt, or in-commit)doubt, or in-commit)


Performance Performance ConsiderationsConsiderations 11

Response time and throughput impact Response time and throughput impact varies;varies;impacted by work load and network impacted by work load and network qualityquality Online transaction processing (OLTP): mostly Online transaction processing (OLTP): mostly

read, light update – low to mediumread, light update – low to medium OLTP: heavy update – high OLTP: heavy update – high Same data updated by many transactions – Same data updated by many transactions –

high high Data Warehouse: light update – lowData Warehouse: light update – low Bulk load, index build & reorganization – high Bulk load, index build & reorganization – high

impact – requires sufficient bandwidthimpact – requires sufficient bandwidth


Performance Performance ConsiderationsConsiderations 22

Additional worker threads per database Additional worker threads per database With constant work loadWith constant work load

Redo_Queue should not be increasingRedo_Queue should not be increasing Will also affect failover timeWill also affect failover time Increases disk log Input/Output activity; log Increases disk log Input/Output activity; log

records to be read from disk instead of being records to be read from disk instead of being applied from memoryapplied from memory

If there is a spike in the work load, increases log If there is a spike in the work load, increases log acknowledgement latencyacknowledgement latency

Stress test thoroughly in your Stress test thoroughly in your environmentenvironment

Two customers report impact about 15 to 20% Two customers report impact about 15 to 20% Your mileage will varyYour mileage will vary


Perfmon CountersPerfmon CountersSQL Server: Database Mirroring SQL Server: Database Mirroring

ObjectObjectOn the Principal:On the Principal: Log Bytes Sent/secLog Bytes Sent/sec Log Send Queue KBLog Send Queue KB Transaction DelayTransaction Delay Transactions/secTransactions/sec Log Bytes Flushed/secLog Bytes Flushed/sec Disk Write Bytes/secDisk Write Bytes/sec

On the Mirror:On the Mirror: Redo Bytes/secRedo Bytes/sec Redo Queue KBRedo Queue KB Disk Write Bytes/secDisk Write Bytes/sec


Performance Performance Measurement Test Measurement Test

ResultsResults Significant ItemsSignificant Items Log generation rateLog generation rate Network latency and bandwidthNetwork latency and bandwidth Transaction safety levelTransaction safety level Concurrency (Number of connections)Concurrency (Number of connections) Transaction size and volumeTransaction size and volume

The impact depends on the work load

Test work load results follow

Your results will be different


Test WorkloadsTest Workloads

CharacteristicCharacteristic Workload1Workload1 Workload2Workload2

Database size (GB)Database size (GB) 4040 2020

Number of concurrent user Number of concurrent user connectionsconnections

10001000 2020

Maximum think time between Maximum think time between transactions (sec)transactions (sec)

44 00

Baseline (No Mirroring) %CPUBaseline (No Mirroring) %CPU 44 4040

Baseline (No Mirroring) Baseline (No Mirroring) Transactions / secTransactions / sec

241241 215215

Baseline (No Mirroring) log Baseline (No Mirroring) log generation rate (KB / sec)generation rate (KB / sec)

720720 1200012000


Transaction Safety vs. Transaction Safety vs. Performance: Workload1Performance: Workload1

Transaction Throughput for Workload1

200

210

220

230

240

250

No Mirroring Safety OFF Safety FULL

Transaction Safety Levels

Tran

sact

ions

/sec


Transaction Safety vs. Transaction Safety vs. Performance: Workload2Performance: Workload2

Transaction Throughput for Workload2

0

50

100

150

200

250

No Mirroring Safety OFF Safety FULL

Transaction Safety Levels

Tra

nsa

ctio

ns/

sec


Impact of Network Latency – Impact of Network Latency – Workload 1 -SyncWorkload 1 -Sync

Synchronous Mirroring with Network Latency for Workload1

0

50

100

150

200

250

2 14 20 50 100 200

Round Trip Time (ms)

Tra

nsacti

on

s /

sec

0

2

4

6

8

10

12

Resp

on

se T

ime (

sec)

Transactions/sec Response Time (sec)


Impact of Network Bandwidth Impact of Network Bandwidth – Workload 1 - Sync– Workload 1 - Sync

Synchronous Mirroring with varied Network Bandwidth for Workload1

0

50

100

150

200

250

300

1 10 100 1000

Network Bandwidth (Mbps)

Tra

nsacti

on

s/s

ec

0

5

10

15

20

Resp

on

se T

ime

(sec)

Transactions/sec Response Time (sec)


Impact of Network Latency – Impact of Network Latency – Workload 2 - SyncWorkload 2 - Sync


Impact of Network Latency – Impact of Network Latency – Workload 2 - AsyncWorkload 2 - Async


Database Mirroring Database Mirroring Monitoring - DMVMonitoring - DMV

sys.database_mirroring_endpointssys.database_mirroring_endpoints Contains one row for the database mirroring endpoint of the server instanceContains one row for the database mirroring endpoint of the server instance

SELECT e.name, e.protocol_desc, e.type_desc, e.role_desc, e.state_desc, SELECT e.name, e.protocol_desc, e.type_desc, e.role_desc, e.state_desc, t.port, e.is_encryption_enabled, e.encryption_algorithm_desc, t.port, e.is_encryption_enabled, e.encryption_algorithm_desc, e.connection_auth_desc e.connection_auth_desc FROM sys.database_mirroring_endpoints e JOIN sys.tcp_endpoints tFROM sys.database_mirroring_endpoints e JOIN sys.tcp_endpoints t ON e.endpoint_id = t.endpoint_id ON e.endpoint_id = t.endpoint_id

sys.database_mirroring_witnessessys.database_mirroring_witnesses Contains a row for every witness role that a server plays in a database mirroring Contains a row for every witness role that a server plays in a database mirroring

partnershippartnership

SELECT principal_server_name, mirror_server_name, database_name, SELECT principal_server_name, mirror_server_name, database_name, safety_level_descsafety_level_desc

FROM sys.database_mirroring_witnessesFROM sys.database_mirroring_witnesses

sys.dm_db_mirroring_connectionssys.dm_db_mirroring_connections Returns a row for each database mirroring network connectionReturns a row for each database mirroring network connection

SELECT login_time, connect_time, is_send_flow_controlled, total_bytes_sent, SELECT login_time, connect_time, is_send_flow_controlled, total_bytes_sent, total_bytes_received total_bytes_received

FROM sys.dm_db_mirroring_connectionsFROM sys.dm_db_mirroring_connections


Database Mirroring Database Mirroring MonitoringMonitoring

sys.database_mirroring DMVsys.database_mirroring DMV One row for each database in the instance of SQL ServerOne row for each database in the instance of SQL Server Not_NULL rows interestingNot_NULL rows interesting

SELECT d.name, d.database_id, m.mirroring_role_desc, SELECT d.name, d.database_id, m.mirroring_role_desc, m.mirroring_state_desc, m.mirroring_safety_level_desc, m.mirroring_state_desc, m.mirroring_safety_level_desc, m.mirroring_partner_name, m.mirroring_partner_instance,m.mirroring_partner_name, m.mirroring_partner_instance, m.mirroring_witness_name, m.mirroring_witness_state_descm.mirroring_witness_name, m.mirroring_witness_state_descFROM sys.database_mirroring m JOIN sys.databases dFROM sys.database_mirroring m JOIN sys.databases dON m.database_id = d.database_idON m.database_id = d.database_idWHERE mirroring_state_desc IS NOT NULLWHERE mirroring_state_desc IS NOT NULL

Database Mirroring State Change Profiler Event 167Database Mirroring State Change Profiler Event 167 A few interesting columns: Integer Data (Prior State Id), A few interesting columns: Integer Data (Prior State Id),

State (New State ID), TextData (Description of the state State (New State ID), TextData (Description of the state change)change)

Database Mirroring MonitorDatabase Mirroring Monitor


Database Mirroring Database Mirroring Monitor Monitor 11


Service Interruption Service Interruption

Database Failover Time

Err

or

Occ

urs

Da

tab

ase

Ava

ilab

le

Re

do

Co

mp

lete

Undo Processing

Err

or

De

tect

ed

Mirr

or

to S

erv

e

New Quorum

House Keeping

Time to apply Redo_Queue records

Hard Failure: Redo likely the dominant component on server availability

Soft Failure: Redo likely complete before

failure is declared

Hard Failure: Quick Detection

Soft Failure: Longer ~ 10 Secs

SERVER

Err

or

Occ

urs

Ap

plic

atio

n

No

tifie

d

Ap

plic

atio

n

Re

con

ne

cts

Hard Failure: Quick Notification

Soft Failure: Much Longer ~ 15-30 SecsHouse

Keeping

Su

cce

ssfu

l C

on

ne

ctio

n

SNAC connects to Primary & Mirror alternately till successful connection or time-out

Waiting for connection Application Processing

CLIENT

Interruption: Longer of the Client or

Server Process


Potential Issues and Potential Issues and SolutionsSolutions 11

Mirror Server or Database crashMirror Server or Database crash Rebuild mirror database using full, incremental and log Rebuild mirror database using full, incremental and log

backupsbackups Reestablish partnershipReestablish partnership

Should not disrupt serving users from the Principal Should not disrupt serving users from the Principal Witness Server crashWitness Server crash

Important role; impacts automatic failover Important role; impacts automatic failover Don’t install on the same server as Principal or MirrorDon’t install on the same server as Principal or Mirror Can be a failover cluster instance Can be a failover cluster instance Recovery options depend on type of failureRecovery options depend on type of failure

Introduce another server as Witness and reestablish Introduce another server as Witness and reestablish relationship or relationship or

Rebuild the server and reestablish the witness roleRebuild the server and reestablish the witness role Should not disrupt serving users when Witness rejoinsShould not disrupt serving users when Witness rejoins

Recommendation: If practical locate at third siteRecommendation: If practical locate at third site


Potential Issues and Potential Issues and Solutions Solutions 22

Database Mirroring depends on responses Database Mirroring depends on responses from network stack in deciding whether from network stack in deciding whether partner is alive partner is alive

Problem resolution requires working with Problem resolution requires working with infrastructure and network engineers; e.g.:infrastructure and network engineers; e.g.: Is WindowsIs Windows®® Firewall blocking a port? Firewall blocking a port? Are cables unplugged?Are cables unplugged? Is DNS working?Is DNS working? Has a (remote) server been re-booted?Has a (remote) server been re-booted? Has a server been renamed?Has a server been renamed? EtcEtc


Measuring Failover TimeMeasuring Failover Time

Use SQL Server ProfilerUse SQL Server Profiler Select the event “Database Mirroring State Change”Select the event “Database Mirroring State Change” Select columns “StartTime” and “TextData”Select columns “StartTime” and “TextData” Start Profiler and observe messages during failoverStart Profiler and observe messages during failover

Sample messages in SQL Server Profiler during failoverSample messages in SQL Server Profiler during failover

StartTimeStartTime TextDataTextData

2005-10-22 12:39:17.9602005-10-22 12:39:17.960 DBM: Synchronized Mirror with Witness -> DBM: DBM: Synchronized Mirror with Witness -> DBM: Connection with Principal LostConnection with Principal Lost

2005-10-22 12:39:18.5702005-10-22 12:39:18.570 DBM: Connection with Principal Lost -> DBM: DBM: Connection with Principal Lost -> DBM: Automatic FailoverAutomatic Failover

2005-10-22 12:39:20.5902005-10-22 12:39:20.590 DBM: Automatic Failover -> DBM: Principal DBM: Automatic Failover -> DBM: Principal Running ExposedRunning Exposed


An Example of Database An Example of Database Failover Times Failover Times

Failover times for Synchronous Mirroring with Witness for Workload1Failover times for Synchronous Mirroring with Witness for Workload1

Failure / Failover TypeFailure / Failover Type Failover Time with Failover Time with no load (sec)no load (sec)

Failover Time with Failover Time with normal load (sec)normal load (sec)

ALTER DATABASE FAILOVER ALTER DATABASE FAILOVER (manual failover)(manual failover)

20.5 20.5 32.232.2

SHUTDOWN WITH NOWAITSHUTDOWN WITH NOWAIT 5.75.7 8.58.5

Stop SQL Server ServiceStop SQL Server Service 3.23.2 10.810.8

Shutdown principal serverShutdown principal server 10.210.2 15.815.8

Power OFF principal serverPower OFF principal server 2.22.2 2.62.6

Pull network cable on Pull network cable on principalprincipal

1.91.9 3.13.1

Failover under load takes longer than under no-load. Failover under load takes longer than under no-load. Manual failover takes longer than automatic failover.Manual failover takes longer than automatic failover.


Mirror Server Mirror Server Configuration Configuration 11

RecommendationsRecommendations Same as partner server (CPU, memory, storage, Same as partner server (CPU, memory, storage,

networking)networking) Mirror writes continuously, not just at checkpointsMirror writes continuously, not just at checkpoints I/O subsystem: plan to handle volume on Mirror I/O subsystem: plan to handle volume on Mirror

Same service pack and patch levels for the Same service pack and patch levels for the operating system and SQL Server as Principal operating system and SQL Server as Principal (exception: rolling upgrade)(exception: rolling upgrade)

Same SQL Server edition as Principal Same SQL Server edition as Principal Identical directory structure for the SQL Server bits Identical directory structure for the SQL Server bits

and database files on both partners and database files on both partners Same SQL Server configuration (trace flags, startup Same SQL Server configuration (trace flags, startup

options, memory settings, etc.) as Principal options, memory settings, etc.) as Principal


Mirror Server Mirror Server Configuration Configuration 22Recommendations …Recommendations …

Identical SQL Server logins / permissions as on Identical SQL Server logins / permissions as on Principal Principal

Copy from Principal - SQL Agent jobs, alerts, Copy from Principal - SQL Agent jobs, alerts, SSIS packages, support databases, linked SSIS packages, support databases, linked server definitions, backup devices, maintenance server definitions, backup devices, maintenance plans, etc. plans, etc.

If adding a disk volume on the Principal => Do If adding a disk volume on the Principal => Do the same on the Mirrorthe same on the Mirror

Develop a process to replicate all changes made Develop a process to replicate all changes made on the Principal on to the Mirror on the Principal on to the Mirror

Test failover, and make sure the application Test failover, and make sure the application works the same on the Mirror works the same on the Mirror


Suspend and ResumeSuspend and Resume Suspend and Resume ALTER commandsSuspend and Resume ALTER commands Log records keep accumulating at the Principal Log records keep accumulating at the Principal

Transaction logs cannot be truncated (even if you Transaction logs cannot be truncated (even if you back up the log)back up the log)

Plan for enough log space depending on generation Plan for enough log space depending on generation rate and suspension durationrate and suspension duration

The log will potentially fill up and database will The log will potentially fill up and database will become unavailablebecome unavailable

Either RESUME or break the mirroring session Either RESUME or break the mirroring session Truncation and reuse prevention indicated byTruncation and reuse prevention indicated by

Value “Database_Mirroring” in column Value “Database_Mirroring” in column “Log_Reuse_Wait_Desc” of Sys.database catalog “Log_Reuse_Wait_Desc” of Sys.database catalog viewview


Planned Downtime Planned Downtime With Safety FULL, potential for rolling upgradeWith Safety FULL, potential for rolling upgrade Exercise the steps for each planned downtime first Exercise the steps for each planned downtime first

in the test environment and document each step in the test environment and document each step prior to production environmentprior to production environment

Suspend the mirroring session and perform the Suspend the mirroring session and perform the change on the mirrorchange on the mirror Principal is running exposed Principal is running exposed

Resume the session – wait for SYNCHONIZED Resume the session – wait for SYNCHONIZED statestate

Switch the principal and mirrorSwitch the principal and mirror In-flight transactions will be rolled backIn-flight transactions will be rolled back Better to stop and redirect the application if practicalBetter to stop and redirect the application if practical

Perform the changes on the old principal Perform the changes on the old principal Once available, it will assume the Mirror roleOnce available, it will assume the Mirror role With Safety OFF, theoretically possible to switch With Safety OFF, theoretically possible to switch

to FULL for rolling upgradeto FULL for rolling upgrade FIRST EVALUATE THE IMPACT OF THE CHANGEFIRST EVALUATE THE IMPACT OF THE CHANGE


Rules of Thumb for Capacity Rules of Thumb for Capacity PlanningPlanning

Database MirroringDatabase Mirroring Very Rough Rules of Thumb for resourceVery Rough Rules of Thumb for resource

planningplanning Additional CPU – synchronous mode – 1/3Additional CPU – synchronous mode – 1/3rdrd more more

Total CPU – not to exceed 50% busy Total CPU – not to exceed 50% busy Design network for low latency. Ping:Design network for low latency. Ping:

Synchronous: < a couple of msecSynchronous: < a couple of msec Asynchronous < 10 msec Asynchronous < 10 msec

Adequate bandwidth – 3 times peak log bytes shipped per second. Adequate bandwidth – 3 times peak log bytes shipped per second. If insufficient:If insufficient:

Synchronous – performance degradesSynchronous – performance degrades Asynchronous – mirror falls behindAsynchronous – mirror falls behind

Extra log space on primaryExtra log space on primary Delays on mirror can hold up truncation on primaryDelays on mirror can hold up truncation on primary Avoid log full conditionAvoid log full condition


Manual Failover: Messages in Manual Failover: Messages in ERRORLOGERRORLOG

Starting up database '<db_name>'.Starting up database '<db_name>'.

Analysis of database '<db_name>' Analysis of database '<db_name>' (<dbid>) is 100% complete (<dbid>) is 100% complete (approximately 0 seconds remain). (approximately 0 seconds remain).

Recovery of database '<db_name>' Recovery of database '<db_name>' (<dbid>) is 100% complete (<dbid>) is 100% complete (approximately 0 seconds remain). (approximately 0 seconds remain). Phase 3 of 3.Phase 3 of 3.

Manual failover takes longer than automatic failover.Manual failover takes longer than automatic failover.

dbmirroring tips use shift enter to open the menu (new window)

Documents

seas06 sql server

database mirroring object

mirror sql server instances

column mirroring

mirroring check

perfmon counters sql

sql instance

mirroring sessions