ad disasters & how to prevent them

71
AD Disasters AD Disasters …and How to Prevent Them! …and How to Prevent Them! Greg Shields, MVP, Greg Shields, MVP, vExpert vExpert Head Geek, Concentrated Technology www.ConcentratedTech.com

Upload: concentrated-technology

Post on 05-Dec-2014

1.162 views

Category:

Technology


6 download

DESCRIPTION

 

TRANSCRIPT

Page 1: Ad disasters & how to prevent them

AD DisastersAD Disasters…and How to Prevent Them!…and How to Prevent Them!

Greg Shields, MVP, vExpertGreg Shields, MVP, vExpertHead Geek, Concentrated Technologywww.ConcentratedTech.com

Page 2: Ad disasters & how to prevent them

This slide deck was used in one of our many conference presentations. We hope you enjoy it, and invite you to use it

within your own organization however you like.

For more information on our company, including information on private classes and upcoming conference appearances, please

visit our Web site, www.ConcentratedTech.com.

For links to newly-posted decks, follow us on Twitter:@concentrateddon or @concentratdgreg

This work is copyright ©Concentrated Technology, LLC

Page 3: Ad disasters & how to prevent them

AgendaAgenda

Topics– Part I: Hardware & Software Failures– Part II: Human Errors– Part III: Complete Disasters

3

Page 4: Ad disasters & how to prevent them

Three Types of DisastersThree Types of Disasters

4

Hardware&

Software

HumanError

CompleteDisasters

MicrosoftScrewed Up

YouScrewed Up

SomebodyREALLY

Screwed Up

Page 5: Ad disasters & how to prevent them

Three Types of DisastersThree Types of Disasters

5

Hardware&

Software

HumanError

CompleteDisasters

Increasing Problem ComplexityIncreasing Troubleshooting Complexity

Page 6: Ad disasters & how to prevent them

Part IPart IHardware & Software Hardware & Software FailuresFailures

Page 7: Ad disasters & how to prevent them

Morphed SYSVOL FoldersMorphed SYSVOL Folders

Problem: When SYSVOL replication finds a name conflict, one of the folders in conflict is renamed to foldername_ntfrs_???????? (hex)– This can break the links to that folder.

This can occur when users attempt to manually replicate folders, two users add folders of the same name at the same time, or during an improper restore of the SYSVOL.

7

Page 8: Ad disasters & how to prevent them

Morphed SYSVOL FoldersMorphed SYSVOL Folders

Solution: Three-steps:– Rename all morphed folders to new names and allow

replication of the new names to fully complete. This ensures a common name for the folder is available on all DC’s and that the new names and GUID’s match.

– Once replication has completed, look in the folders and determine which is correct and which does not belong.

– Rename the correct folder back to its original name and again allow replication to complete. Delete the unnecessary folder.

– This is OK as FRS tracks files by their GUID.

8

Page 9: Ad disasters & how to prevent them

Broken GPT/GPC LinkagesBroken GPT/GPC Linkages

Problem: Group Policy Objects are made up of two parts, the Group Policy Template and the Group Policy Container.– GPC’s are stored in Active Directory and replicate

through AD replications.

– GPT’s are stored in the SYSVOL and replicate through FRS.

Broken GPT/GPC linkages can cause GPO’s to malfunction and should be fixed.

Same with mismatched version numbers.

9

Page 10: Ad disasters & how to prevent them

Broken GPT/GPC LinkagesBroken GPT/GPC Linkages

Solution: Use GPOTOOL.EXE from the Resource Kit Tools to identify GPT’s/GPC’s that are not synchronized.

GPOTOOL.EXE with no switches validates and reports on the GPT/GPC linkage.

If someone has accidentally changed permissions on the GPT, you can also use the /checkacl switch.

GPMC will notify when permissions are not consistently set and request to reset those permission.

GPMC will reset the permission on the GPT to match the permission on the GPC.

10

Page 11: Ad disasters & how to prevent them

Broken GPT/GPC LinkagesBroken GPT/GPC Linkages

11

Page 12: Ad disasters & how to prevent them

DNS Aging & Scavenging DNS Aging & Scavenging Not EnabledNot Enabled Problem: If DNS Aging & Scavenging is not

enabled on a domain, stale DNS records caused by DHCP lease changes can pile up.

Group Policy application as well as correct name resolution requires a one-to-one mapping between FQDN and IP.

Stale DNS records mean multiple IP’s per host and/or multiple host records per IP.

Systems management tools like SMS can fail.

12

Page 13: Ad disasters & how to prevent them

13

Aging Aging & Scavenging& Scavenging When DNS was static, keeping active and inactive

records straight was a nightmare. Now that DNS is dynamic, inactive recordkeeping is

improved, when configured correctly.

Aging– All dynamically updated resource records have a time stamp

– That time stamp is reset whenever a record is created, modified, or refreshed.

– Windows hosts refresh their record…At startupAt DHCP lease renewalEvery 24 hours

Page 14: Ad disasters & how to prevent them

14

Aging Aging & Scavenging& Scavenging Windows DNS servers that accept dynamic updates need to

have Scavenging enabled or records will quickly grow stale.– This is especially problematic if DHCP is active and has a short

lease time.

Be aware the DNS scavenging on AD-integrated zones can have an impact on AD replication.

Refresh Interval– If a client does not refresh its record by the end of this period,

the scavenging process will remove the record.

No-Refresh Interval– A period of time before the refresh interval where client

refreshes are ignored by the server.

– This is done to reduce DNS replication requirements.

Scavenging increases AD replication

Page 15: Ad disasters & how to prevent them

15

Aging Aging & Scavenging& Scavenging

7 Days

RecordCreated

7 Days

7 Days 7 Days

7 Days 7 Days

7 Days 7 Days

No RefreshInterval

RefreshInterval

RefreshAccepted

RefreshAccepted

Record Deleted

Time

Page 16: Ad disasters & how to prevent them

16

Aging Aging & Scavenging& Scavenging

Global Setting Per-Zone Setting

Let’s discuss strategies for Aging & Scavengingin Small, Large, & Enterprise Networks…

Page 17: Ad disasters & how to prevent them

DNS Aging & Scavenging DNS Aging & Scavenging Not EnabledNot Enabled Solution: Enable DNS Aging & Scavenging on

all zones populated by DHCP. DNS Aging & Scavenging enabled in two

locations.

17Global Setting Per-Zone Setting

Page 18: Ad disasters & how to prevent them

DNS Aging & Scavenging DNS Aging & Scavenging Not EnabledNot Enabled Solution: Use DNSCMD.EXE command-line tool

to automatically age and scavenge all records after enabling Aging & Scavenging

DNSCMD.EXE ageallrecords DNSCMD.EXE startscavenging

18

Page 19: Ad disasters & how to prevent them

Disable Unused Network Disable Unused Network CardsCards Problem: Unused network cards can auto-

populate DNS with incorrect entries.– With regular servers this doesn’t often cause a big

problem, but with DC’s, auto-registration populates SRV records as well.

– This can cause bad resolution to DC services.

Solution: Disable any unused network cards.– Disabling unused network cards prevents them from

registering their incorrect values into DNS.

19

Page 20: Ad disasters & how to prevent them

Tombstones & ZombiesTombstones & Zombies Problem: When an AD object is deleted, it goes

into a special container called “Deleted Items”. It’s movement there is replicated. The object is not removed until the tombstone lifetime is exceeded.– Windows 2000 tombstone lifetime is 60 days

– Windows 2003 tombstone lifetime is 180 days

– Upgraded Windows 2003 tombstone lifetime is still 60 days.

When a DC comes back on-line after being down for longer than the tombstone lifetime or a restore from a tape older than the tombstone lifetime, zombies are created.

20

Page 21: Ad disasters & how to prevent them

Tombstones & ZombiesTombstones & Zombies Solution: Never bring on-line a DC that’s been

down for greater than 60/180 days. Never use tapes to restore objects older than 60/180 days.

If you do, you’re in a world of hurt. …but what if you forget…?

21

Page 22: Ad disasters & how to prevent them

Lingering ObjectsLingering Objects

Problem: So, you’ve gone ahead and accidentally reanimated a tombstoned object? What now?

Reanimation of these lingering objects can break replication in some cases.

22

Page 23: Ad disasters & how to prevent them

Lingering ObjectsLingering Objects

Solution: Use REPADMIN.EXE /REPLSUM from the Support Tools to verify if lingering objects are resident in Active Directory. REPADMIN.EXE to remove them. These tools only work on W2003.

Step 1: Find the GUID of a DC:– repadmin.exe /showrepl

Step 2: Check for lingering objects:– repadmin.exe /removelingeringobjects * <DC GUID>

dc={mydomain},dc={com} /advisory_mode

Step 3: Remove any lingering objects found:– Remove the /advisory_mode switch from Step 2.

23

Page 24: Ad disasters & how to prevent them

Lingering ObjectsLingering Objects

Solution: Use REPADMIN.EXE /REPLSUM from the Support Tools to verify if lingering objects are resident in Active Directory. REPADMIN.EXE to remove them. These tools only work on W2003.

Step 4: Enable strict replication consistency.– Strict replication consistency is only enabled by default on

2003 DCs (not upgraded) that were promoted into a Forest that was built as 2003 (not upgraded from 2000).

– All other DCs will only have this setting enabled manually.

– Enable strict replication consistency on all DC’s by setting the DWORD value for Strict Replication Consistency to 1 at the key HKLM\System\CurrentControlSet\Services\NTDS\Parameters.

24

Page 25: Ad disasters & how to prevent them

Improper Time SynchronizationImproper Time Synchronization

Problem: Time synchronization is critical for Kerberos authentication and many applications.

Time skew greater than 5 minutes can prevent logins and cause log files to barf.

Users with administrator rights can reconfigure time sync to another time server.

Very slight differences in time between stratum 1, 2, and 3 servers, usually caused by Internet conditions.

Using different time servers in a network can cause problems for time-sensitive network applications.

25

Page 26: Ad disasters & how to prevent them

Improper Time SynchronizationImproper Time Synchronization

Solution: Configure all machines in the domain to synchronize against the same time server.

Choose to use NT5DS or NTP mode, but choose one for all systems.– NT5DS is accurate to ~20 seconds.

– NTP can be accurate to <1 second.

Some applications require greater time resolution, so consider a 3rd party time sync tool with an on-site stratum 3 time device.– “Domain Time” from Symmetriccom

26

Page 27: Ad disasters & how to prevent them

Bad DNS SRV RecordsBad DNS SRV Records

Problem: Improperly decommissioning DC’s can lead to their SRV records not being expunged from the DNS database.

Also, missing DNS SRV records can prevent AD from functioning properly.

This can cause error messages in the Event Log, replication problems, etc. due to the missing server.

This happens most often when AD DNS is not hosted on Windows and dynamic updates are not enabled.

27

Page 28: Ad disasters & how to prevent them

Bad DNS SRV RecordsBad DNS SRV Records

Solution: Ensure DNS SRV records are consistent.

Use ipconfig /registerdns to force DC to re-register DNS SRV records along with it’s A records.

Be careful of multiple interfaces on DC’s. Disable any unused interfaces.– Unused interfaces can register themselves in DNS.

– Bridged interfaces can cause routing problems.

Delete stale DNS SRV records from DNS database (you’ll know which are stale).

28

Page 29: Ad disasters & how to prevent them

Orphaned Domains & DCOrphaned Domains & DC’’ss

Problem: Old domains and Domain Controllers are still resident in Active Directory.

These extra domains are unnecessary, can cause Event Log errors and odd problems during contact attempts.

Orphaned DC’s can prevent a domain from being decommissioned.

Orphaned DC’s in child domains can prevent a parent domain from being decommissioned.

29

Page 30: Ad disasters & how to prevent them

Orphaned Domains & DCOrphaned Domains & DC’’ss

Solution: Remove the offending Domains and/or DC’s from your infrastructure. This is a multi-step process.

NTDSUTIL.EXE to remove from Active Directory ADSIEDIT.MSC to remove from LDAP DNSMGMT.MSC to remove from DNS

30

Page 31: Ad disasters & how to prevent them

Orphaned Domains & DCOrphaned Domains & DC’’ss

Solution: Remove the offending Domains and/or DC’s from your infrastructure. This is a multi-step process.

NTDSUTIL.EXE to remove from Active Directory– NTDSUTIL– METADATA CLEANUP– CONNECTIONS– CONNECT TO SERVER {SERVER NAME}– QUIT– SELECT OPERATION TARGET– SELECT SERVER {SERVER NAME}– REMOVE SELECTED SERVER | QUIT

31

Page 32: Ad disasters & how to prevent them

Orphaned Domains & DCOrphaned Domains & DC’’ss

Solution: Remove the offending Domains and/or DC’s from your infrastructure. This is a multi-step process.

ADSIEDIT.MSC to remove from LDAP.– This step is required if the domain is not at W2003 SP1.

– Navigate to DC={MYDOMAIN},DC={COM},OU=DOMAIN CONTROLLERS

– Delete the offending Domain Controller.

DNSMGMT.MSC to remove from DNS– Delete any FQDN’s and/or associated GUID’s related to

that DC.32

Page 33: Ad disasters & how to prevent them

Stale AD Site LinksStale AD Site Links

Problem: AD Site Links are usually created and managed by the KCC. However, some administrators want to get their hands in on replication.

Once Site Links are manually created, the KCC no longer manages them, which can cause them to grow stale as the network changes.

33

Page 34: Ad disasters & how to prevent them

Stale AD Site LinksStale AD Site Links

Solution: (Except in the very largest of networks) Remove any manually created Links and allow the KCC to manage links.

In Windows 2003 SP1, the link-managing capabilities of the KCC are improved by multiple orders of magnitude.

Older versions in larger networks had timing problems with KCC optimization passes.

Also, improperly decommissioned DC’s may need to be removed from AD S&S.

34

Page 35: Ad disasters & how to prevent them

No DNS Reverse ZonesNo DNS Reverse Zones Problem: DNS reverse zones must be enabled

for proper functionality of Active Directory.

Needed so clients can identify the site they reside in.

Needed so clients can find the closest DNS server.

Needed for correct processing of some attributes of Group Policy.

35

Page 36: Ad disasters & how to prevent them

No DNS Reverse ZonesNo DNS Reverse Zones Solution: Enable DNS reverse zones for each

zone active in your network infrastructure.

Ensure that all zones have similar configuration and dynamic updates enabled.

Don’t forget Aging & Scavenging.

36

Page 37: Ad disasters & how to prevent them

DSRM Passwords UnknownDSRM Passwords Unknown

Problem: Directory Services Restore Mode passwords are set individually on each Domain Controller as that controller is DCPROMO’ed.

This is arguably the most forgotten password in a Windows network because it is only used again during a restore operation.

Not having this in a crisis can inhibit restoration activities.

37

Who here knows their DSRM password?

Page 38: Ad disasters & how to prevent them

DSRM Passwords UnknownDSRM Passwords Unknown

Solution: Run NTDSUTIL.EXE to reset DSRM passwords before a failure occurs.

NTDSUTIL.EXE SET DSRM PASSWORD RESET PASSWORD ON SERVER {Server Name} {Enter New Password} {Re-Enter New Password} QUIT / QUIT (Consider “bagging” the password…)

38

Page 39: Ad disasters & how to prevent them

DSRM Passwords UnknownDSRM Passwords Unknown

Solution: Windows Server 2008 + KB961320 enables DSRM password synchronization to a domain account.

Create a standard domain user.– This user does not need to be a member of any special groups

or the Domain Admins group.

NTDSUTIL SET DSRM PASSWORD SYNC FROM DOMAIN ACCOUNT <userName> This process can also be scheduled via a GPP

scheduled task– “SET DSRM PASSWORD” “SYNC FROM DOMAIN ACCOUNT

<userName>” Q Q

39

Page 40: Ad disasters & how to prevent them

Why 2008 R2 is a goodWhy 2008 R2 is a goodidea for ADidea for AD AD Module for PowerShell and PowerShell

cmdlets– Every AD task is now automate-able via PowerShell

AD Administrative Center– Improved, task-based GUI for ADUC

AD Recycle Bin– Tough to use, but better than Authoritative Restore

AD Best Practices Analyzer– Are you the weakest link in your AD infrastructure?

Offline Domain Join– Handy for W7 upgrades and VDI

40

Page 41: Ad disasters & how to prevent them

Why 2008 R2 is a goodWhy 2008 R2 is a goodidea for ADidea for AD Managed Service Accounts

– Eliminate service account nightmares

AD Web Services & AD Management Gateway– Simplified PowerShell and 3rd party management

integration

Authentication Mechanism Assurance– Deliver a different set of resources when users login via

smart cards.

AD OpsMgr Management Pack– If you haven’t incorporated OpsMgr yet, see me after

class…

41

Page 42: Ad disasters & how to prevent them

A Review of Useful AD LogsA Review of Useful AD Logs

NTDS Diagnostics Logging By default, AD only records critical and error

events to the Directory Service log. OK during normal operations, but during problem

troubleshooting additional logging is necessary.

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\NTDS\Diagnostics

Set any of the 24 subkey’s DWORD value to a number between 0 and 5

42

Page 43: Ad disasters & how to prevent them

A Review of Useful AD LogsA Review of Useful AD Logs

43

Page 44: Ad disasters & how to prevent them

A Review of Useful AD LogsA Review of Useful AD Logs

Extended DCPROMO Logging During a W2003 DCPROMO, two log files are

created in %systemroot%\debug: dcpromo.log and dcpromoui.log.

The log level on dcpromoui.log can be increased to help when troubleshooting promotions/demotions.

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Current Version\AdminDebug\dcpromoui

Set the DWORD value for LogFlags to FF0003 (hex).

44

Page 45: Ad disasters & how to prevent them

A Review of Useful AD LogsA Review of Useful AD Logs

NETLOGON Logging Hunt down problems with client log-ins,

repeatedly locked-out accounts and log-in activity across forest trusts by increasing the log level on NETLOGON.

HKEY_LOCAL_MACHINE\SYSTEM\Current ControlSet\Services\NetlogonParameters

Set the DWORD value for DBFlag to 2080FFFF (hex), then restart the NETLOGON service.

NETLOGON.log is found in %systemroot%\debug.

45

Page 46: Ad disasters & how to prevent them

A Review of Useful AD LogsA Review of Useful AD Logs

Kerberos Logging Increasing the Kerberos logging level can track

down problems with disabled or expired accounts, missing usernames, and clock synchronization.

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Kerberos\Parameters

Set the DWORD value for LogLevel to 1 and look for events in the System Event Log.

46

Page 47: Ad disasters & how to prevent them

A Review of Useful AD LogsA Review of Useful AD Logs

USERENV Debug Logging This logging helps identify problems with

loading/unloading of user profiles, login/logout delays, and Group Policy application.

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon

Set the DWORD value for UserEnvDebugLevel to 0x00030002 and watch %systemroot%\debug\usermode\userenv.log

Match wall clock time.

47

Page 48: Ad disasters & how to prevent them

A Review of Useful AD LogsA Review of Useful AD Logs

GPO Client Logging To troubleshoot the enumeration and application

of GPO’s, increase the log level for GPO application at the client.

HKEY_LOCAL_MACHINESoftware\Microsoft\ Windows\CurrentVersion\Diagnostics

Set the DWORD value for RunDiagnosticLoggingGroupPolicy to 1, reboot the system and watch the Application Event Log.

48

Page 49: Ad disasters & how to prevent them

A Review of Useful AD LogsA Review of Useful AD Logs

Group Policy Logging changes with Vista/08– With Vista/08, Group Policy elements are

moved to their own process.Out of WinLogon

– Enabling Group Policy logging is now done by setting the DWORD value for GpSvcDebugLevel to 10002 for HKLM\Software\Microsoft\Windows NT\CurrentVersion\ Diagnostics

– The userenv debug log is now moved to %systemroot%\debug\usermode\gpsvclog.log

– The format of this log is easier to use, aligned with the movement of GP to a service.

49

Page 50: Ad disasters & how to prevent them

Part IIPart IIHuman ErrorsHuman Errors

50

Page 51: Ad disasters & how to prevent them

GPOGPO’’s Not Easily Restorables Not Easily Restorable

Problem: Group Policy Objects can be restored if they’re accidentally deleted, but this process involves a complicated authoritative restore, et cetra.

This authoritative restore can be a source of downtime to complete the restore and takes a while to complete.

51

Page 52: Ad disasters & how to prevent them

GPOGPO’’s Not Easily Restorables Not Easily Restorable

Solution: Create a Scheduled Task that backs up all GPO’s in the Domain to a text file using GPMC.– This task will use both scripts to ensure that GPO settings

and any logon/logoff/startup/shutdown scripts are also saved. Ensure that this text file is part of the nightly backup scheme.

cscript.exe %PROGRAMFILES%\gpmc\scripts\ BackupAllGPOs.wsf %SYSTSEMDRIVE%\backup\ GPOData /domain:<DomainFQDN>

cscript.exe %PROGRAMFILES%\gpmc\scripts\ GetReportsForAllGPOs.wsf %SYSTSEMDRIVE%\ backup\GPOReports

(Can also backup and restore in R2 with PowerShell)52

Page 53: Ad disasters & how to prevent them

Incorrect FSMO PlacementIncorrect FSMO Placement

Problem: Incorrect FSMO role placement in domains where all DC’s are not GC’s can cause a loss of data.

The Infrastructure Master role cannot reside on a Domain Controller that also runs the Global Catalog role

Except in the situation where the forest contains a single domain and all Domain Controllers are Global Catalogs.

To check FSMO role placement,NETDOM QUERY FSMO

53

Page 54: Ad disasters & how to prevent them

Incorrect FSMO PlacementIncorrect FSMO Placement

Solution: Either enable the Global Catalog role on all Domain Controllers or move the Infrastructure Master role to a DC that is not a GC.

NTDSUTIL.EXE ROLES CONNECTIONS CONNECT TO SERVER {Server Name} QUIT TRANSFER {Role} QUIT | QUIT

54

Page 55: Ad disasters & how to prevent them

Fat FingerFat Finger(Not that I(Not that I’’m calling your finger fat)m calling your finger fat)

Problem: You’ve done it again and accidentally deleted a series of objects or an entire OU from AD.

The default configuration of Active Directory allows everyone with administrative access to delete any object in AD.– Though, we all make mistakes…

In W2008, OU properties include a box box “Protect this Object/Container from Accidental Deletion”.– But we’re not at W2008 yet!

55

Page 56: Ad disasters & how to prevent them

Fat FingerFat Finger(Not that I(Not that I’’m calling your finger fat)m calling your finger fat)

Solution: That checkbox, revealed in W2008 is actually just a skin for a supported feature of W2003.

By checking this box:– “Deny Delete” and “Deny Delete Subtree” permissions for

the Everyone group are set on the object.

– You can set these permissions manually for any AD object or container inside the ADUC:Enable ADUC Advanced FeaturesNavigate to the Object, select Properties, and view the Security tabApply the Advanced privileges “Deny Delete” and “Deny Delete Subtree” to the object

56

Page 57: Ad disasters & how to prevent them

Unnecessary Apps InstalledUnnecessary Apps Installed

Problem: Every additional application installed to a server is an expansion of that server’s attack surface.– WinZip versions prior to v10.

– Java JRE prior to Version 5.

– Real Player

– Office

– Acrobat

Solution: Never install applications to your Domain Controllers. Ensure that any apps installed are always patched.– There’s more to patching than just Microsoft patching.

57

Page 58: Ad disasters & how to prevent them

Letting DCPROMO Do DNSLetting DCPROMO Do DNS

Problem: Generally a bad idea to let DCPROMO handle configuring DNS for Active Directory.

Tends to do a poor job of it, if at all. Better in W2008.

Solution: Ensure DNS properly configured before starting a DCPROMO process.

Three tests:– nslookup dchostname– nslookup dchostname.dcdomainame.com– nslookup 10.1.3.4

If success, no errors, and no time delays, then OK.58

Page 59: Ad disasters & how to prevent them

VM-level Backups for AD DRVM-level Backups for AD DR

Problem: With virtualized Domain Controllers using VM-level backups to backup DC’s can corrupt AD.

USN number mismatch between restored DC and existing DC’s.– Which USN’s have correct high water mark?

Solution: Always use authoritative/non-authoritative restore for AD DR. Never VM-level backups.

In fact, never use VM-level backups for any transactional database for the same reason.

Want more justification? http://support.microsoft.com/kb/888794

59

Page 60: Ad disasters & how to prevent them

Part IIIPart IIICompleteCompleteDisastersDisasters

60

Page 61: Ad disasters & how to prevent them

Snapping an Offline DC VMSnapping an Offline DC VM

Problem: Need to create an offline DC VM for testing purposes, but am concerned about lingering objects.

Solution: Use this process…– Create a new site in AD

– Add a member server VM to the domain in the new site.

– DCPROMO.

– Wait for replication to complete, then shut down the DC.

– Copy/Paste the virtual machine, then restart the DC.

– Demote this DC back to a member server and remove it from the production network.

– Start the DC, reconfigure network, and seize all FSMO roles.

– Use the new DC to complete testing.61

Page 62: Ad disasters & how to prevent them

What you Need to Back UpWhat you Need to Back Up

Problem: What exactly needs to be backed up to ensure a successful DC restore. A successful authoritative restore of the AD database.

Solution: Never try to restore the AD database from one DC to another DC.

So, all files that make up that DC must be backed up:– C:\

– System State

62

Page 63: Ad disasters & how to prevent them

Lack of Defined DR Policy & Lack of Defined DR Policy & ProceduresProcedures

Problem: Most companies do not have a defined DR Policy and DR Restoration Procedures.

This is usually the case because the project can get over-scoped. Consider just the steps necessary to start a recovery.

Solution: Build a simple DR plan and recovery steps.

Does not need to be complicated. Just the basic steps necessary to start recovery.– When you’re under the spotlight, you don’t want to be

searching for recovery steps on TechNet…63

Page 64: Ad disasters & how to prevent them

3 DR Scenarios3 DR Scenarios Scenario 1: A subset of objects within Active

Directory or the SYSVOL is accidentally or maliciously removed from the database.

Scenario 2: An Active Directory domain controller is functionally and irrecoverably down and must be rebuilt to return to operations.

Scenario 3: The entire Active Directory forest and domain is functionally and irrecoverably down and must be rebuilt to return to operations.

64

Page 65: Ad disasters & how to prevent them

Scenario 1: Deleted ObjectsScenario 1: Deleted Objects

Locate a DC that is also a GC. Disconnect this server from the network.

Reboot that server into Directory Services Restore Mode using the DSRM password.

Restore the AD database to the DC from tape or file backup (non-authoritative).

Perform an authoritative restore of the deleted object:– NTDSUTIL

– AUTHORITATIVE RESTORE

– RESTORE SUBTREE {Object to Restore}<Object to Restore> is the DN of the object to restore.For example, to restore the Accounts OU, the DN would be “OU=Accounts,DC={MyDomain},DC={com}”

– QUIT / QUIT

65

Page 66: Ad disasters & how to prevent them

Scenario 1: Deleted ObjectsScenario 1: Deleted Objects

Reconnect the DC and reboot the DC into normal operations.

Ensure the restored object has replicated to all DC’s in the domain.

As the DC reboots from DSRM mode, it will generate .LDF files that include back-link information for the restored objects.– As an example, back-links are groups the object is a member of.

These files are of the formatar_{date}-{time}_links_{Domain Name}.ldf.

Restore the back-links for each file found:– ldifde –i –k –f ar_{date}-{time}_links_{Domain Name}.ldf

66

Page 67: Ad disasters & how to prevent them

Scenario 2: A DC Goes DownScenario 2: A DC Goes Down

Validate the DC is completely failed and a restoration is not feasible.

If the DC is functional, but the AD database is corrupt, attempt a forced demotion:– DCPROMO /FORCEREMOVAL

Remove the failed server’s server objects from a functioning DC:– NTDSUTIL

– METADATA CLEANUP

– REMOVE SELECTED SERVER {DN of Server}

– QUIT | QUIT

Within the active DNS for the domain, manually remove any references to the failed DC and its SID in either A or SRV records.

67

Page 68: Ad disasters & how to prevent them

Scenario 2: A DC Goes DownScenario 2: A DC Goes Down

Build a replacement server at the same Service Pack and patch level.

DCPROMO the member server Validate a complete promotion and verify the AD database

has resynchronized to the domain.

68

Page 69: Ad disasters & how to prevent them

Scenario 3: Corrupted ForestScenario 3: Corrupted Forest

As of the last time I checked, Microsoft PSS has never been called to perform a complete forest restoration.

Validate that the complete Active Directory is completely and irreparably failed and a restoration is not feasible.

Call Microsoft PSS at 800-936-2200 and declare a Priority 1 “Crit-Sit”.

http://www.microsoft.com/downloads/details.aspx? displaylang=en&FamilyID=3EDA5A79-C99B-4DF9-823C-933FEBA08CFE

69

Page 70: Ad disasters & how to prevent them
Page 71: Ad disasters & how to prevent them

This slide deck was used in one of our many conference presentations. We hope you enjoy it, and invite you to use it

within your own organization however you like.

For more information on our company, including information on private classes and upcoming conference appearances, please

visit our Web site, www.ConcentratedTech.com.

For links to newly-posted decks, follow us on Twitter:@concentrateddon or @concentratdgreg

This work is copyright ©Concentrated Technology, LLC