www.novell.com practical nds ® imonitor: case studies in novell edirectory ™ diagnosis duane buss...

www.novell.com

Practical NDS® iMonitor: Case Studies in Novell eDirectory™ Diagnosis

Practical NDS® iMonitor: Case Studies in Novell eDirectory™ Diagnosis

Duane BussSenior Software EngineerNovell, Inc.dbuss@novell.com

Tom DomanSenior Software EngineerNovell, Inc.tdoman@novell.com

Steve McLainSenior Software EngineerNovell, Inc.smclain@novell.com

Gary J. PorterSenior Network AnalystMindWorks, Inc.porter@myrealbox.com

Vision…one NetA world where networks of all types—corporate and public, intranets, extranets, and the Internet—work together as one Net and securely connect employees, customers, suppliers, and partners across organizational boundaries

MissionTo solve complex business and technical challenges with Net business solutions that enable people, processes, and systems to work together and our customers to profit from the opportunities of a networked world

Deployed Versions Novell eDirectory™ and Novell Directory Services® (NDS®)

Product Version Build Version

Platforms

NetWare 5.1 SP4 (NDS 7) DS.nlm v7.57 NetWare 5.1

NetWare 5.1 SP 4 (NDS 8) DS.nlm v8.79 NetWare 5.1

eDirectory 8 DS.nlm & DS.dlm v8.79

NetWare 5.0,Win NT/2K

eDirectory 8.5.x DS v85.23 NetWare 5.x,Win,Solaris

NetWare 6 (eDirectory 8.6) DS.nlm v10110.20 NetWare 6

eDirectory 8.6.1 DS v10210.43 NW 5.1,NW 6,Win,Solaris,Linux

NetWare 6 SP1 (eDirectory 8.6.2)

DS.nlm v10310.17 NetWare 6

eDirectory 8.6.2 DS v103xx.xx NW 5.1,NW 6,Win,Solaris,Linux

eDirectory 8.7 DS v10410.xx NW 5.1,NW 6,Win,Solaris,Linux,AIX

Differences between eDirectory and Novell Directory Services (NDS)

NetWare 6

NetWare

NDS eDirectory

NOS directory focused on managing NetWare® servers

A cross-platform, scalable, standards-based directory

used for managing identities that span all aspects of the network—eDirectory

is the foundation for eBusiness

NetWare 5

Introduction

• Historical diagnostic tools• Problems vs. symptoms• eDirectory diagnostic case studies using

iMonitor The case of the unknown object The case of the attribute that just wouldn’t

sync The case of the inconsistent replica The case of the security-minded administrator The case of the inconsistent entry The case of the under-performing agent

Historical NDS Diagnostic Tools

• Diagnostic tools DSTrace DSBrowse DSRepair DSDiag NDS Manager

• Tool access Server console Remote console Telnet pcAnywhere Insight Board

The eDirectory Utility Knife

• Multiple tools in one

• You don’t leave the tool to go to another one• You don’t have to access several different

servers

Agent SummaryAgent SummaryAgent ConfigurationAgent Configuration

NDS TraceNDS TraceNDS RepairNDS Repair

DirXMLDirXML™™ Tools ToolsReport ToolReport Tool

Search ToolSearch Tool

Agent HealthAgent HealthObject and Schema BrowseObject and Schema Browse

Agent SynchronizationAgent Synchronization

Partition ListPartition ListKnown ServersKnown ServersAgent Process StatusAgent Process Status

Agent Activity, Verb and Event StatisticsAgent Activity, Verb and Event Statistics

Plus:Plus:

Inbound and Outbound Connection MonitorInbound and Outbound Connection MonitorError InformationError Information

What Kind of Problems Can I Diagnose?

• What problems are you seeing? Time sync issues Synchronization issues NDS agent version problems Communication issues Schema issues Improperly moved/removed servers Inconsistent object/database Agent process errors Performance issues DirXML™ issues Distributed issues Many others…

Disclaimer

The following case studies are based on real-world scenarios and depict systems which have been, at times, brutally battered, beaten, or otherwise mistreated. Viewer discretion is advised. The names of the perpetrators have been changed to protect the guilty. Their crimes include

Inappropriate usage of undocumented/advanced support switchesin NDS Repair

Hardware failure Improper removal of servers and/or replicas from the tree Incorrect system configuration Overzealous administration Running pre-release (beta) code in production Attempting to correct symptoms of the real problem Poor network infrastructure and/or monitoring

Case #1

unknowobject

the case of the

Unknown Object Causes

• An object referenced by a mandatory attribute has been deleted

• Object is only a forward reference

• Object is an External Reference and the object has not yet been Backlinked, or the real object is unknown

• Object has Auxiliary Classes and you are viewing the object on a non-Aux Class compatible replica

• Object is being deleted

• Object is actually damaged (rare)

• Schema inconsistencies (rare)

• Ghost Objects (extremely rare)

Unknown Object—Missing Mandatory

• Detecting the case Examine the attributes “Unknown Base Class”

and “Unknown Auxiliary Class”

Compare the mandatory attributes required in the schema to the attributes on the object

Unknown Object—Missing Mandatory

Unknown Object—Missing Mandatory

• Resolving the issue Don’t panic Is the missing attribute missing on all replicas or

just some of the replicas• If the attribute is missing on all replicas, add the missing

attribute using LDAP, ConsoleOne®, or iManage (the object will remain unknown)

• If the object is consistent on some replicas but not others use iMonitor to resend that one object from the consistent replica to the other replicas

As a last resort, remove the object, then recreate it

Unknown Object—Forward Reference

• Detecting the case Entry information flags show “Reference” The replica type shown in the entry information

is something other than subordinate The object may not have all attributes Walking the replica ring shows the

object is not unknown on all replicas

Unknown Object—Forward Reference

Unknown Object—Forward Reference

• Resolving the issue Don’t panic, forward references happen all the

time in the course of synchronization and will become known when the actual object successfully synchronizes

Check for and resolve any schema and object sync problems, then wait for the sync operation to finish

In rare cases use “Single Object Send” to send the entry from a consistent replica to all other replicas

Unknown Object—External Reference

• Detecting the case Entry information flags show “Reference” There are not ‘real’ server names in the replica

frame The partition type is subordinate The attribute list is abbreviated

although the authenticated user has full rights to the object being viewed

Unknown Object—External Reference

Unknown Object—External Reference

• Resolving the issue Don’t panic—this is not generally a problem External References are only viewable in

iMonitor or DSBrowse If the entry information flags show “Temporary

Reference,” by design, this server may never receive the base class of the real object

Check and resolve any errors shown in “Agent Process Status” in the External Reference section

Start the “Reference Check” background process and wait for it to complete

Unknown Object—Aux Class

• Detecting the case Check the version of the servers in the replica

ring Examine the “AuxClass Object Class Backup,”

“auxClassCompatibility,” and “Object Class” attributes

Unknown Object—Aux Class

Unknown Object—Aux Class

• Resolving the issue Don’t panic Not a problem, it is safe to ignore these

unknowns Upgrade older servers to 8.x or later version of

eDirectory and apply appropriate service patches

Unknown Object—Deleted

• Detecting the case Entry information flags don’t show “Present” There may be obituary attributes on the object These objects are only visible in utilities

such as iMonitor

Unknown Object—Deleted

Unknown Object—Deleted

• Resolving the issue This object will generally finish deleting without

manual intervention Wait for sync to finish Run the “Purger” background process

Unknown Object—Ghost Object

• Detecting the case Entry information flags show “Reference” Walking the replica ring shows the object is

unknown on all replicas

Unknown Object—Ghost Object

Unknown Object—Ghost Object

• Resolving the issue Delete the object if it is not needed

Case #2

Attribute Mismatch?

Attribute Mismatch?

Filter Desired Attributes

Getting to Replica Synchronization

Replica Synchronization

Getting to Entry Synchronization

Entry Synchronization

Take Action?

Schema Definition

Release Version 8.6 and Later

Houston, We Have a Problem Obituary Report

Entry with Obituaries

Houston, We Have a Problem Unknown Objects

Houston, We Have a Symptom

Case #3

replica

the case of the

inconsiste t

Houston, What Exactly Is the Problem? Using NDS Trace

Analyzing NDS Trace Data

Target NDS Agent

Update

Packet

Switching to Trace on Another Server

Switching to Trace on Another Server

Switching to Trace on Another Server

Houston, What Exactly Is the Problem?Using NDS Trace

More Info on -609

NDS Error Information

Inspect William Object

Inspect Schema Class Definition

MandatoryMandatory

Filter Desired Attributes

Compare the Compare the Object Around Object Around the Replica the Replica RingRing

Aaaaaaha!Aaaaaaha!

Quicker Check of Synchronization

Resynchronize All Data from the Master

Case #4

Obituary ReportOne Entry Still Has Not Been Fully Processed

Report Configuration

Report ConfigurationServer Information

ReportServer Information

ReportServer Information

Obviously, the 609-6 NDS Agent Is Up

Agent Process StatusLimber

Agent Process StatusLimber

Inspect the Server Entry

This is an External Reference

This is a real copy of the 609-6 object

Aaaaaaha! An Overzealous

Security-Minded Administrator

Case #5

Agent Synchronization Error

Replica Synchronizati

on Detail

Replica Synchronization Detail

Replica Synchronization Detail

Inconsistent Object

Go to NDS Repair

Single Object Repair

Troubleshooting Guidelines

• Don’t panic• Look for root causes, not symptoms• After taking steps to correct a problem, make sure

the correct background processes run to completion

• Get training• General rules

1. Solve communication problems first2. Solve synchronization issues first (Schema and Object)3. Make sure your system is correctly time-synced4. Run the correct (latest) support pack

Case #6

eDirectory Performance Factors

• Replica type and placement• NDS version mix• Overall tree design—• IO115—Directory or Database: Choosing the Right Tool

for the Job• TUT223—Avoiding the Top eDirectory Issues

• TUT329—Novell eDirectory Deployment

• TUT33—eDirectory In Depth

• Load and application distribution• Hardware and network capability• Database cache settings• Index definitions• Bindery usage

Agent Activity

Agent Activity

Agent Configuration

Fault to Hit Ratio

Current vs.

Maximum

New in eDirectory 8.5

Profiling Data

Conclusion

• Historical diagnostic tools• Problems vs. symptoms• eDirectory diagnostic case studies using

iMonitor The case of the unknown object The case of the attribute that just wouldn’t sync The case of the inconsistent replica The case of the security-minded administrator The case of the inconsistent entry The case of the under-performing agent