slides - exchange 2007 sp1 core roles troubleshooting

Exchange Server 2007 SP1Core Roles Troubleshooting

João [email protected]

Agenda

Introduction to Exchange Server 2007 Troubleshooting

Troubleshooting Client Access Server (CAS)

Troubleshooting Hub Transport Server (HT)

Troubleshooting Mailbox Server (MBX)

Microsoft Confidential2

Introducing to Exchange Server 2007 Troubleshooting

Introduction

Exchange Services Overview

Active Directory Provider


Exchange Services shared by all Server Roles


Microsoft Exchange Active Directory Topology (MSExchangeADTopology)

Provides Active Directory topology information to several Exchange Server components

This service does not have any dependencies

EDGE not dependent

Microsoft Exchange Monitoring (MSExchangeMonitoring)

Provides a remote procedure call (RPC) server, used to invoke diagnostic cmdlets.

Microsoft Exchange Transport Log Search(MSExchangeTransportLogSearch)

Provides message tracking and transport log searching

UM not dependent

Exchange Client Access Role Services


Microsoft Exchange File Distribution Service (MSExchangeFDS)

Used to distribute offline address book

Dependent upon AD Topology Service

Microsoft Exchange IMAP4 (MSExchangeIMAP4)

Provides IMAP4 services to IMAP clients


Microsoft Exchange POP3 (MSExchangePOP3)

Provides POP3 services to POP3 clients


Microsoft Exchange Service Host (MSExchangeServiceHost)

Config of RPC virtual directory in IIS

Registry Config or ValidPorts, NSPI Interface Protocol Sequences, and AllowAnonymous for Outlook Anywhere


Exchange Hub Transport Role Services


Microsoft Exchange Transport (MSExchangeTransport)

Provides SMTP server and transport stack


Microsoft Exchange EdgeSync (MSExchangeEdgeSync)

Connects to ADAM instance on subscribed Edge Transport servers over secure LDAP

If there are no Edge Subscriptions configured, this service can be disabled


Microsoft Exchange Anti-spam Update (MSExchangeAntispamUpdate)

Auto download anti-spam filter updates

Exchange Edge Role Services


Microsoft Exchange Transport (MSExchangeTransport)

Provides SMTP server and transport stack

Microsoft Exchange ADAM (ADAM_MSExchange)

Stores configuration and recipient data on the Edge Transport server

Microsoft Exchange Credential Service (EdgeCredentialSvc)

Monitors credential changes in ADAM and installs the changes on the Edge Transport server

Microsoft Exchange Anti-spam Update (MSExchangeAntispamUpdate)

Auto download anti-spam filter updates

Exchange Mailbox Role Services


Microsoft Exchange Information Store (MSExchangeIS)

Manages Exchange Server databases

Provides data storage for messaging clients

Dependent upon : Event Log, NT LM Security Support Provider, Remote Procedure Call (RPC), Server, and Workstation

Microsoft Exchange System Attendant (MSExchangeSA)

Provides monitoring, maintenance, and directory lookup services

Dependent upon: Event Log, NT LM Security Support Provider, Remote Procedure Call (RPC), Server, and Workstation

Microsoft Exchange Mail Submission Service (MSExchangeMailSubmission)

Notifies Hub Transport server located in the Mailbox server's Active Directory to pickup from a sender's outbox


Exchange Mailbox Role Services cont


Microsoft Exchange Mailbox Assistants(MSExchangeMailboxAssistants)

Provides functionality for Calendar Attendant, Resource Booking Attendant, Out of Office Assistant, and Managed Folder Mailbox Assistant.


Microsoft Exchange Replication Service (MSExchangeRepl)

Provides log shipping functionality for LCR, CCR and SCR.


Microsoft Exchange Service Host (MSExchangeServiceHost)

Config of RPC virtual directory in IIS

Registry Config or ValidPorts, NSPI Interface Protocol Sequences, and AllowAnonymous for Outlook Anywhere


Exchange Mailbox Role Services cont


Microsoft Exchange Search Indexer (MSExchangeSearch)

Provides content to the Microsoft Search (Exchange Server) service for indexing.

Dependent upon AD Topology Service and the Microsoft Search (Exchange Server) service.

Microsoft Search (Exchange Server) (MSFTESQL-Exchange)

Provides full-text indexing of mailbox data content

Exchange-customized version of Microsoft Search

Dependent upon the Remote Procedure Call (RPC) service

Exchange Unified Messaging Role Services


Microsoft Exchange File Distribution Service (MSExchangeFDS)

Distribute custom Unified Messaging prompts


Microsoft Exchange Speech Engine (MSS)

Provides speech processing services

Dependent upon Windows Management Instrumentation service

Microsoft Exchange Unified Messaging (MSExchangeUM)

Provides Unified Messaging features:

Storing inbound faxes and voice mail messages

Access to mailbox via Outlook Voice Access

Dependent upon AD Topology Service and the Microsoft Exchange Speech Engine service


What is the AD Provider?

Components

Who uses the AD Provider?

Exchange Active Directory Topology Service

AD Topology vs Exchange Topology



The majority of components and services in Exchange 2007 are built on managed code:

Replay Service

Exchange Transport Service

Mailbox Assistants

Search

Unified Messaging

Unmanaged componentsInformation Store Service

System Attendant

DAV


Active Directory Provider: What is the AD Provider?

What is the AD Provider?New component in Exchange 2007 that leverages the advantages of being built on managed code

Provides an efficient, robust mechanism for managed applications to communicate with Active Directory

Loaded by all managed applications

Unmanaged applications continue to load DSAccess.dll

Improvements over DSAccess include

Support for paged queries

Support for very large multivalued attributes

Does not access and store directory information in a cache


Active Directory Provider: Components

AD Recipient Objects

AD System Configuration Objects

AD Driver

The engine inside the AD Provider

Provides the following functions:

Determines which server to connect to

Maintains connections pools

Performs connection failover

Manages Exchange and AD topology discovery

CRUD operations against AD

Provides interfaces to recipient and system configuration objects


Active Directory Provider: Who uses the AD Provider?

Who uses the AD Provider?Microsoft Exchange EdgeSync

Microsoft Exchange File Distribution

Microsoft Exchange Service Host

Microsoft Exchange Transport

Microsoft Exchange Transport Log Search

Microsoft Exchange Replication Service

Microsoft Exchange Mail Submission

Microsoft Exchange Mailbox Assistants

Microsoft Exchange Search Indexer

Microsoft Exchange Unified Messaging

Exchange Management Console

Exchange Management Shell

Setup


Active Directory Provider: Exchange AD Topology Service

Unmanaged Windows service that provides an RPC server interface in order to allow managed code processes access to AD topology information maintained by DSAccess

Effectively a wrapper for DSAccess that makes specific DSAccess functions available via RPC to the AD Driver running within managed code process

Dependency service for all managed applications


Active Directory Provider: AD Provider and AD Topology Service

AD Provider and AD Topology Service


Active Directory Provider: AD Topology vs Exchange Topology

Active Directory TopologyList of DCs and GCs in the local site and closest sites

Details about each server:

Is it reachable via ping?

Is it a DC or GC?

Is it synchronized?

Which domain it lives in?

OS version

SACL Right

This data is used in failover and load balancing


Exchange TopologyAD sites, site links and costs

Subnets

VDirs

Location of Exchange servers

Examples of use:

Mail Routing

Mapping of Client Access, Hub Transport and Unified Messaging server to the appropriate Mailbox server

PF referrals

Troubleshooting Client Access Server (CAS)

Agenda

Introduction

Overview

Locating CAS Configuration and Topology Data

Troubleshooting:

Autodiscover

Availability Service

Offline Address Book

Client Access Security

Outlook Web Access

Exchange ActiveSync


Introduction – What CAS is used for?

• AutoDiscover Service

• Availability Service

• Calendaring/Scheduling Assistant

• OOF configuration

• UM configuration

Outlook 2007 Clients only

• Outlook Anywhere

• Exchange ActiveSync

• POP3/IMAP4

• Outlook Web Access

Previous Version Clients


Introduction – CAS Role

At least one in each AD site that contains Mailbox server roleIt is NOT supported to work with no CAS server

CAS role can be combined with any other role except:Edge Transport Server role

A server in a cluster


Agenda

Introduction

Overview


Troubleshooting:

Autodiscover




Outlook Web Access

Exchange ActiveSync


Client Access Server - Overview


Site A

Site B

Outlook Express

Outlook Web AccessWindows Mobile

HTTPS

HTTPSIMAP4/POP3

SMTP

Encrypted RPC(MAPI)

Outlook 2007(via Outlook Anywhere)

Outlook 2003/2007(via MAPI)

Outlook

Anywhere

RPC

(MAPI)

Encrypted RPC(MAPI)

Encrypted RPC(MAPI)

Client Accessserver roles

Hub Transportserver roles Mailbox

server roles

Mailboxserver roles

Hub Transportserver roles

Client Accessserver roles

HTTPS

Encrypted RPC(MAPI)

SMTP over TLS

Agenda

Introduction

Overview


Troubleshooting:

Autodiscover




Outlook Web Access

Exchange ActiveSync


Locating CAS Configuration and Topology data

Critical Data Location

Microsoft Office Outlook Web

Access Web Site and Web.config

file

File System \Client Access\Owa

IMAP4 and POP3 protocol settings File system \ClientAccess\PopImap

Availability service Active Directory configuration container and

file system, including the Web.config file

\ClientAccess\exchweb\ews

Autodiscover •IIS metabase

•Active Directory configuration container

Exchange ActiveSync •Active Directory configuration container

•File system, including the Web.config file in

the \ ClientAccess\Sync folder

•IIS Metabase

Outlook Web Access virtual

directories

Active Directory configuration container and

file system \ClientAccess\

Web services configuration IIS metabase


Agenda

Introduction

Overview


Troubleshooting:

Autodiscover




Outlook Web Access

Exchange ActiveSync


Autodiscover

What Autodiscover does:Automatically configure Outlook profiles without knowing where the mailbox is located

Provide Web Service URLs to Outlook 2007 Clients

Use both RPC and HTTPS connection

What Autodiscover doesn‘t configure:

Cache mode vs. Online mode

Security Settings

Remote Mail Settings

Anything under Tools/Options in Outlook


Service Connection Point

CAS Installation will create:A new virtual directory called „AutoDiscover“ in IIS

Used by Outlook 2007 Clients only

Service Connection Point (SCP) in Active Directory which contains authoritative list of AutoDiscover Service URLs

CN=<CAS_server>,CN=AutoDiscover,CN=Protocols,CN=<CAS_Server>,CN=Servers,CN=Exchange Administrative Group,CN=AdministrativeGroup, CN=<Organization>,CN=Services,[Configuration Naming Context]

SCP Objects are accessed by domain joined Outlook 2007 clients to locate AutoDiscover service

Non domain joined clients rely on DNS to locate AutoDiscoverservice


Autodiscover

Configuring Outlook 2007 profiles and Web services URLs


Exchange

captures Outlook request, and

builds specific

connection settings for

Outlook AD

XML

Config

AutoDiscover

Service

HTTP

Request

Configuration

Information

AD

Lookup

autodiscover.contoso.com

Outlook 2007

[email protected]

1 Outlook uses e-mail address

to locate an Exchange Client Access servers at a

pre-defined location

(autodiscover.domain.com)

2

Configuration settings

are downloaded by Outlook and applied

to profile

3

Client

Access

server

role

0 If domain joined, Outlook

automatically fills out the user‟s email address and

password

•Outlook Anywhere

settings

• Server locations

• Web service URLs

• Authentication

information

• OAB download

location

Locating Autodiscover

To locate AutoDiscover:Internal domain joined clients use SCP

Non domain joined or external clients use DNS

For Outlook Anywhere and remote clients an host record for Autodiscover server should be created on an external DNS

Without AutoDiscover access client can access mailbox but certain functions like F/B, OOF, OAB and UM will not be accessible

If AutoDiscover is located via DNS, Outlook will try pre-determined order of URLs to connect to AutoDiscover Server. For example:

https://domain.com/autodiscover/autodiscover.xml

https://autodiscover.domain.com/autodiscover/autocover.xml


https://domain.com/autodiscover/autodiscover.xml

https://autodiscover.domain.com/autodiscover/autocover.xml

New DNS SRV Record for Locating Autodiscover Service

Predefined URL method requires valid SSL certificate for URLs being used

Generally different DNS names are used for Outlook Anywhere and OWA

HTTP Redirect needs additional Web Site in IIS and two Public IP Addresses

New Software for Outlook performs and additional check forDNS SRV record for Autodiscover Service

This feature is available as part of following update rollup forOutlook : (http://support.microsoft.com/kb/939184/) Description of the update rollup for Outlook 2007: June 27, 2007


Troubleshoot Autodiscover

Client side

Test E-mail Autoconfiguration

Outlook Logging

Server side

Test-OutlookWebServices

Event Logs

Exchange Management Shell


Troubleshoot Autodiscover – Client.

Results tab: Web service URLs

Log tab: URLs used and error codes

Popular error codes

80072EE7 – ERROR_INTERNET_NAME_NOT_RESOLVED

80072EFD – ERROR_INTERNET_CANNOT_CONNECT

80072F17 – ERROR_INTERNET_SEC_CERT_ERRORS

Outlook logging

OLKDISC.log in temp directory

OLKAS directory


Troubleshoot Autodiscover – Test-OutlookWebService

Examples of failures:


When using

self signed

certificates

A DNS issue

or a general

server

performance

problem

The

AutoDiscover

XML file is

missing.

Troubleshoot Autodiscover – Server Event Logs

Event logging

Three MSExchange AutoDiscover event categories\Core

\Provider

\Web

Set-EventLogLevel "MSExchange AutoDiscover\Core" –Level:Expert


Agenda

Introduction

Overview


Troubleshooting:

Autodiscover




Outlook Web Access

Exchange ActiveSync



Calendaring functionality for free/busy, meeting suggestions and Out-of-Office (OOF) depends on Availability Web Service

Availability Service is used only by Exchange 2007 Mailboxes

For Exchange 2007 Mailboxes, Calendar data will be read from user‘s mailbox directly

OL 2007/Exchange 2007 users access Exchange 2003 mailbox free/busy data by using Availability Service to look up free/busy Public Folders on Exchange 2003 Servers


Availability Service – Cross Forest Access

To access cross-forest free/busy data, make sure free/busy information is replicated between forests

To see free/busy data of Exchange 2003 mailboxes in the other forest, configure the availability service by running the following command on any server in the Exchange 2007 forest:

Add-AvailabilityAddressSpace -ForestName:<forest name e.g. msft.com> -AccessMethod:PublicFolder


Troubleshooting Availability Service


In Outlook 2007, Ctrl-Right-click on the Outlook system tray icon.

Enter your email address and password

Troubleshooting Availability Service, cont.

There are two ways to check if Availability service is functioning correctly:

Event Log4001 The Availability Service could not discover an Availability Service in the remote forest

4003 Public Folder Request Failed

4004 Unable to find a public folder server for the organizational unit

4005 Could not find information in Active Directory to allow cross-forest requests

4011 Cross-forestRequestFailed

Test-OutlookWebServices Cmdlet

Test-OutlookWebServices -id:[email protected] -TargetAddress: [email protected]


Troubleshooting Availability Service, cont.

OWA vs Outlook 2007OWA runs against the Availability Service API(s) on the CAS server

OUTLOOK 2007 runs against the Availability Service Web Service and relies on the Autodiscover service to find the Availability URL

Mostly, free/busy problems in Outlook 2007 might be related with configuration of Autodiscover rather than Availability Service

Following commands can be used to get more information:Get-webservicesvirtualdirectory

Get-WebServicesVirtualDirectory -Identity EWS(default web site)

Get-WebServicesVirtualDirectory -Identity CAS01


Test-OutlookWebServices

Test-OutlookWebService is a diagnostic task to verify AutoDiscover , Availability Service, RPC/HTTP and OAB distribution configuration for connectivity onlyTest-OutlookWebServices -Identity <Alias, Domain\User or SMTP address> -ClientAccessServer <FQDN or NetBIOS name> -TargetAddress<Alias, Domain\User or SMTP address>

Scope can be set for: For an Individual User: -Identity <Alias, Domain\User or SMTP address>

For a specific CAS Server: -ClientAccessServer <FQDN or NetBIOS name>

Free/Busy queries: -TargetAddress < Alias, Domain\User or SMTP address>

Returns information about SSL Certificate problems

Determines the validity of the returned service URLs

The request is made for one day of free busy data and the data is not returned in the task output.


Test-OutlookWebServices – Basic Functionality

Step 1: Get a user context

Step 2: Determine the Autodiscover URL

Step 3: Submit an Autodiscover request

Step 4: Validate that services exist

Step 5: Return results of all tests to the console in the form of events.


Troubleshooting Free/Busy using Outlook 2007 Logging

Outlook 2007 can be used to troubleshoot problems with the Autodiscover service

The Availability service log files are located in the \Documents and Settings\ <username> \Local Settings\Temp folder

Three Log types:OOF (Out of Office)

MS (Meetings Suggestions)

FB (Free/Busy)

Example:

20070305-110303994-fb.log


Free/Busy Log Files

Generated each time a user is added to the meeting request from Scheduling Assistant tab or for each request sent from the client

Includes GetUserAvailabilityRequest XML message

There are only three blocks of interest that contain the detail information needed for diagnostics:• MessageText – Contains information about the failure.

• ExceptionCode– Contains the exception that caused the failure.

• ResponseCode– Contains the web response code for the failure.


Troubleshooting Free/Busy specific failures

Mailbox Logon Failure : Check the status of the target user's mailbox to see if it is available

<MessageText>Mailbox logon failed., inner exception: Cannot open mailbox /o=Fourthcoffee/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=mod4user11.</MessageText><ResponseCode>ErrorMailboxLogonFailed</ResponseCode>

Permissions Error : Engage the target mailbox owner to confirm calendar permissions

<MessageText>Caller does not have access to free busy data. </MessageText>

<ResponseCode>ErrorNoFreeBusyAccess</ResponseCode>


Troubleshooting Free/Busy specific failures – cont.

Proxy Failures : The configuration need to enable cross-forest sharing of free/busy information is incomplete or miss-configured

<MessageText>The proxy request failed because the remote server returned an error…………

<ResponseCode>ErrorProxyRequestProcessingFailed</ResponseCode>

Legacy Free/Busy Failures : Public folder store on Exchange 2003 is not mounted or inaccessible

<MessageText>The remote server returned an error: (503) Server Unavailable..

<ResponseCode>ErrorPublicFolderRequestProcessingFailed</ResponseCode>


Availability Service Error Codes

Description of Exchange 2007 Availability Service Error CodesRequestStreamTooBig = 5000

IdentityArrayEmpty = 5001

IdentityArrayTooBig = 5002

TimeIntervalTooBig = 5003

InvalidMergedFreeBusyInterval = 5004

ResultSetTooBig = 5006

InvalidClientSecurityContext = 5007

MailboxLogonFailed = 5008

MailRecipientNotFound = 5009

InvalidTimeInterval = 5010

PublicFolderServerNotFound = 5011

InvalidAccessLevel = 5012

And more…


Agenda

Introduction

Overview


Troubleshooting:

Autodiscover




Outlook Web Access

Exchange ActiveSync


Offline Address Book (OAB)

Used for offline GAL view and cached mode GAL lookups

New OAB uses HTTPs & BITS (Background Intelligent Transfer Service)

Legacy support available by using public folders distibution

Relies on:

OABGen

Exchange File Distribution

OAB Virtual Directory

Autodiscover

BITS Client does not support self-signed certificates, so by default OAB Distribution Points use HTTP

SSL can be enabled with a fully trusted certificate in IIS


Offline Address Book - Process


Troubleshooting Offline Address Book

Legacy clients

• Need to publish OAB to public folders

• Need OAB Public folder replication

Outlook 2007

• Make sure Autodiscover works and the URL’s are correct

• Check the OAB distribution on nearest CAS server

• Check IE proxy settings (KB939765)

Non client specific issues

• GAL deleted

• No permissions


Agenda

Introduction

Overview


Troubleshooting:

Autodiscover




Outlook Web Access

Exchange ActiveSync


SSL handshake


SYNC (TCP_Port = 443)

SYNC + ACK

ACK

Outlook clicks on URL to

access Secure Webmail Server

(https://mail.msft.com)

The browser establishes a

TCP connection on the HTTPS TCP Port 443

CLIENT_HELLO

SERVER_HELLO

CERTIFICATE

SERVER_DONE

SSL Handshake on the new TCP connection

To continue with the authentication process, clientshould verify the server‟s certificate

1 Is today's date within the validity period?2 Is the issuing Certificate Authority (CA) a trusted CA?3 Does the issuing CA's public key validate the issuer's digital signature?

The server is authenticated. 5

Outlook Web Access Client Access Server

Does the domain name in the server's certificate match the domain name of the server itself?

4

Certificates and Subject Alternate Name

By default during CAS installation a new self signed certificate will be generated and assigned to the Default Web Site for encrypted HTTP communication

The authenticity of self signed certificate cannot be verified during Internet access

For Internet access to CAS Servers, it is recommended to use a 3rd party CA for authenticity to work

However the certificate which was assigned to company name may not match the CAS Web Address

Subject Alternate Names can be used as an alternative web address for an existing certificate to override that problem


Certificate Request


https://cas01/owa

https://cas01.domain.com/owa

https://mail.domain.com/owa

https://autodiscover.domain.com/

Self-signed Certificates & Outlook Anywhere


Client Access Server

Outlook Provider

CertPrincipalname =

Null

By default, the

CertPrincipalName

parameter for

OutlookProvider is

not configured

Outlook Anywhere

OL uses the

ExternalHostname

parameter for

OutlookAnywhere to

populate the server name

listed after “MSSTD:”

By default, the

ExternalHostname parameter

will not match the default

“Issued To” value on the self-

signed certificate

If you use self-signed certificates, OL will not successfully connect using HTTPS by using default settings pushed by Autodiscover

Troubleshooting Self-signed Certificates

There are two methods to solve self-signed certificate problems with Outlook Anywhere:

Get a new certificate where the “Issued To” property matches the Certificate Principal Name

-OR-

Change the CertPrincipalName Value on OutlookProvider:

set-outlookprovider -identity EXPR -server 'owatest.mail.msft' –CertPrincipalName 'msstd:owatest.mail.msft'

This allows OL2007 to get complete the Autodiscover phase of Outlook Anywhere profile creation

Domain-joined clients do not display Invalid CA certificate warnings

Note: Self signed certs would generate warnings for end user and we recommend our customers to buy the required certificates before deploying CAS for the end-users.


Certificate Issues for Combined CAS&HUB Role

Events 1037 and 2019 if third party certificateIs not enabled for SMTP Service on CAS/Hub Server –and-

Includes Netbios name in the Certificate Request

To fix this:If the domain name parameter includes Netbios or Server FQDN in thecertificate request, the certificate should be enabled for SMTP service.

Run Enable-ExchangeCertificate command to enable for SMTP

Alternatively, do not use Netbios or Server FQDN in the certificate request, use only Public FQDN


Common Certificate Issues

Using the self signed certificate

• The certificate common name is the server NetBIOS name

• There is no automatic way to make the self signed certificate trusted

Using the old exchange 2003 certificate

• Does not have Autodiscover URL

Using new certificate without considering Autodiscover

ISA server is not publishing Autodiscover URL


Troubleshooting CAS Security & Certificate Issues

Tools Components

Get-ExchangeCertificate Command used to view certificates from the local certificate store

EXTRA Tracing Enable the following components/tags:•Common\Certificate Validation•Networking Layer\Certificate •Transport\Certificate

Protocol Logging Tool to troubleshoot transport security problems

IIS Log Files Access Used to troubleshoot authentication errors while accessing web service or any other web page

Certlib Unofficial utility to dump themsExchServerInternalTLSCert value in AD into a readable format


Outlook Anywhere– Authentication methods after SP1

Exchange 2007 RTM

Mandatory Parameter: ExternalAuthenticationMethod

Used to update OL2007 clients using Autodiscover service

Basic and NTLM Authentication methods were always reenabled on /rpc virtual directory regardless of this parameter

Exchange 2007 SP1

Ability to choose the authentication methods

New parameters:

ClientAuthenticationMethod

IISAuthenticationMethods

DefaultAuthenticationMethod

set-OutlookAnywhere -IISAuthenticationMethods <Basic or NTLM>


Exercise 1: Introduction to SSL Certificates

Exercise 2: Subject Alternative Names (SAN)


LAB 1 Troubleshooting Certificates

Exercise 1: Understanding the AutoDiscover Service

Exercise 2: Configuring the AutoDiscover Service for use by external Outlook clients


LAB 2 Troubleshooting Autodiscover

Agenda

Introduction

Overview


Troubleshooting:

Autodiscover




Outlook Web Access

Exchange ActiveSync


Outlook Web Access – Overview


Exchange 2003 Mailbox

OWA 2003 rendering

Exchange 2007Mailbox

SharePoint &File Shares

Active Directory

HTTP toExchange 2003

OWA &WebDAV

HTTP toExchange 2007

OWA

Client Access Server

OWA

Auth

ISAPI

IIS

SSL

/owa

/exchange

/exchweb/public

OWA

2007

rendering

Spell-

Checkers

Active

Directory

Driver

Doc. HTML

transcoder

CAS

business

logic

Outlook

Web Access

Proxy

Authentication Methods

Authentication Method

Security LevelHow Passwords

Are SentClient Requirements

Basic authentication

Low (unless Secure Sockets Layer [SSL] is enabled)

Base 64-encoded clear textAll browsers support Basic authentication

Digest authentication

Medium HashedMicrosoft Internet Explorer® 5 or laterversions

Integrated Windows authentication

High

•Hashed when IntegratedWindows authentication isused•Kerberos ticket when Kerberos is used•Integrated Windows authentication includes the Kerberos and NTLM authentication methods

•Internet Explorer 2.0 or later versions forIntegrated Windows authentication;•Microsoft Windows 2000 Server or later versions with Internet Explorer 5 or later versions for Kerberos

Forms-basedauthentication

High Uses cookies to help secure a user's name and password

Internet Explorer


Troubleshooting OWA – FBA Login


OWA Client CAS Server

Anonymous GET /owa

Owaauth

.dll

Intercept

Redirect to owa/auth/logon.asp

Anonymous GET /owa/auth/logon.asp

Return FBA logon page

POST including username + password

Redirect to /owa + set Auth Cookie Auth Cookie

Authenticated Request GET /owaAuth Cookie

Owaauth

.dll

Set Cookie

Owaauth

.dll

Authenticated

Troubleshooting OWA: Tools

Exchange Management Shell (EMS)Get-OwaVirtualDirectory (Set/Remove/New)

Test-OwaConnectivity Cmdlet

Test-OwaConnectivity –ClientAccessServer:ServerName

Test-OwaConnectivity -URL:https://mail.domain.com/owa -MailboxCredential:(get-credential DomainName\AccountName)

Get-CASMailbox (Set)

Get-casmailbox UserName | fl owa*

Exchange Managment Console (EMC)


Note: Looking for a specific command? Use Get-Help with correct wildcards

Example: get-help *OWA*

Troubleshooting OWA: Tools – cont.

Internet Information managerMSExchangeOWAAppPool must be started using Local System identity

Web Services Extensions must be enabled for: ASP.NET (), Microsoft Exchange Client Access Server (owaauth.dll)

CAS Only: Microsoft Exchange Server (exprox.dll)

CAS + MBX: Microsoft Exchange Server (davex.dll)

Check Mapped application for Legacy Virtual Directories (/Exchange, /Public and /EXCHWEB)

CAS Only: exprox.dll

CAS + MBX: davex.dll

Check Authentication settings (change using EMC or EMS)

Anonymous for owa/auth folder


Troubleshooting OWA: Configuration

Check Configuration filesForms Registry file: Registry.xml

Web.Config

Registry KeysDisable LDAP Encryption (Troubleshooting ONLY)

Key: HKLM\SYSTEM\CurrentControlSet\Services\MSExchangeADAccess

DWORD: Disable LDAP Encryption

Value: 1 = LDAP Encryption disabled


Troubleshooting OWA: Microsoft Fiddler HTTP debugger

Fiddler is a HTTP Debugging Proxy which logs all HTTP trafficInternet Explorer Plug-In

View HTTP(S) traffic (real-time)

HTTP(S) Statistics

Capturing web traffic logs

Session Inspector

RAW view of HTTP traffic

And many more

Download available at http://www.fiddlertool.com/fiddler/

IMPORTANT: Fiddler is a Client Side debugging Tool and should never be installed onto a production Exchange server. Serious problems have been reported with Active Sync when Fiddler is installed on an Exchange 200x server!


http://www.fiddlertool.com/fiddler/

Understanding Proxying and Redirection


Internet

OWA EAS

CAS in

User’s mailboxAD Site

Find a CAS

In MailboxAD Site

Best CAS

has ExternalURLSet?

REDIRECT Client

using ExternalURL

PROXY request to

CAS using InternalURL

Execute request

on this CAS

OWA/EAS

not available

Note: If the mailbox is on E2K3, the

CAS will proxy directly to the back-end. Integrated Windows authentication for /Exchange and /Microsoft-Server-

ActiveSync virtual directories must be enabled

Note: CAS to CAS proxying is not

supported between virtual directories that use Basic authentication, the virtual directories must use Integrated

Windows authentication.

Yes

No

Yes

YesNo

No

Note: Redirection is only

supported for OWA

Understanding Proxying and Redirection - cont.


CAS->MBX comm. Between

AD sites

RedirectionCAS->CAS

CAS-> proxying between AD sites

Comments/Consequences

OWA No Yes Yes

Must have a CAS server in each Exchange AD site to use OWA/EAS/Web Services

EAS NoUnnecessary: Autodiscover

Yes

Web Services used by 3rd party LOB applications

No No Yes

Availability Service used by Outlook 2007

NoUnnecessary: Autodiscover

Yes

Outlook Anywhere Yes, RPCUnnecessary: Autodiscover

Not applicable

WebDAV and OWA 2000/2003

Yes, HTTP No Not applicableProxying to legacy E2003 server

IMAP4/POP3 No No No

IMAP/POP clients must access a CAS in the mailbox AD Site directly

CAS Proxy Scenarios

Between Exchange 2007 Client Access ServersInternet facing CAS proxy requests to other CAS with no Internet presence

Known as CAS-CAS proxying

Between an Exchange 2007 Client Access Server and an Exchange Server 2003 Back-end server when:

OWA Clients connect to /Exchange virtual directory

EAS Clients connect to /Microsoft-Server-ActiveSync virtual directory


Issues with Coexistence and DAVEX

When CAS and Mailbox roles are combined together, OWA clients are prompted for credentials two times or redirected to a different server to access an Exchange 2003 mailbox

For script mapping, /Exchange virtual directory will use Davex.dll instead of Exprox.dll

Davex.dll cannot act as a proxy for mailbox requests

Instead it redirects the requests to the Exchange 2003 Mailbox Server based on the internal (intranet) name of the server

External users will get DNS errors if internal name is not exposed to Internet


Troubleshooting Proxying - Redirection

Verify Configuration using Exchange Management ShellInternalURL and ExternalURL values

RedirectToOptimalOWAServer=$True

Certificates SelfSigned/Public/Private vs

Set Diagnostic Logging for MSExchangeOWA\Proxy

set-EventLogLevel -Identity "ServerName\MSExchange OWA\Proxy" -Level "Expert"


Proxying Performance and Scalability & Debugging

ASP.NET Proxying performance and scalability (KB821268)

Debug Registry Keys:

Allow Proxying without SSLRegistry Key: AllowProxyingWithoutSSL = 1

Allow Proxying only using Trusted Certificate

Registry Key: RequireTrustedCertForProxying = 1


Exercise 1: CAS to CAS proxying

Exercise 2: Configuring OWA Redirection


LAB 3 Troubleshooting Proxying

Web Ready Document Registry Keys

Set in the registry of the Client Access server under:

HKLM\SYSTEM\CurrentControlSet\Services\MSExchange OWA


Value Name Value

Type

Value Default Value

RecycleByConversions DWORD # > 0 1000

ExcelRowsPerPage DWORD # > 0 200

MaxDocumentInputSize DWORD 1000000 > # > 0 5000 (in KB)

MaxDocumentOutputSize DWORD 1000000 > # > 0 5000 (in KB)

TempFolderLocation String Valid Path %SYSTEMROOT%\Temp

CacheDiskQuota DWORD # > 0 1000 (in MB)

ConversionTimeout DWORD # > 0 20 (seconds)

Troubleshooting WebReady Document Viewing


Exchange Management Shell (EMS)Get-OwaVirtualDirectory (Set/Remove/New)

Get-CASMailbox (Set)

Exchange Managment Console (EMC)

ADSIEditCN=owa (Default Web Site),CN=HTTP,CN=Protocols, CN=<ServerName>, CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=<Exchange Organization>,CN=Microsoft Exchange,CN=Services, CN=Configuration,DC=<Domain>,DC=<Domain>

Exercise 1: Examining the OWA login process (Forms based Authentication)

Exercise 2: WebReady Document viewing


LAB 4 Troubleshooting OWA

Agenda

Introduction

Overview


Troubleshooting:

Autodiscover




Outlook Web Access

Exchange ActiveSync


Exchange ActiveSync Overview

Synchronization protocol optimized to work with high-latency and low-bandwidth networks

Based on HTTP and XML

Enables mobile device users to access their e-mail, calendar, contacts and tasks

Enhanced in Exchange Server 2007:


Support for HTML messages

Support for follow-up flags

Support for fast message retrieval

Meeting attendee

information

Enhanced Exchange Search

Windows SPS and LinkAccess

document accessPIN reset

Enhanced device security through

password policies

AutoDiscover for over the air provisioning

Support for OOF configuration

Support for tasks synchronization

Troubleshooting Exchange ActiveSync

Coexistence

Exchange 2003 Back end authentication (KB937031)

Read IIS logs

Use logparser

Certificates

Use device emulator

EXTRA Logging


All Troubleshooting Tools

IIS Logs

LDAP tools (LDP, ldifde, adsiedit, etc …)

Event Logging

Performance Monitoring

Exchange Troubleshooting Assistant (Supervised Trace Logs)

Browser

Outlook logging


Additional resources

Support WebCast: Introduction to AutoDiscover in Microsoft Exchange Server 2007http://support.microsoft.com/kb/935438

White Paper: Exchange 2007 Autodiscover Servicehttp://technet.microsoft.com/en-us/library/59adba4e-44e1-4aa2-b09d-06988cbeab2d.aspx

Autodiscover and Exchange 2007http://technet.microsoft.com/en-us/library/7c44814d-bb46-4fb8-9b6b-a082be35afdc.aspx

Managing the Autodiscover Servicehttp://technet.microsoft.com/en-us/library/aa995956.aspx

Exchange 2007 Autodiscover and certificates http://msexchangeteam.com/archive/2007/04/30/438249.aspx

More on Exchange 2007 and certificates - with real world scenariohttp://msexchangeteam.com/archive/2007/07/02/445698.aspx

Exchange 2007 Offline Address Book Web Distributionhttp://msexchangeteam.com/archive/2006/11/15/431502.aspx


http://support.microsoft.com/kb/935438

http://technet.microsoft.com/en-us/library/59adba4e-44e1-4aa2-b09d-06988cbeab2d.aspx



http://technet.microsoft.com/en-us/library�/59adba4e-44e1-4aa2-b09d-06988cbeab2d.aspx










http://technet.microsoft.com/en-us/library/7c44814d-bb46-4fb8-9b6b-a082be35afdc.aspx



http://technet.microsoft.com/en-us/library�/7c44814d-bb46-4fb8-9b6b-a082be35afdc.aspx










http://technet.microsoft.com/en-us/library/aa995956.aspx



http://msexchangeteam.com/archive/2007/04/30/438249.aspx



Troubleshooting Hub Transport Server (HT)

Agenda

Troubleshooting Mail flow Local Delivery & Mail Submission: Architecture

Local Delivery & Mail Submission: Troubleshooting

Remote Delivery: Architecture

Troubleshooting Mail Flow

Troubleshooting Mail QueuesMail Queues: Architecture

Mail Queues: Queues

Mail Queues: Troubleshooting

Troubleshooting Transport Certificates Certificates and Transport: Overview

Certificate Selection

Troubleshooting STARTTLS

Internal Transport

Troubleshooting Direct Trust

SP1 Changes

122

Local Delivery & Mail Submission: Architecture

Local DeliveryProcess by which messages are delivered from a Hub Transport server to mailbox on a local Mailbox server

Mail Submission

Process of picking up a message from a user’s mailbox and getting it into the Submission queue on a local Hub Transport server

Multiple components involvedExchange System Objects (XSO)

Microsoft Exchange Mail Submission Service

Store Driver

123


Store DriverRuns on Hub Transport Server

Submits and retrieves mail to and from a local Mailbox server

Performs MAPI to MIME and MIME to MAPI conversion

Two primary components

Mail Submission RPC Server

Used to accept New Mail Notifications from the Mail Submission Service on Mailbox Server

XSO

Used to submit and retrieve mail to and from a Mailbox Server using MAPI.Net over RPC

Performs the MAPI to MIME and MIME to MAPI conversion

124


Mail Submission ServiceRuns on Mailbox server

Notifies Hub Transport server of new messages for delivery

Two Primary components

Assistant Infrastructure

Processes and manages events that occur within the Information Store

Receives “Event” notifications about new messages for delivery

Triggers Mail Submission Service to generate new message notifications for Store Driver

Mail Submission RPC Client

Connects to Mail Submission RPC Server on Store Driver

Submits New Mail Notifications generated by Mail Submission Service

125

Local Delivery & Mail Submission: Local Delivery

Local delivery processOne Queue per Mailbox Server

Does not require Mailbox Store to run on local server

Does not use hidden SMTP Mailbox for conversion

No Store Driver on Mailbox Server

126

Mailbox Role

Hub Transport Role

EdgeTransport.exe

Store.exe

Outlook

Client

MAPI

JET

STORE STORESTORE

StoreDriver

XSOMailSubmission

RPC Server

RPC

Submission

Queue

Local

Delivery

Queue

CAT

MailSubmissionSvc.exe

Assistant

Infrastructure

(AI)

MailSubmission

RPC Client

MAPI.

Net

RPC

Dumpster

Message submission is completed Convert to

MAPI

Mailbox is in

local AD Site

Local Delivery & Mail Submission: Mail Submission

Mail Submission process

127

Mailbox Role

Hub Transport Role

EdgeTransport.exe

Store.exe

Outlook

Client

MAPI

JET

STORE STORESTORE

StoreDriver

XSOMailSubmission

RPC Server

RPC

Submission

Queue

Local

Delivery

Queue

CAT

MailSubmissionSvc.exe

Assistant

Infrastructure

(AI)

MailSubmission

RPC Client

MAPI.

Net

RPC

Dumpster

Locate Hub

Server

New Message Notification

Mailbox Server

DN, Sender„s MB

GUID, Message„s

EntryID, etc

Convert to MIME

Move message from Outbox to

Sent Items

Agenda

Troubleshooting Mail flow






Mail Queues: Queues





Internal Transport


SP1 Changes

128


Run the Exchange Mail Flow Troubleshooter

Check Queues on the Hub server

Run Test-ServiceHealth on Mailbox and Hub server

Check event logs on Mailbox and Hub Server

Test basic RPC connectivity between Hub and Mailbox server

Enable connectivity logging

Verify AD site configuration

Verify DNS configuration

Check permissions for the Exchange Servers group on the Mailbox server object

129


Run the Exchange Mail Flow TroubleshooterAvailable in the EMC Toolbox

Analysis is symptom specific, so choice of correct symptom is critical to success

130

Inbound

NDR

Outbound

Queue

Submission

EdgeSync


Check Queues on the Hub serverUse Queue Viewer or the Get-Queue cmdlet to check the Status and LastError fields of the Mailbox Delivery queue

If Status is Retry then LastError should be an error message that can be used to help identify the problem

Run Test-ServiceHealth cmdlet on Mailbox and Hub serverEnsures all required services configured to start automatically have started

Returns an error for any service that is required by a configured role and is set to start automatically but is not currently running

131


Check event logs on Mailbox and Hub ServerCheck both Mailbox and Hub server for errors or warnings

If necessary increase diagnostics logging using the registry or Set-EventLogLevel

Mailbox server: MSExchangeMailSubmission\General

Hub Transport server: MSExchange Store Driver\General

The possible logging levels that you can set are:

0 (Lowest), 1 (Low), 3 (Medium), 5 (High), 7 (Expert)

Always return the logging level to the default setting after completing your troubleshooting activities

132


Test basic RPC connectivity between Hub and Mailbox serverUsing RPCPing

Two components

Rpings.exe (server-side RPC ping utility)

Rpingc.exe (client-side RPC ping utility)

Verify RPC connectivity in both directions

Using Test-MAPIConnectivity cmdlet

Logs onto the system mailbox or a specified mailbox using the -Identity parameter

Verifies that the MAPI server, Exchange store, and Directory Service Access (DSAccess) are working

133


Test-MailflowTests mail submission, transport and delivery

Tests services by verifying that each mailbox server can successfully send itself a message.

Remote functionality to test between remote Mailbox Servers

134


Enable connectivity loggingWhen the Hub server is having problems sending to the Mailbox server

Not enabled by default

get-transportserver|set-transportserver -ConnectivityLogEnabled:$TRUE

Provides summary information on outbound connections via SMTP or StoreDriver

Much less verbose than Protocol logs

Protocol logs more useful for SMTP conversation

Useful when wanting to see connectivity issues with the sending StoreDriver as protocol logs do not capture this information

Multiple events/connection, each using same connection ID for correlation

135


Verify AD site configurationEnsure AD sites and subnet configuration is correct

Run Nltest.exe /dsgetsite on Hub and Mailbox servers to verify they are in the same AD site

Beware: Changing IP address of Hub server may cause it to fall under another AD site and therefore leave Mailbox server in an AD site with no other Hub servers

136


Verify DNS configurationCorrect AD site determination is heavily dependant on correct DNS configuration

Verify Hub and Mailbox servers are configured with the correct internal DNS server

Ensure correct AD information such as domain controller SRV and A records as well as correct AD site information is stored in DNS

Use DCDIAG /TEST:DNS on domain controllers (use /X and /XSL to log to XML for easier reading)

Ensure correct A records are stored in DNS for Hub and Mailbox servers

Nslookup can be used to help verify the above

137


Check permissions for the Exchange Servers group on the Mailbox server object

Using ADSIEdit.msc verify that the Exchange Servers group has an explicit allow set on the Mailbox server object for

Store Constrained Delegation

Store Read and Write Access

Store Read only Access

Store Transport Access

Ensure there are no explicit denies set as this will override the allow

Hub server requires these permissions in order to retrieve messages from and submit messages to mailboxes

138

Agenda







Mail Queues: Queues





Internal Transport


SP1 Changes

139


Remote Delivery refers to messages delivered using SMTP toAnother Hub server in the organization

An Edge server

Another remote SMTP mail system

Can also be to a foreign mail system using the Drop directory

140


DNSExchange 2007 uses an enhanced DNS client to resolve the next hop selection to a list of target server names

Standard DNS client used to resolve list of server names to IP addresses

Enhanced DNS also provides load-balancing for Hub servers by using round robin

SMTPUsed for communication when messages are relayed between SMTP servers

141


Routing TablesHolds information that the routing component uses to make routing decisions

Composed of a map of topology components and their relationship to one another

Linked connectors map

Server map

Legacy server map

MDB map

Active Directory site map

Routing groups map

Send connectors map

142

Remote Delivery: Routing Architecture

Routing TablesBuilt every time that a transport server is started

Recalculated when configuration changes are received

Configuration changes can be detected in the following ways

Active Directory change notifications

Configuration reloading caused by service control commands

Periodic reload to track changes that are not supported by Active Directory notifications

Information in the routing tables is logged to routing logs

C:\Program Files\Microsoft\Exchange Server\TransportRoles\Logs\_ Routing

New log generated every time routing tables are calculated

If the Hub server is unable to contact AD routing will use currently cached routing data

144


Viewing the Routing Table LogRouting Log Viewer used to view the Routing Table Log

New tool with Microsoft Exchange Server 2007 SP1

145

Can be used to

Find the Lowest Cost Path to a Site

Find the Preferred Connector for a Given Address

Can open a second routing log and determine changes that have occurred within the routing topology between two time periods

Remote Delivery: Demo

In this demo your instructor will show youHow to use the Routing Log Viewer tool to

Find the lowest cost path to a site

Find the preferred connector for a given address space

Compare two routing logs to identify changes

147


Determining Site MembershipNetLogon service determines site membership for the computer upon startup

Uses DNS queries to compare the local IP address to defined subnets

Exchange AD Topology service retrieves the site membership value from the NetLogon service

msExchServerSite attribute

Value of this attribute is the DN of the AD site of an Exchange server

Reduces overhead associated with DNS queries

Also used to associate a non-domain computer, such as a subscribed Edge server to an AD site

Populated and kept up to date by the Exchange AD Topology service

148


Detecting Site Membership ChangesMay occur due to IP address change or AD subnet association change

Exchange 2007 must update its configuration data so that the change is considered when making routing decisions

The NetLogon service polls frequently for changes in AD site membership

The Exchange AD Topology service queries NetLogon regularly to determine the AD site membership of the local Exchange server and updates the MsExchServerSite attribute if necessary

The Exchange servers in the organization update the routing tables with the new AD site membership attribute value

150


Controlling IP Site Link CostsBy default, Exchange Server 2007 uses the cost that is assigned to the AD IP site links

You can use the Set-AdSiteLink cmdlet to assign an Exchange-specific cost to an IP site link connector

Exchange-specific cost is a separate attribute and overrides the AD assigned cost for the purpose of determining the Exchange routing path

Allows the Exchange administrator to override existing links however it does not allow the ability to create links where none exist

Useful when the AD IP site link costs do not result in an optimal Exchange message routing topology

Using it for permanently overriding costs simply adds complexity

Better to work with AD and network administrators

152


Message size limit for IP site linksNew to Exchange 2007 SP1

By default Exchange does not impose a maximum message size limit on messages that are relayed between Hub servers in different AD sites

Useful when low-bandwidth connections to remote sites exist

Use the Set-AdSiteLink cmdlet to configure

Set-AdSiteLink -Identity <IP Site link name> -MaxMessageSize 10MB

153

Remote Delivery: Troubleshooting

Back PressureBack pressure is a system resource monitoring feature of the Exchange Transport service

If utilization of a system resource exceeds the specified limit the server stops accepting new connections and messages

Prevents the system resources from being completely overwhelmed and enables the server to deliver the existing messages

When utilization returns to a normal level the server accepts new connections and messages

For each monitored system resource the following three levels of resource utilization are applied:

Normal

Medium

High

154

Agenda







Mail Queues: Queues





Internal Transport


SP1 Changes

155


Back Pressure (Cont...)System resources monitored

Free space on the drive that stores the message queue database

Free space on the drive that stores the message queue database transaction logs

The number of uncommitted message queue database transactions that exist in memory

The memory that is used by the EdgeTransport.exe process

The memory that is used by all processes

156


Understanding Non-Delivery ReportsNDRs redesigned in 2007 to make them easier to read and understand

Separated into user information and diagnostic information for admins

157


Understanding Non-Delivery ReportsThe following fields are present in most NDRs

Generating server

Rejected recipient

Remote server

Enhanced status code

SMTP response

Original message headers

Enhanced status codes

Success is indicated by a 2.x.x status code

Persistent Transient Failure is indicated by a 4.x.x status code

Permanent Failure is indicated by a 5.x.x status code

158


Message Tracking LogsDetailed log of all message activity as messages are transferred

161

Enabled by default

EventID describes tracking event action

Source describes component involved

SMTP, STOREDRIVER, ROUTING, AGENT

Use Set-TransportServer or Set-MailboxServer cmdlets to configure


Message tracking log event typesBADMAIL

DELIVER

DEFER

DSN

EXPAND

FAIL

POISONMESSAGE

162

RECEIVE

REDIRECT

RESOLVE

SEND

SUBMIT

TRANSFER


Filtering message tracking logsSearch depends on the Microsoft Exchange Transport Log Search service

Get-MessageTrackingLog cmdlet or Message Tracking tool in EMC to filter and display results

Get-MessageTrackingLog -sender [email protected] -start “07/01/2007 09:00 AM” -end “07/01/2007 09:30 AM”

In SP1 use GetMessageTrackingLogE2EwithTime.ps1 to search for specific entries in all message tracking logs on all Hub and Mailbox servers

163


Protocol LogsSMTP protocol conversation without data

Useful for diagnosing SMTP mail flow problems

Use Set-ReceiveConnector or Set-SendConnector cmdlets or the EMC to enable

164


Troubleshooting mail flow issues between Hub serversUse EXMFA with symptom matching the case

Determine where message delivery failed or where a non-delivery report (NDR) is being generated by

Using the Mail Flow Troubleshooter tool

Using the Queue Viewer on a Hub server to determine where message delivery failed

Checking the NDR to verify which server and component are generating the NDR

Use message tracking in order to determine the path messages are taking and at what point delivery is failing

Check the delivery status notification (DSN) error codes contained in the NDR and search the Microsoft support site for the error code

Run the Test-ServiceHealth cmdlet on the Hub servers where message delivery failed or on the transport server that generated the NDR

165


Troubleshooting mail flow issues between Hub servers (Cont...)Check the application event log on the Hub Transport servers that are involved in the delivery of the message

Increase diagnostic logging levels on the Exchange processes that are generating errors if necessary (SMTP Send and Receive, RemoteDelivery, Routing etc)

Verify that back pressure is not occurring

Event ID 15001 and 15002 for RTM and 15004 and 15005 for SP1

Check Physical connectivity

From the server where message delivery is failing ping the next hop servers by IP and FQDN and ensure a reply is received

At the same time ensure that the correct IP is resolve

Run netstat –anb on the receiving Hub server and verify that MSExchangeTransport.exe is listening on port 25

166


Troubleshooting mail flow issues between Hub servers (Cont...)From the server where message delivery is failing verify you can connect to SMTP port 25 of the next hop servers by using telnet

Verify the necessary connectors are enabled and configured appropriately on the Hub servers involved

Default receive connector configuration, Remote IP range, message size restrictions, authentication methods and types

Verify AD site configuration

Ensure AD sites and subnet configuration is correct

Run Nltest.exe /dsgetsite on next hop Hub servers to verify they fall in the correct AD site

Check msExchServerSite attribute using ADSIEdit.msc for next hop Hub server objects to ensure they are stamped with the correct AD site

167


Troubleshooting mail flow issues between Hub servers (Cont...)Verify DNS configuration

Verify Hub servers are configured with the correct internal DNS server

Ensure correct information such as A records for next hop servers are stored in DNS

Ensure correct AD information such as domain controller SRV and A records as well as correct AD site information is stored in DNS

Use DCDiag /TEST:DNS on domain controllers

Ensure the correct A records are stored in DNS for the Hub servers

Nslookup can be used to verify the above

Enable protocol logging for the Send connector on the sending Hub server and Receive connector on the receiving Hub server

168


Troubleshooting mail flow issues between Hub servers (Cont...)Look for possible certificate issues

If custom certificates are installed ensure they are enabled for the SMTP service using Get-ExchangeCertificate

Use network monitor to capture network traffic between the sending and receiving server

Test-Mailflow

Tests mail submission, transport and delivery

Tests services by verifying that each mailbox server can successfully send itself a message.

Remote functionality to test between remote Mailbox Servers

169


Troubleshooting outbound internet mail flowFollow the same steps as when troubleshooting mail flow issues between hub servers

Ensure that an appropriate Internet send connector has been configured with a * address space

When the send connector is configured to use DNS

From the sending server use Nslookup to determine external email domains MX records

Set type=MX

From the sending server telnet to port 25 of the identified remote host and send a test message

When the send connector is configured to use a Smart Host

From the sending server telnet to port 25 of the Smart Host and send a test message

170


Troubleshooting inbound internet mail flowEnsure that an authoritative accepted domain has been configured for the email domain and has public MX records registered

Ensure that an email address policy has been enabled for the accepted domain and users have the email address applied

Run netstat –anb on the Hub or Edge servers responsible for receiving inbound internet mail and verify that MSExchangeTransport.exe is listening on port 25

Verify the public MX records registered for the receiving email domain using nslookup

Set type=MX

May need to point nslookup to an external DNS server

“server <External DNS Server IP>”

Using a machine on the internet telnet to port 25 of the server identified in the MX record and send a test message

171


Troubleshooting inbound internet mail flow (Cont...)Ensure that any firewalls are configured to forward inbound internet messages onto the Exchange servers responsible for inbound internet mail

Verify the necessary receive connectors are enabled and configured appropriately on the Hub or Edge servers involved

Default receive connector configuration, Remote IP range, message size restrictions, authentication methods and types (specifically anonymous connections)

For Hub servers default Receive connector will need to be modified to allow anonymous connections

172

173

LAB 5 Mailflow

Agenda






Troubleshooting Mail Queues

Mail Queues: Architecture

Mail Queues: Queues





Internal Transport


SP1 Changes

174


Queues are temporary holding locations for messages that are waiting to enter the next stage of processing

Each queue represents a logical set of messages that a transport server processes in a specific order

Exist only on Hub Transport or Edge Transport server roles

Exchange 2007 all messages stored in the same location

All queues are stored in a single ESE database

175


ESE Database Queue filesESE Database files centrally located on the server

By default located at ...\Exchange Server\TransportRoles\data\Queue

Circular logging is used

Configuration options stored in EdgeTransport.exe.config

File Description

Mail.que ESE Database file that stores all the queued messages

Tmp.edb Temp database file used to verify the queue database schema

on startup

Trn*.log Transaction logs that record all changes to the queue

database

Trntmp.log Temporary transaction log created in advance

Trn.chk Tracks the log entries that have been committed to the

database

Trnres00001.jrs & Trnres00002.jrs

Transaction reserve log files. Used when the hard disk drive

that contains the transaction log runs out of space to stop the queue database cleanly

176

Agenda








Mail Queues: Queues





Internal Transport


SP1 Changes

177

Mail Queues: Queues

Several queues in Exchange Server 2007Submission

Remote Delivery

Mailbox Delivery

Poison Message

Unreachable

178

Mail Queues: Queues

Submission QueueUsed by the categorizer to gather all messages that have to be resolved, routed, and processed by Transport agents

Only one Submission queue per transport server

Messages remain here until categorization is complete

Messages enter Submission Queue from various sources

179

Mail Queues: Queues

Messages may queue up in the Submission Queue due toIssues with AD such as slow DCs, AD permissions, unable to communicate with a DC

Problems with Transport Routing Agents

Performance issues on the server itself such as a disk or processor bottleneck

Large nested or query-based distribution groups

180

Mail Queues: Queues

Remote Delivery QueuesHandles the queuing of a message to a specific SMTP based target destination

Remote SMTP Host

Drop Directory

Queue for every remote domain

Created and deleted dynamically

Each remote delivery queue contains messages being routed to the same delivery destination

Edge Servers always places messages into Remote Delivery Queues unless they are poison or unreachable

181

Mail Queues: Queues

Messages may queue up in the Remote Delivery queue due toConnectivity issues such as network links being down or ports being blocked

DNS name resolution issues

Mis-configurations of receive connectors such as incorrect Authentication settings or Remote IP Ranges

Certificate issues

Performance issues on either the sending or receiving server

182

Mail Queues: Queues

Local Delivery QueueHolds messages being delivered to a Mailbox server in the local AD site using encrypted Exchange RPC

Only available on Hub Servers

Processed by Store Driver

More than one mailbox delivery queue can exist

183

Mail Queues: Queues

Messages may queue up in the Local Delivery queue due toConnectivity issues to the Mailbox server such as network links being down or ports being blocked

Services or stores being offline on the Mailbox server

DNS name resolution issues

Performance issues on the local transport or receiving Mailbox server

Unable to log onto mailbox due to incorrect permissions for the Mailbox server

184

Mail Queues: Queues

Poison Message QueueUsed to isolate messages that are detected to be potentially harmful

Contains messages that caused the Transport Worker Process to crash

Exchange 2003 messages could repeatedly crash SMTP Service

Required manual extraction

Exchange 2007 can detect if a message crashes Transport

Message is removed from processing and placed in the Poison Message Queue

Maintains “Poison Message Count” on each message

If count exceeds threshold, moved to Poison Message Queue

Threshold controlled by PoisonThreshold value on TransportServer settings

Default 2

Messages can be deleted by admin, exported for debug, resubmitted or expire

185

Mail Queues: Queues

Unreachable QueueContains messages that cannot be routed to their destinations

Messages that have no route due to configuration errors

Problems with connectors

Missing attributes on mail enabled objects (HomeMDB)

Messages without routes don’t NDR immediately they are placed in the Unreachable queue

Each transport server can have only one Unreachable queue

Messages are automatically resubmitted for categorization if routing topology changes are detected

186

Mail Queues: Queues

Scenarios where messages are placed into the Unreachable queue

The recipient is a valid Active Directory recipient object. However a routing path cannot be calculated for that recipient

The recipient is an external SMTP address and a matching connector cannot be found for the address space

The recipient is a distribution group and the expansion server for the distribution group is invalid or does not have the Hub Transport server role installed

The recipient is an SMTP address recipient of a message that was received on a Receive connector that is linked to a Send connector that is ignored by the routing component of the categorizer because it is disabled or misconfigured in some way

187

Mail Queues: Queues

Scenarios where messages are NOT placed into the Unreachable queue and an NDR is generated instead

The routing path cannot be calculated for a recipient because constraints, such as message size restrictions, prevent delivery of the message using the single, deterministic route calculated by the categorizer

The recipient is a non-SMTP address and a matching connector cannot be found. Or the matching connector is disabled or misconfigured

The recipient is a non-SMTP address recipient that was received on a Receive connector that is linked to a Send connector that is ignored by the routing component of the categorizer because the Send connector is disabled or misconfigured

188

Mail Queues: Queues

Queue message processing in RTMAll queues except for Submission queue process messages using First In/First Out (FIFO)

Submission Queue uses “Round Robin”

For every X number of higher priority messages processed process a lower priority message

Keeps higher priority messages from stopping delivery of low priority

Ratio cannot be modified

189

Mail Queues: Queues

Priority Queuing in Exchange 2007 SP1Priority queuing affects the transmission of messages from a delivery queue to the destination messaging server

When enabled higher priority messages are transmitted before lower priority messages to their destinations

Helps admins define specific SLA requirements for message delivery times

Enabled or disabled in EdgeTransport.exe.config

PriorityQueuingEnable parameter

When enabled the priority message queue limits (such as expiration of messages) in EdgeTransport.exe.config override the message queue limits set by Set-TransportServer

190

Mail Queues: Queues

QUEUING

OUTBOUND

INBOUND

SUBMISSION

CATEGORIZER

POISON

UNREACHABLE

REMOTE

MAPI

SMTPOut

SMTPIn

RESUBMIT

STOREDRIVER

PICKUP/REPLAY

DIRECTORY

AGENTS

191

Agenda








Mail Queues: Queues





Internal Transport


SP1 Changes

192


Troubleshooting mail queues involves looking at queue and message properties to help isolate the issue, specifically

Status

Either Active, Ready, Retry or Suspended

Next Hop Domain

Available for queues only

Shows the destination of the queue such as a Smart Host or another SMTP or mailbox server

Last Error

Lists the reason for message delivery failure

If a queue is or messages are in a retry state look at the Last Error value to help isolate the reason for failure

Use the Next Hop Domain value to help determine where the message is being delivered

193

194

LAB 6 Disaster Recovery

Agenda








Mail Queues: Queues





Internal Transport


SP1 Changes

195

Certificate and Transport: Overview

SMTP STARTTLSMutual TLS

Opportunistic TLS

Force Required TLS

By Connector

Domain Security

POP

IMAP

Edge Synchronization

Certificate and Transport: Overview (cont.)

Direct Trust CertificateRetrieved from AD

Must be available also in local Certificate Store

SMTP X-AnonymousTLSHub to Hub

Hub to Edge/Vice Versa

Direct Trust Authentication


Agenda








Mail Queues: Queues


Troubleshooting Transport Certificates

Certificates and Transport: Overview



Internal Transport


SP1 Changes

199

Certificate Selection - STARTTLS

Process that components go through to determine which certificate should be used for an incoming connection

FQDNAlways based upon the connector FQDN

Get-<SEND | RECEIVE>Connector | FL FQDN

Certificate Selection - STARTTLS (cont.)

List of valid Certificates are found based on the connector FQDN

Has either no extended key usage extension or has the extended key usage extension containing the Server Authentication Object IdentifierHas either no key usage extension or it has the key usage extension with digital signature bit assertedIt has a RSA public key > 1024 bits in sizeHas a valid certificate chain up to a trusted root (or self signed)Revocation checking passes on the certificate chainThe private key is present and accessible (Network Service)The private key is not stored in a removable deviceThe private key is not UI protected

Certificate Selection - STARTTLS

From the remaining list, pick the best certificate (in order of preference)

Trusted CA Issued Certificate preferred over Self-Signed

Newest installed certificate over oldest

An older 3rd Party CA issued certificate would be used over a newer self-signed certificate

Certificate Selection Process – Inbound STARTTLS


POP3 and IMAP4

Selection process is similar to SMTP STARTTLS

Three Exceptions:

Instead of FQDN they use X509CertificateNameGet-POPSettings

Get-IMAPSettings

Newest valid certificate wins – changed in SP1

Does not support Wildcards such as *.fourthcoffee.com –changed in SP1

Agenda








Mail Queues: Queues






Internal Transport


SP1 Changes

205

Troubleshooting - STARTTLS

Tools UsedExchange Management Shell

TELNET.EXE (is STARTTLS advertised?)

TCP.EXE (What certificate is being served?)

CertUtil to verify certificates

Application Logs

Troubleshooting - STARTTLS

What is the FQDN of the Receive ConnectorGet-ExchangeCertificate –DomainName <FQDN>Get-ExchangeCertificate –DomainName <FQDN> | FL *Is there more than one certificate?

Are any certificates issued from a trusted CA?Are the certificates valid?What is the newest certificate?Can we access the CRL Distribution Point? Proxy?

Troubleshooting

[PS] C:\>Get-ExchangeCertificate 91BFBDD870D8928018E22B736922411645218B85 | fl *

CertificateDomains : {clt-e2k7.fourthcoffee.com, clt-e2k7}

CertificateRequest :

IisServices : {IIS://clt-e2k7/W3SVC/1}

IsSelfSigned : False

RootCAType : Enterprise

Services : IIS, SMTP

Status : Valid

PrivateKeyExportable : True

Archived : False

FriendlyName : Microsoft Exchange

IssuerName : System.Security.Cryptography.X509Certificates.X500Distin

guishedName

NotAfter : 9/17/2009 10:11:42 AM

NotBefore : 9/18/2007 10:11:42 AM

HasPrivateKey : True

SerialNumber : 610FCD9F000000000011

SubjectName : System.Security.Cryptography.X509Certificates.X500Distin

guishedName

Thumbprint : 91BFBDD870D8928018E22B736922411645218B85

Version : 3

Handle : 133224224

Issuer : CN=LON-E2K7, DC=fourthcoffee, DC=com

Subject : CN=clt-e2k7.fourthcoffee.com

Troubleshooting

[PS] C:\Get-ExchangeCertificate 6CC3257C2236DFC88BA40CD9A374C9E53CC18E2B | fl *

CertificateDomains : {clt-e2k7, clt-e2k7.fourthcoffee.com}CertificateRequest :

IisServices : {}

IsSelfSigned : True

RootCAType : Registry

Services : SMTP

Status : ValidPrivateKeyExportable : False

Archived : False

FriendlyName : Microsoft Exchange

IssuerName : System.Security.Cryptography.X509Certificates.X500Distin

guishedName

NotAfter : 9/19/2008 9:44:29 AM

NotBefore : 9/19/2007 9:44:29 AM

HasPrivateKey : TrueSerialNumber : 9816558F2C99EF924E7A5AA1730498DA

SubjectName : System.Security.Cryptography.X509Certificates.X500Distin

guishedName

Thumbprint : 6CC3257C2236DFC88BA40CD9A374C9E53CC18E2B

Version : 3

Handle : 68983992

Issuer : CN=clt-e2k7

Subject : CN=clt-e2k7

Troubleshooting

TCP.EXE

Troubleshooting

[PS] C:\>certutil -verify certnew.cerIssuer:

CN=LON-E2K7DC=fourthcoffeeDC=com

Subject:CN=clt-e2k7

Cert Serial Number: 610cbc3c000000000010

dwFlags = CA_VERIFY_FLAGS_DUMP_CHAIN (0x40000000)

ChainFlags = CERT_CHAIN_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT (0x40000000)HCCE_LOCAL_MACHINE

CERT_CHAIN_POLICY_BASE-------- CERT_CHAIN_CONTEXT --------ChainContext.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)

ChainContext.dwErrorStatus = CERT_TRUST_REVOCATION_STATUS_UNKNOWN (0x40)ChainContext.dwErrorStatus = CERT_TRUST_IS_OFFLINE_REVOCATION (0x1000000)ChainContext.dwRevocationFreshnessTime: 4 Days, 22 Hours, 34 Minutes, 49 Seconds

SimpleChain.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)

SimpleChain.dwErrorStatus = CERT_TRUST_REVOCATION_STATUS_UNKNOWN (0x40)SimpleChain.dwErrorStatus = CERT_TRUST_IS_OFFLINE_REVOCATION (0x1000000)SimpleChain.dwRevocationFreshnessTime: 4 Days, 22 Hours, 34 Minutes, 49 Seconds

CertContext[0][0]: dwInfoStatus=102 dwErrorStatus=1000040Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=comSubject: CN=clt-e2k7Serial: 610cbc3c000000000010Template: WebServer06 99 41 18 54 db 2d 8b 2c ae 0a 5d d7 b5 27 54 42 d8 20 0b

Element.dwInfoStatus = CERT_TRUST_HAS_KEY_MATCH_ISSUER (0x2)Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)

Element.dwErrorStatus = CERT_TRUST_REVOCATION_STATUS_UNKNOWN (0x40)

Certutil Sample :

Troubleshooting


Element.dwErrorStatus = CERT_TRUST_IS_OFFLINE_REVOCATION (0x1000000)CRL 15:

Issuer: CN=LON-E2K7, DC=fourthcoffee, DC=com

93 a6 c0 1b ad cf 8f 9a 91 3b 6e b5 7e bc 93 ed 53 89 89 5c

Delta CRL 16:


35 9e 39 96 9d e8 08 ce 3c 16 a5 99 d5 aa 28 89 d1 54 db 3e

Application[0] = 1.3.6.1.5.5.7.3.1 Server Authentication

CertContext[0][1]: dwInfoStatus=10c dwErrorStatus=0


Subject: CN=LON-E2K7, DC=fourthcoffee, DC=com

Serial: 115643e01d0eab874e228cc4545d7e6c

d6 80 20 2f 11 ad f2 39 53 b5 92 df c1 5a 26 28 c4 5c e5 90

Element.dwInfoStatus = CERT_TRUST_HAS_NAME_MATCH_ISSUER (0x4)

Element.dwInfoStatus = CERT_TRUST_IS_SELF_SIGNED (0x8)

Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)

Exclude leaf cert:

ef af 86 b5 a7 9e 25 51 44 18 98 b6 69 f9 df 62 c5 65 31 66

Full chain:

a8 51 bc 17 d8 b4 94 5a 6a 3d b9 01 89 bc c6 63 37 8e 0b ea


Subject: CN=clt-e2k7

Serial: 610cbc3c000000000010

Template: WebServer

06 99 41 18 54 db 2d 8b 2c ae 0a 5d d7 b5 27 54 42 d8 20 0b

The revocation function was unable to check revocation because the revocation se

rver was offline. 0x80092013 (-2146885613)

Revocation check skipped -- server offline

ERROR: Verifying leaf certificate revocation status returned The revocation func

tion was unable to check revocation because the revocation server was offline. 0

x80092013 (-2146885613)

CertUtil: The revocation function was unable to check revocation because the rev

ocation server was offline.CertUtil: -verify command completed successfully.

Certutil Sample cont: :

Agenda








Mail Queues: Queues






Internal Transport


SP1 Changes

213

Internal Transport - Certificate

The Direct Trust Certificate:Whenever X-AnonymousTLS is used

Hub to HubHub to EdgeEdge to Hub

Used to establish “Direct Trust” authentication after X-AnonymousTLS negotiation between a Hub and Edge server.Used to establish secure LDAP connections from Hub for Edge SynchronizationUsed to encrypt and decrypt EdgeSynchronization credentials which are stored in the directory.

Direct Trust Certificates = Default Certificate

Internal Transport - Certificate selection

The Internal Transport Certificate selection process simply loads from msExchServerInternalTLSCert property on the Exchange Server object

Must also be found in the local Computer Store

Expired Direct Trust Certificates do not affect mail flow

Warning:

Error:

Event Type: Warning

Event Source: MSExchangeTransportEvent Category: TransportServiceEvent ID: 12017

Description:A direct trust certificate will expire soon. Thumbprint:

2135A85FC400DF078D56A7A1EBB1E4330DD68596, hours remaining: 720

Event Type: Warning

Event Source: MSExchangeTransportEvent Category: TransportServiceEvent ID: 12015

Description:A direct trust certificate expired. Thumbprint:

2135A85FC400DF078D56A7A1EBB1E4330DD68596

Internal Transport - Determining Certificate

RTM use Certlib.ps1

SP1 use Get-TransportServer

Internal Transport

Error when Certificate Store cannot be found

Occurs when the Direct Trust certificate has been “forcibly” removed from the system (e.gMMC)

Agenda








Mail Queues: Queues






Internal Transport


SP1 Changes

218


Tools Used:Exchange Management Shell

TELNET

Protocol Logs

CertLib.ps1

1st Determine this is a direct trust issue

Troubleshooting Direct Trust - Direct Trust Issue?

[PS] C:\>Get-Queue 5 | fl

Identity : clt-e2k7\5DeliveryType : SmtpRelayWithinAdSiteToEdgeNextHopDomain : edgesync - northamerica to internetNextHopConnector : 4039e148-2af7-4889-9de8-6cf6f1b2716eStatus : RetryMessageCount : 1LastError : 451 4.4.0 Primary target IP address responded with:

454 4.7.0 Temporary authentication failure. Attempted failover to alternate host, but that did not succeed.

Either there are no alternate hosts, or delivery failed to all alternate hosts

.LastRetryTime : 9/19/2007 8:45:33 AMNextRetryTime : 9/19/2007 8:50:33 AMIsValid : TrueObjectState : Unchanged

Is ExchangeServer authentication enabled on the receive connector?

Make sure both Hub and Edge show X-ANONYMOUSTLS available


Confirm X-AnonymousTLS

Troubleshooting Direct Trust - Protocol Logging

2007-09-19T12:27:38.994Z,0,,10.0.200.200:25,*,,attempting to connect

2007-09-19T12:27:39.004Z,1,+,,

...

2007-09-19T12:27:39.014Z,17,>,X-ANONYMOUSTLS,

2007-09-19T12:27:39.024Z,18,<,220 2.0.0 SMTP server ready,

2007-09-19T12:27:39.024Z,19,*,,Sending certificate

2007-09-19T12:27:39.024Z,20,*,CN=clt-e2k7.fourthcoffee.com,Certificate subject

2007-09-19T12:27:39.024Z,21,*,"CN=LON-E2K7, DC=fourthcoffee, DC=com",Certificate issuer name

2007-09-19T12:27:39.024Z,22,*,610FCD9F000000000011,Certificate serial number

2007-09-19T12:27:39.024Z,23,*,91BFBDD870D8928018E22B736922411645218B85,Certificate thumbprint2007-09-19T12:27:39.024Z,24,*,clt-e2k7.fourthcoffee.com;clt-e2k7,Certificate alternate names

2007-09-19T12:27:51.292Z,25,*,,Received certificate

2007-09-19T12:27:51.292Z,26,*,3B1C9C4472B9ED9E9981262F48F164E4EDB02F0D,Certificate thumbprint

2007-09-19T12:27:51.292Z,27,*,SMTPSendEXCH50 SendRoutingHeaders SendForestHeaders SendOrganizationHeaders,Set Session Permissions

2007-09-19T12:27:51.292Z,28,*,,DirectTrust certificate

2007-09-19T12:27:51.292Z,29,*,CN=E2K7-EDGE1,Certificate subject

2007-09-19T12:27:51.292Z,30,*,CN=E2K7-EDGE1,Certificate issuer name

2007-09-19T12:27:51.292Z,31,*,126DA18D9F7FE69B48044E9D4985B894,Certificate serial number

2007-09-19T12:27:51.292Z,32,*,3B1C9C4472B9ED9E9981262F48F164E4EDB02F0D,Certificate thumbprint

2007-09-19T12:27:51.292Z,33,*,E2K7-EDGE1;E2K7-EDGE1.fourthcoffee.com,Certificate alternate names

2007-09-19T12:27:51.292Z,34,>,EHLO clt-e2k7.fourthcoffee.com,

2007-09-19T12:27:51.302Z,35,<,454 4.7.0 Temporary authentication failure,

2007-09-19T12:27:51.302Z,36,>,QUIT,

2007-09-19T12:27:51.302Z,37,-,,Remote


[PS] C:\>gettlscertfromad clt-e2k7Running on an Edge Server - pulling cert details from AdamSystem.DirectoryServices.DirectoryEntryRunning on an Edge Server - pulling cert details from Adam(&(objectclass=msExchExchangeServer)cn=clt-e2k7)Getting Prop

Thumbprint Subject---------- -------715F6840E3B4CC241CFDC9D912A5E8491B2BDE74 CN=clt-e2k7

[PS] C:\>gettlscertfromad e2k7-edge1Running on an Edge Server - pulling cert details from AdamSystem.DirectoryServices.DirectoryEntryRunning on an Edge Server - pulling cert details from Adam(&(objectclass=msExchExchangeServer)cn=e2k7-edge1)Getting Prop

Thumbprint Subject---------- -------3B1C9C4472B9ED9E9981262F48F164E4EDB02F0D CN=E2K7-EDGE1

Confirm Certificate Thumbprints:

Agenda








Mail Queues: Queues






Internal Transport


SP1 Changes

224

SP1 Changes

Certificate Logging

Get-ExchangeCertificate –Domain FOO.COM

Opportunistic TLS fallback

Direct Trust Certificate

Changing Terminology -“Internal Transport Certificate”

Updated anytime SMTP provided as service

Warnings upon update

Don’t have to re-subscribe w/update on Hub

No more 1037/2019 events

Attempts to fallback to alternate certificates if can’t be loaded

POP/IMAPSelection now prefers PKI

Supports Wildcards

SP1 Changes

EBDDE4C98F71840199B4256B9368F265A92DDB0C: Rejected. Unable to access the associated private key for the certificate.

Searching for a certificate that has one of the following FQDNs :

mail.fourthcoffee.com

60CF94F4522A25DFEBB734F4636FB1D1F77D819A: Is not valid for signing, dropping from consideration.

Considering certificate A7C7D7B88B767BAA6003A1F0DBE0DF0937210C68

A7C7D7B88B767BAA6003A1F0DBE0DF0937210C68: Rejected. Has a key size less than 1024 bits, dropping from consideration.

Considering certificate CA979162AC2854BBE389153D965358379F8CD43E

CA979162AC2854BBE389153D965358379F8CD43E: Selected. PKI issued certificate.

Logging Sample

Troubleshooting Mailbox Server(MBX)

Agenda

Databases

Public Folders

Recipient Management

Distribution Lists & Address Lists


Exchange Search

ExMON

ESE System

Transaction

Log File

715

525

415

841

10

Database Storage Files

Memory

7 15 25 4

17 8 3 1

10

.EDB Storage File

1 2 3 4

5 6 7 8

9 10 11 12

8 KB

Making Changes to the Database

Transaction

Log File

715

525

415

841

10

Database Storage FilesMemory

980

77

1001

7

1001

7

.EDB Storage File

1 2 3 4

5 6 7 8

1001

7

9 10 11 12

1001

Important JET Database Files

Three important file extensions to remember:EDB (Exchange database files)

LOG (Exchange Transaction logs)

CHK (Exchange transaction checkpoint files)

Checkpoint File

Exxnnnnn.log

Exx.chk

Transaction LogEntries Writtento the Database

Transaction LogEntries Not Yet Written to the

Database

Storage Groups

Set of all databases that share common log files

Separate instance of Jet

Up to 50 Storage groups per Server (Enterprise)

5 Databases per Storage Group but 50 maximum databases (Enterprise)

Recommendation is to have one database for one storage group

Utilities in 2007

No change compared to 2003!Eseutil command and options are the same but now can be also used on Hub and Edge transport databases

Isinteg command and options are the same

Troubleshooting Exchange Databases

For database problems all the troubleshooting techniques used in 2003 are still valid


Continuous Replication flavors

LCR: Local Continuous Replication

CCR: Cluster Continuous Replication

SCR: Standby Continuous Replication


Troubleshooting Continuous Replication

Exchange Management Console

Configuration

Management

Monitoring

Exchange Management ShellGet-StorageGroupCopyStatus

Test-ReplicationHealth

Performance countersMany counters available to monitor Continuous Replication


Exchange 2007 SP1 Changes

Online Database Checksum (DBScan)

Addresses CCR scenario where backup occurs off passive (Active will never be completely scanned)

Opt in feature via Registry key

Set on a per server basis

Throttle parameter (unlimited by default)

Uses half of the Online Defrag Maintenance Window time

Will notify corruption via eventlog (-1018, -1022 etc)


Online Defrag Monitoring:New Perfmon Counter:MSExchange Database->Online Defrag Pages Freed/sec

Extended Event information:Event Type: Information

Event Source: ESE

Event Category: Online Defragmentation

Event ID: 703

Date: 6/20/2007

Time: 6:34:26 AM

User: N/A

Computer: DF-MBX-30

Description:

MSExchangeIS (19052) SG06: Online defragmentation has completed the resumed pass on database 'e:\MDB06\priv06.edb', freeing 42794 pages. This pass started on 6/16/2007 and ran for a total of 124919 seconds, requiring 7 invocations over 4 days. Since the database was created it has been fully defragmented 14 times over 73 days.


How to determine if OLD is running often enough?2 week rule of thumb or analyze serverAnalyze Perfmon log taken during OLD window

121:1

If Read:Freed ratio is greater than 100:1 then the OLD window can be reducedIf Read:Freed ratio is less than 50:1 then the OLD window should be increased

Why reduce?Increase backup windowReduce snapshot/block level differential sizes (DPM v2)Validate that Online Checksum/Page Zeroing can be introduced with current OLM window

Agenda

Databases

Public Folders




Exchange Search

ExMON

Public Folder Replication

Mail based as in previous versions of ExchangeEnsure the public store has an email address

Check message tracking

Enable diagnostic logging

CCR cannot have a public store if there is more than one PF store

Force Public Folder replication

In the management consoleUpdate Hierarchy

Update content

Using the management shellUpdate-PublicFolderHierarchy

Update-PublicFolder

Public Folder Replication Storm

Public folder replication storm can still occur in Exchange 2007

Suspend-PublicFolderReplication

Stops all replication except hierarchy

Resume-PublicFolderReplication

Public Folder Referrals

Defines Client Public folder access

Routing Group based in Exchange 2000/2003

Active Directory Site based in Exchange 2007

Troubleshooting Referrals

Connection status in Outlook

Get-PublicFolderDatabase

UseCustomReferralServerList

CustomReferralServerList

Link cost (AD and RGC)

Agenda

Databases

Public Folders




Exchange Search

ExMON


Simplified Recipient Provisioning for the Exchange Administrator

Support for “Split Permissions” within a single forest

Ability to delegate Recipient management to a lower level administrator

Ability to create Active Directory object and mail- or mailbox-enable it

Instant-on recipients – no need to wait or “kick” the RUS to stamp objects

Rich “filtering” support – includes domain- and forest-wide scoping

Allows administrators to see only the objects relevant to them

New recipient types plus clear distinction of all recipient types

Conference Room and Equipment Mailbox (Resource Mailbox)

Policy support for select mailbox settings

Ability to apply the same settings to all recipients associated with a policy

Unified Messaging, Messaging Records Management, and ActiveSync

Recipient Policies still exist but are now called E-Mail Address Policies

Page 253

Working with Recipients

Recipients are primarily mailbox-enabled Active Directory users

Recipients are managed through the Exchange Management Console or the Exchange Management Shell

Active Directory Users and Computers (ADUC) is no longer extended for management of Exchange recipients

User AD properties relevant for the Global Address List can be managed through the Recipient Configuration container in Exchange Management Console

Active Directory User accounts can be created from within the Exchange Management Console when they are mailbox-enabled

Page 255

Working with Recipients and ADUC

Active Directory Users and Computers (ADUC) is no longer extended to manage Exchange recipients

It is not supported to mailbox-enable user accounts using ADUC when the mailboxes will be housed on Exchange 2007 servers.

If there is an Exchange Server 2003 RUS server operational, the ADUC mailbox operation will succeed, so the mailbox will be able to send and receive messages

Mailbox is considered legacy and certain features or actions, or properties will be blocked

Set-Mailbox -ApplyMandatoryProperties

Page 256

Scoping

Recipient Configuration Center supports domain- andforest-wide scoping

Ability to specify which DC Console should connect to

Scope is configurable, even down to OU

$AdminSessionADSettings session variable (in shell)

Domain Scope is default behavior

Determined by domain of which the Server is a member:

Only recipients (e.g., redmond\evand) in selected domain can be found

Referenced recipients (e.g., Membership, Delegate, Owner, etc.) are exempt

Reduces issues related to replication

Forest Scope can display and find all recipients within the forest

Provides a complete view of the GAL

Enable/Disable vs. New/Remove

Enable/Disable

Adds or removes Exchange attributes from existing Active Directory® objects

Enable – adds attributes to an existing Active Directory object –mail-enabled or mailbox-enabled

Disable – removes attributes returning Active Directory object to non-Exchange state

StoreMailbox in MDB will fall under mailbox retention and will eventually be purged

New/Remove*Creates or deletes Active Directory® objects plus adds and removes Exchange attributes in one step

New – creates Active Directory object and mail-enables or mailbox-enables the objectDefault Remove – removes Active Directory object. StoreMailbox in MDB will fall under mailbox retention and will eventually be purged-Permanent: removes Active Directory object and StoreMailboxin MDB will be purged immediately (shell only)

* Must have Account Operator privileges

Page 262

Email Address Policies

Created pre-canned filters to simplify definition and usage forcommon cases

All Recipient Types, Users with Mailboxes, Resource Mailboxes, Mail-Enabled Contacts, and Mail-Enabled Groups

Conditions Supported: State or Province, Department, Company, and Custom Attributes

Ability to schedule the creation and application of Email Address Policies foroff-hour execution when using EMC

RUS as a service no longer needed, resulting in reduced system processing demand

Mailbox Manager functionality separated from EAPs

Replaced by Messaging Records Management functionality

Advanced or Non-Mainline (Shell Only)

Custom Filters - will be visible, but not editable, in GUI

Page 263

Managing Mailboxes

Well-known functionality are still there

New mailbox

Move mailbox

Delete mailbox

Change Mailbox properties

Page 265

New Mailbox Management tasks

Statistics

Get-LogonStatistics

Get-MailboxStatistics

Get-MailboxFolderStatistics

Troubleshooting Mailbox access

Test-MapiConnectivity

Outlook logging

831053 How to turn on the Enable Mail Logging option for troubleshooting in Outlook 2003 and Outlook 2007

Network

Take network trace

Reproduce the problem locally

http://support.microsoft.com/default.aspx?scid=kb;EN-US;831053




Moving Mailboxes

You can use the Exchange Management Console or the Exchange Management Shell to move mailboxes

You can move mailboxes across mailbox databases, across servers, across domains, across Active Directory sites and across forests

You can also move mailboxes among different versions of Microsoft Exchange Server (2000/2003/2007 only)

Move mailbox is more resilient (Pre-Validation)

Exchange Management Shell Command: move-mailbox

More options available

Note: You cannot use the Exchange Management Console to move mailboxes across forests. You must use the Exchange Management Shell instead.

Page 269

Troubleshooting Mailbox Move

Email Address Enforcement

IgnoreRuleLimitErrors cmdlet option

Damaged or corrupted messages

BadItemLimit cmdlet option

Skip errors validation from EMC Move-MailboxWizard

MfcMapi

Isinteg

Exmerge Replacement Need

Exmerge is not shipped with Exchange Server 2007

The Move-Mailbox, Export-Mailbox, and Restore-Mailbox tasks are implemented to cover many of the scenarios where ExMerge is used with Exchange Server 2003

Export to PST is possible in SP1

Page 274

ExMerge Replacement cmdlets

Export-MailboxExport mailbox content to another mailbox or PST

Must run 32-bits if exporting to PST

Must have Outlook installed

Can filter content

Can delete source message

Will export dumpster

Import-MailboxImports from PST

Must run 32-bits console

Restore-Mailbox


LAB 7 Troubleshooting MAPI access

LAB 8 Using MFCMAPI

Agenda

Databases

Public Folders




Exchange Search

ExMON

Distribution List Types

Mail-enabled Universal Distribution Group

Mail-enabled Universal Security Group

Mail-enabled Non-Universal Group

Dynamic Distribution Group

Automatic Group conversion

Users can select Universal Distribution Group to set permissions on folders

Exchange will automatically convert the group to security

Can potentially growth user security token

Can be disabledSet msExchDisableUDGConversion through ADSIEdit

Interaction with Exchange 2003

Dynamic Distribution groups created in Exchange 2003 must beupgraded to be modified in Exchange 2007

Get-DynamicDistributionGroup | Format-List Name,*RecipientFilter*,ExchangeVersion

If RecipientFilterType is "Legacy“ and ExchangeVersion is "0.0 (6.5.6500.0)” Set-DynamicDistributionGroup –recipientfilter {... } –forceupgrade$true

Exchange 2007 Distribution List can only use Universal groupscope

Common Issues

Unable to send to the Distribution Group from users external tothe Organization

When “Require that all senders are auhenticated “ flag is set on DLProperties

To solve the issue run: Set-DistributionList –RequireSenderAuthenticationEnabled $true

Unable to view the Distribution Group in EMC

When the group scope is Global or Domain Local

To solve this issue change the group scope to Universal scope

Default Address Lists

Default Global Address List

All Contacts

All Groups

All Rooms

Public Folders

All Users

Populating the Address Lists

In Exchange 2007 there is no Recipient Update Service (RUS)

PreCanned Filters

In order to update an Address List you have to run the cmdlet Update-AddressList

Update-AddressList can be scheduled to run using Exchange Management Shell

Issues

Unable to Edit the Address List Properties (Address List must beupgraded)

If ALs created by using Exchange 2003

Upgrade them to Exchange 2007 to use OPATH filtersSet-GlobalAddressList "Default Global Address List" -RecipientFilter {(Alias -ne $null -and (ObjectClass -eq 'user' -or ObjectClass -eq 'contact' -or ObjectClass -eq 'msExchSystemMailbox' -or ObjectClass -eq'msExchDynamicDistributionList' -or ObjectClass -eq 'group' -or ObjectClass-eq 'publicFolder'))}

Address List are not updated after modifying them

Exchange 2007 has no RUS to update them

Address Lists must be updated using EMC or EMS (Update-AddressList)

RUS API can be troubleshooted via :

Get-EventLogLevel MSExchangeAL | Set-EventLogLevel -Expert

Agenda

Databases

Public Folders




Exchange Search

ExMON

What is the Offline Address Book?

An Offline copy of the Global Address List

Used by Outlook clients in Offline mode or Cache mode

Several versions appeared over time

Version 2 appeared in Exchange 5.5

For clients Outlook 98 and later

Version 3 Appeared in Exchange 2003

For clients Outlook 2003 and later

Version 4 Appeared in Exchange 2003 SP2

For clients Outlook 2003 SP2 and later


Offline Address Book Overview

Exchange 2007 introduces a new mechanism for distributing the OAB that does not require Public Folders

HTTP(S) and Background Intelligent Transfer Service (BITS) can be used by Outlook 2007 clients

Advantages of this new method are:

Support for more concurrent clients

Reduction in bandwidth usage

More control over the distribution points

The web distribution is only available for OABv4

Legacy Clients will still need to utilize public folders for OAB obtainment

Outlook 2007 clients can utilize the web distribution for obtaining the OAB

3/24/2009 | Page 294

Offline Address Book generation

The Offline Address Book (OAB) is generated as usual by the OABgen component on a Mailbox Server

Files are published to \\mbxserver\ExchangeOAB share and to the public store if available

On all Client Access servers, an OAB virtual directory is created to serve the OAB

The Exchange File Distribution Service that runs on the CAS servers is responsible to getting the OAB content from the OABGen server

The virtual directory points to the directory %programfiles%\microsoft\exchange server\ClientAccess\OAB

In that directory, the different OABs are stored per <guid>

The .lzx files contains the OAB data in V4 format

The oab.xml contains metadata for Outlook 2007

Outlook 2007 is configured to retrieve the OAB via the OAB URL that is obtained through AutoDiscover. Otherwise it will download OAB from public folders like all other legacy clients

//mbxserver/ExchangeOAB





Offline Address Book size and network bandwidth usage

Started to become an issue with Outlook cache mode deployments

No limit for Public Folder connections

OAB throttling to control network bandwidth usage

Outlook Random Full OAB Request Timer

Key: HKCU\Software\Microsoft\Exchange\Exchange Provider

DWORD: Max Full OAB Download Wait

Value: Integer >=1


OAB V4 Improvements

OAB V4 is more compressed

Binpatch technology and LZX compression method used

Rebuild needs are reduced

Indexes generated by Outlook Client

Limited property sizes

Web distribution optimizes network bandwidth usage


Offline Address Book Web Distribution Scenario

Scenario

Corporate Headquarters (London)

User A

Remote Office

(Sao Paulo)

User B

User C

The Internet

CAS Server

Mailbox ServerFast Connection

Slow Link

Legend

Outlook

Web distribution self healing


OAB generated files are kept within the System Attendant mailbox

Deleted files from the mailbox role OAB share will be copied back

Deleted files from the CAS web virtual directory will be copied back from the Mailbox OAB share

Offline Address Book Public folder distribution


Outlook clients will connect through RPC to the public folder server holding a replica of the OAB

To reduce bandwidth usage you should:

Make sure to use OAB V4

Replicate the OAB on a public folder in every Active Directory site holding a Mailbox role

Or create an OAB per site and assign the mailbox stores to the local OAB

Don’t forget OAB Threshold registry setting

OAB Version used

Outlook 2003 SP2 and Outlook 2007 can use V4

Will failback on previous version if not available

Ensure that Version4 of the OAB is enabled

Get-OfflineAddressBook | fl Name,Server,Versions

Set-OfflineAddressBook –Versions Version4

If the profile is ANSI, OAB V2 will be used

Mainly for profiles linked to mailboxes moved from Exchange 5.5

Deploy GPO to force Profile conversion

Set registry key to force OAB V4 use

Key: HKCU \Software\Microsoft\Exchange\Exchange Provider

DWORD: OAB v4 Only

Value: 1


Troubleshooting OAB using diagnostic logging

Set Diagnostic level

Set-EventLogLevel -Identity “MSExchangeSA\OAL Generator” -Level Expert

Read event logs

Using the event viewer

Using Powershell

Get-EventLog Application | Where {$_.Category -eq"OAL Generator"}


Offline Address Book Integrity Checker (OabInteg)

Tool to simulate

Client connection to download OAB files from public folder store

Does not yet test web distribution (should be available soon)

OAB generation process

Downloadable from InternetYou can download OABInteg from here: http://gotdotnet.com/Community/UserSamples/Download.aspx?SampleGuid=A2338E73-F521-4071-9B1D-AAF49C346ACD

If run from the server install CDO 1.2.1 to test MAPI access. Downloadable from:

http://www.microsoft.com/downloads/details.aspx?FamilyID=e17e7f31-079a-43a9-bff2-0a110307611e&DisplayLang=en


http://gotdotnet.com/Community/UserSamples/Download.aspx?SampleGuid=A2338E73-F521-4071-9B1D-AAF49C346ACD



















OAB Generation Errors

Exchange server configured to generate the OAB

By default it is first Exchange Server in org which isExchange 2003 in mixed modes

In mixed mode:

Move OAB from Exchange 2003 server to Exchange 2007 server

Local replicas of OAB on Exchange 2007 server should be successfully replicated

All mailbox stores on Exchange 2007 server under Client Settings tab should have Default Offline Address Book associated


OAB Generation on CCR Clusters

CCR cluster

Only one node is generating OAB

When the node becomes passive OAB is not updated

Logs error event 9395

How to Fix:HKLM\System\CurrentControlSet\Services\MSExchangeSA\Parameters\Server-Name\EnableOabGenOnThisNode ="ThisNodeName"


OAB Download troubleshooting

Outlook send receive errors are saved in the “sync issues” folder

Use Err.exe to interpret the error

Enable Outlook logging

831053 How to turn on the Enable Mail Logging option for troubleshooting in Outlook 2003 and Outlook 2007


Check which OAB to download

Configured either on the mailbox store or on the recipient itself

Check Application log

Take a network trace





OAB Download from public folder troubleshooting

Has the OAB been generated on public folder?Is the OAB public folder enabled?

Is the version we want to download available?

Verify Application log on the OAB generator server

Look in the OAB public folder

Use OABInteg

Is the OAB public folder reachable by Outlook?

Verify where are the replicas of the OAB we should download

Make sure the public store holding the nearest replica is reachable

Check public folder replication or referrals

Look in the Outlook connection status if Outlook did effectively connect to the server holding the replica

Throttling might be preventing download


OAB Download from Web distribution troubleshooting

Make sure OAB is configured to generate V4Only V4 are web distribution enabled

Make sure the Outlook profile is UnicodeAnsi profile wouldn’t be able to download V4

Check if OAB has been generated on the Mailbox serverWeb distribution won’t be generated if system attendant can’t logon to its mailbox

Check if it has been replicated to the OAB virtual directories

Exchange file distribution service will copy it from the mailbox server to the CAS

Verify Outlook auto configuration (See CAS module)


Troubleshooting out of date OAB

Verify by connecting Online that inconsistency is only on OAB

Entries not stamped by Recipient Update API

Update-Recipient

Force an update and check event logUpdate-OfflineAddressBook

Out of date only for old Outlook clients

Those clients will use old OAB version. Might be an issue when Exchange 2007 is introduced in an existing organization

Active Directory replication

The Global Catalog used to generate the OAB may have directory replication issues. Use tools like replmon or repamin to verify



LAB 9 Troubleshooting OAB

Agenda

Databases

Public Folders




Exchange Search

ExMON

Exchange Search

Understanding Exchange Search

Difference between Exchange Search and store search

Unexpected results scenarios

Troubleshooting Exchange Search


Microsoft Exchange Server 2007 Search is a feature that allows you to quickly search text in messages through the use of pre-built indexes

Indexes occupy approximately 5 percent of the total mailbox database size

Kept separately in same location as database files

Used by OWA and Outlook online mode

Outlook cached mode uses new client-side search – Windows Desktop Search

Instant Search goes through attachments in Outlook

Can be extended to use any filter in Windows

Performance Enhancements

Outlook in online mode

Exchange Server 2007 Search Indexer and advanced find in Outlook 2007

Faster indexing than Exchange Server 2003 and Exchange Server 2000

New messages indexed in under a minute

Small storage tax (~5%) for indexes

Indexes/searches message bodies and attachments

Uses any filter installed in Windows

Can install new filters later

Outlook in Cached Exchange Mode

On Windows XP, Outlook uses Windows Desktop Search

On Windows Vista, Outlook uses Vista‟s built-in search engine

Performance Improvements

Mailbox


Index

MS Information Store

Service

Exchange Search Service

Notificatio

n

Update

Difference between Exchange Search and store search

Exchange Search Exchange Store Search

Faster Slower

Based on words Based on bytes stream

Search attachments* Cannot Search attachments

Uses index to search Uses serial scans

Not case sensitive Case Sensitive

Doesn‟t support MAPI

restrictionsSupport MAPI restrictions

* Attachments types that are supported by the installed filters

Unexpected results scenarios

Documents that are encrypted with the Digital Rights Management feature will not be indexed.

For attachments that do not have associated filters, the attachment will not be indexed, but the e-mail message will be indexed.

Advanced search grammar (for instance, typing "From:xyz" in the basic search bar searches the from: property for the string "xyz) is supported only when Instant Search is enabled. Instant Search requires that Windows Desktop Search 3.0 is installed.

Troubleshooting Exchange Search

Step 1• Is the MSExchangeSearch service started?

Step 2

• Is the IndexEnabled parameter set to true

• Get-MailboxDatabase |ft Name,IndexEnabled

Step 3

• Has the Exchange database been crawled?

• MSExchange Search Indices performance object=0

Step 4• Run the Test-ExchangeSearch

Step 5

• Check Event Viewer

• Source: MSExchangeSearch Indexer

Step 6• restart the Microsoft Search

Test-ExchangeSearch

The Test-ExchangeSearch cmdlet creates a message and attachment that only the Microsoft Exchange search can find. Unless a mailbox is specified in the Identity parameter, the message is stored in the System Attendant mailbox. The command waits for the message to be indexed and then searches for the content. The command reports success if the message content is found. The command reports failure if the content is not found after the interval set in the IndexingTimeout parameter has elapsed.

To run the Test-ExchangeSearch cmdlet, the account you use must be delegated the following:

1. Exchange Recipient Administrator role-and-

2. Exchange Server Administrator role and local Administrators group for the target server

How to rebuild the search index

Programmatically: use the ResetSearchIndex.ps1

Manually stop the service and deleting the file

GetDatabaseForSearchIndex.ps1 When the index directory files are provided, this script returns the associated mailbox database names.

GetSearchIndexForDatabase.ps1 This script returns index directories for the specified mailbox database names.


LAB 10 Troubleshooting Search

Agenda

Databases

Public Folders




Exchange Search

ExMON

What is Exmon?

Originally developed by Microsoft to understand user load on servers

Shows per user activity in details

Allows to track down high users

Introducing ExMon

Administrators can view the following using ExMon:

IP addresses used by clients

Microsoft Office Outlook® versions and mode (Cached Exchange Mode versus classic online)

Outlook client-side monitoring data

CPU usage

Server-side processor latency

Total latency (network and processing)

Network bytes

In Exchange 2007 works for all mailbox access


When to Use ExMon

Some Outlook users are complaining about latencies regarding mailbox access

RPC Average Latency is high

Want to know what Outlook versions are really in use

Want to find high RPC activity users

Are they in cache mode?

Want to know who is working among connected users

Determine usage pattern on healthy systems


Collecting ExMon Data

ExMon data can be collected in 3 ways

Live from the ExMon User Interface

Command Line

Perfmon collection

Command line or perfmon are recommended

The 'Live' mode constantly rolls, does not save data

Can use server CPU to process data

Command Line and Perfmon accommodates large files and ability to script and control

Larger files give much more insight including

Better aggregate statistics

Some data (like process name) is only traced on MAPI logon

ExMon data should be collected to local disks only

Consumes roughly 1MB/hour for every RPC Operation/sec

Viewing ExMon Data

Must be viewed on same OS or higher as collected on

Windows 2003 Server required to view data from Windows 2003 Server

Large files can take a long time to open, use CPU

Saving your work…

Command Line can save any 'By …' without displaying UI

File->Save will save all 'By …' views in one .csv

By Event (for a given user) can be saved only in UI

The Save Icon on the toolbar instructs ExMon to save ETL files captured during ‘Live Capture’

Analyzing Exmon Data

Know your environment

Establish baseline to compare

Detect RPC Average Latency peak using the performance wizard

Main Exmon points

CPU Time

Server Latency

Client Latency

Foreground Client Latency

Network Bytes

Basic Principles

Focus on the most expensive Users or Operations (unless you are troubleshooting a particular user)

Statistics are best for expensive operations or LOTS of inexpensive one

Look for problem to repeat in ExMon, then tackle

Longer captures are better than short ones

Expect some expensive operations to happen

Full sync of an OST

Occasional searches and sorts

Trick is to find the ones that happen frequently or really hurt

When looking at an individual user

Look for patterns of repetition

Compare to 'normal behavior'

© 2008 Microsoft Corporation. All rights reserved.

Information in this document, including URL and other Internet Web site references, is subject to change without notice. Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

For more information see Microsoft Copyright Permissions at http://www.microsoft.com/permission

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

The Microsoft company name and Microsoft products mentioned herein may be either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

THIS DOCUMENT IS FOR INFORMATIONAL AND TRAINING PURPOSES ONLY AND IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, WHETHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.

Terms of Use