-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
1/119
Chalk & Talk OnlineExchange Troubleshooting andRporting basics (tools and howto)Mihai Bobu
upport Engineer
Microso!t "rance
#o$e%ber nd
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
2/119
Content
Obecti$e* +now tools to troubleshoot and pro$ide
reporting and how to use the% withExchange
,hat ou will .nd*
/eneral presentation ,hen to use what
Tools and techni0ues !or di1erent kind o!issues
2ractical exa%ples 3&4
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
3/119
To begin with5
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
4/119
/eneral 2resentation
Scope: Exchange ''67 ''8 & ' Structure: C47 9:B7 MB; peci.c troubleshooting tools and
techni0ues !or each role as well asco%%on ones that can be used acros
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
5/119
Exchange er$ices
st
thing to do when so%ething goeswrong*
Check i! all Exchange ser$icescon.gured in 4uto%atic %ode are5 inser$ice ?er si%ple task which can sa$e a lo
o! troubleshooting ti%e on occasions
@
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
6/119
Exchange er$ices
Exchange ''6*
A
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
7/119
Exchange er$ices
Exchange ''8*
8
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
8/119
Exchange er$ices
Exchange ''*
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
9/119
Exchange er$ices
%essage and check 4pplication andste% e$ent logs
Man possible reasons wh restart
!ails* Docal ser$er con.gurationG
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
10/119
Exchange er$ices
Tool that < .nd %ost use!ul in case @Haspx
'
http://technet.microsoft.com/en-us/sysinternals/bb896645.aspxhttp://technet.microsoft.com/en-us/sysinternals/bb896645.aspxhttp://technet.microsoft.com/en-us/sysinternals/bb896645.aspxhttp://technet.microsoft.com/en-us/sysinternals/bb896645.aspx -
7/26/2019 Exchange Troubleshooting and Reporting Basics1
11/119
Exchange er$ices IExa%ple ISSUE:4!ter patching7 Outlook clients could no longconnect to one speci.c C4 ser$er
TROUBLESHOOTING: Ran Test-ServiceHealthand noticed tha
the R2C Client 4ccess ser$ice was not
running
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
12/119
Exchange er$ices IExa%ple TROUBLESHOOTING:Tried to start the RC4 ser$ice but it !ailed
with*
#othing %ore in the E$ent logs
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
13/119
Exchange er$ices IExa%ple TROUBLESHOOTING: Ran 2rocMon and .ltered !or 2rocess and Thread
4cti$it onl !or the RC4 executable in the .rstplace*
6
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
14/119
Exchange er$ices IExa%ple TROUBLESHOOTING: 4dded "ile 4cti$it to the $iew and started lookin
!or Results that were not uccess
,ent into JBin !older
to look !or the DD5
The .le was no longerthere
>
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
15/119
Exchange er$ices IExa%ple SOLUTION: Cop the DD !ro% another C4 ser$
(sa%e 2 and R: le$el)The ser$ice could be started a!terwaTurned out that there was a proble%
during patching while upgrading thatspeci.c DD
@
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
16/119
Exchange er$ices IExa%ple ISSUE:There was an Exchange ser$ice no longerstartingH
"or this exa%ple7 < used the RC4 ser$ice to illustrate the
that < saw although another Exchange ser$ice was in$ol$edthat ti%eH
A
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
17/119
Exchange er$ices IExa%ple TROUBLESHOOTING: 4gain7 nothing use!ul in the E$ent logs5 "iltering step b step in 2rocMon we hit th
issue*
8
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
18/119
Exchange er$ices IExa%ple ecurit tea% haddone so%e
rein!orce%ents*
SOLUTION: /a$e back the
appropriate per%issions
to #etwork er$ice and
started the ser$ice
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
19/119
C4
Troubleshooting
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
20/119
C4
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
21/119
C4
C4 ser$ers ha$e gotten %oreintelligent and the handle di1erenttasks (O4B distribution7 %ailbox %o$etcH)
till7 the .rst thing that co%es into o%ind when we think o! C4 is clientconnecti$it
2las a decisi$e role in end userexperience
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
22/119
C4
,hen discussing client connecti$it7there are %ain tpes o! connection M42< connections such as Outlook
K,ebL connections such as O,47
4cti$enc and all the other ,eb ser$icewhich are being handled b the
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
23/119
C4 logging* RC4 The de!ault settings !or this logging are as !ollow
and located in the .le calledMicrosoft.Exchange.RpcClientAccess.Service.exeng 2rotocol logging is enabled b de!ault
e!ault !older setting* "%Excha!eIstallir%#Lo!!i!#R$
liet &ccess#" Max sie in +B that a single log .le can grow to be!ore a new on
generated* "'()*("
Max sie in +B that the entire director o! logs can grow to be!othe oldest log is deleted* +'(*,./"
Dength o! ti%e in hours a log will be kept be!ore being deleted*+.)("
Dog tpe tags to be logged* oectiscoect0 Lo!o06
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
24/119
C4 logging* RC4 Nou can change the de!ault $alues as needed
The R2C Client 4ccess ser$ice %ust be restarted to pick up thischange The %ost use!ul changes are the e!ault "older and the Dog Tag
e3ault 1ol4er Setti! Nou %a want to change the location where the log .les are stored i! ou want to
the% to a di1erent dri$e etcHLo! Ta!s to 5e lo!!e4
9ere ou can speci! %ore details to log our other options are*
Rops 6re7ote operatios8 Rops option o1ers ou a top le$el %ention o! the operation being per!or%ed (Dog
Open"older7 Release7 /etContentsTable7 CreateMessage)
OperatioSpeci9c This option shows %ore detail !or each RO2 (!or exa%ple7 et2rops on CreateMes
operation)
Throttli!>
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
25/119
C4 logging* RC4 This logging can be $er help!ul to $eri! i! a use
has connected in the past at allH The ad%inistrato%ight recei$e a co%plaint b a user who sas theare not able to connect at allH ,ith protocol loggiturned on b de!ault7 the ad%inistrator can checwhen the last ti%e was the did success!ull
connect to the ser$er or i! the user has e$erconnected be!ore
E$en though this logging is designed to si%pl ke
track o! who is connecting and disconnecting7 it cpro$ide $aluable troubleshooting in!or%ation@
C l i C l
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
26/119
C4 logging* RC4 Exa%ple
9ere is a trace where :ser4 is creatina new %essage and sending the%essage to :serB*
'-'>-'TA*'8*'H'AGP7AG7G>7FoQContosoFouQExchange4d%inistrati$e /roup("N
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
27/119
C4 logging* RC4 Exa%ple
9ere is a trace with throttlingin!or%ation !or :ser4*
'-'>-'TG*>6*@@HG8P777FoQContosoFouQExchange
4d%inistrati$e /roup("N
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
28/119
C4 logging*
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
29/119
C4* #TDM $s +erberos Sceario:Both Outlook 4nwhere and Outlook T
clients are unable to connectH The get pro%pted%ultiple ti%es and still cannot connectH Theconnection status window %a show KConnecting
$ossi5le as@er:B de!ault7 #etlogon onl allo concurrent 42< calls !or authentication re0uestsour clients are connecting with #TDM it is possibthat a large nu%ber o! users could cause theseauthentication re0uests to ti%eout i! the CVs areo$erloadedH
2hat to 4o: 2rioritie +erberos authentication o$er #TDM
Change the MaxConcurrentAPIpara%eter on Cs and G
C4 E l
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
30/119
C4 I Exa%ple
ISSUE: o%e users are not able to access an
o! the Exchange ,eb er$ices*4utodisco$er7 O,47 a$ailabilit7 O4B
#ot all users are ha$ing this issue
6'
C4 E l
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
31/119
C4 I Exa%ple
TROUBLESHOOTING: Reproduced the issue with an a1ecte
user I tried to access O,47 but goterror K>'' Bad re0uestL in the web
browser Dooked into the
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
32/119
C4 I Exa%ple
TROUBLESHOOTING: #eed to see i! the re0uest %ade b the us
actuall gets up to the C4 ser$er o7 got down one le$el and checked
9TT2ERR logs Error >'' is $isible here with a clear indica
to the issue*'-'-' *6A*@ WWHWWH'HA AA WWHWWH@AH >>9TT2FH 2OT Fautodisco$erFautodisco$erHx%l >'' -Re0uestDength -
6
C4 E l
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
33/119
C4 I Exa%ple
EA$L&N&TION: :ser %aking the re0uest has a large
+erberos ticket7 as he is %e%ber in S''securit groups
RESOLUTION: 4pplied the !ollowing +Bs*http*FFsupportH%icroso!tHco%FkbF>G6@>
http*FFsupportH%icroso!tHco%FkbF''G>6
2lease note that the regke66
C4 E a%ple
http://support.microsoft.com/kb/2491354http://support.microsoft.com/kb/2020943http://support.microsoft.com/kb/2020943http://support.microsoft.com/kb/2020943http://support.microsoft.com/kb/2491354http://support.microsoft.com/kb/2491354 -
7/26/2019 Exchange Troubleshooting and Reporting Basics1
34/119
C4 I Exa%ple
ISSUE: Exchange er$er '' 2TM/ re$erse prox OO" does not work in Outlook and O,
6>
C4 Exa%ple
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
35/119
C4 I Exa%ple
TROUBLESHOOTING: Reproduced the issue !ro% the userVs
workstation Dooked into the
>'6H> translates to D re0uired
4sked !or a test !ro% O,4 per!or%ed dire
on the C4 to bpass TM/7 we got success
/ET ec r!rQowa& QOr anie 4uto%aticRe liesHslab >>6 HH6@
C4 Exa%ple
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
36/119
C4 I Exa%ple
EA$L&N&TION: D is acti$ated !or both O,4 and EC2 4uto%aticReplies de%and is !orwarded to
2ort ' Q 9TT2
The TM/ O,4 rule is con.gured to use 9Tnot 9TT2 !or the traXc between TM/ andExchange
SOLUTION: isabled D !or O,4 and EC27 did
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
37/119
C4 I Exa%ple 6
ISSUE: Exchange er$er '' 9ardware Doad-balanced C4 arra Outlook '' O4B download is not possible
68
C4 Exa%ple 6
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
38/119
C4 I Exa%ple 6
TROUBLESHOOTING: Made an atte%pt to download the O4B Checked F'@F'6
C4 Exa%ple 6
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
39/119
C4 I Exa%ple 6
TROUBLESHOOTING: "orced Outlook to connect directl to the
C4 (C4
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
40/119
C4 I Exa%ple >
ISSUE: 4!ter %o$ing the %ailbox !ro%
Exchange ''8 to Exchange ''7user cannot connect an%ore
9e gets the !ollowing error*KCannot open our de!ault e-%ail !olders7 Microso!t Excha
is not a$ailableH Either there are network proble%s or theExchange ser$er is down !or %aintenanceHL
>'
C4 Exa%ple >
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
41/119
C4 I Exa%ple >
TROUBLESHOOTING: Dooked into the RC4 logs and !ound the
!ollowing*'-'@-'T*'G*8H@'8P7>@>7YFoQ;NPFouQ4BC-;;;FcnQPPPPPPFcnQ?????FcnQabc Q TTTTY77O:TDOO+HE;E7>H'H>8A'H'''7Classic777ncacniptcp777'xAB4(rpc**Exception)7''*''*''H'@A''7essionropped7RpcEnd2oint* Zer$er:na$ailableException[ Conn%ust be re-established -S ZessioneadException[ The pri%ar owner logon has !ailedH ropping a
connectionH
'-'@-'T*'G*8HP7>@@7'YFoQ;NPFouQ4BC-;;;FcnQPPPPPPFcnQ?????FcnQabc Q TTTTY77O:TDOO+HE;E7>H'H>8A'H'''7Classic7##H##H'HG7!e'**c>*@A>G*8cca*adG\7ncacniptcpect7'7''*''*''7U
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
42/119
C4 I Exa%ple >
EA$L&N&TION:
C4 Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
43/119
C4 I Exa%ple @
ISSUE: Exchange ''8F'' %ixed
en$iron%entTM/ publishing rules !or 4cti$enc :sers with a certain tpe o! %obile
phone are co%plaining that !ro% ti%to ti%e their 6
C4 Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
44/119
C4 I Exa%ple @
TROUBLESHOOTING: 2ushed 4cti$enc logging at %ax on
Exchange '' C4 ser$ers 4sked users to call 9elp esk as soo
as the hit the issue and %ention theti%e as accuratel as possible
id that !or one da and then got the
logs (E$ent logs7
C4 I Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
45/119
C4 I Exa%ple @
&N&LDSIS: ,hen the users reported the issue7 we ha$
,arning '' in 4pplication E$ent logThis log alone does not tell us what happe
it is known to gi$e !alse positi$es5
C4 I Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
46/119
C4 I Exa%ple @
&N&LDSIS: ,hen we %atch the null\!\>null\!\@\c"C\6a'''\!'\c2olic\6ae!aultThrottling2olic\@"b'8Ga-eG'>->!e>
aGb-88ec8dG!>e\c#or% >>6 COR2J^^^^ '@H6HGHG 4pple-i2hone6CFG'H'A '' ' A> 8>
>A
C4 I Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
47/119
C4 I Exa%ple @
&N&LDSIS: o7 the error is co%ing !ro% the 9TT2 lae Dooked at the 9TT2ERR logs and saw the
!ollowing*
Connectionropped MExchangenc4pp2ool ,hen the i2honeFi2ad recei$es two @''
errors consecuti$el it resets the connectiand proceeds to re-download the userVs
entire %ailboxThis could co%e either !ro% the C4 ser$e
>8
C4 I Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
48/119
C4 Exa%ple @
SOLUTION:
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
49/119
C4 Exa%ple A
ISSUE: Exchange er$er ''6 BE cluster
hosting approxH 6''' %ailboxesThe ,6,2HE;E process runs at al%o
''\ C2:H 4s a conse0uence74cti$enc snchroniation is slow odoes not work at all
>G
C4 I Exa%ple A
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
50/119
C4 Exa%ple A
TROUBLESHOOTING: 4sked !or the
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
51/119
C4 Exa%ple A
TROUBLESHOOTING:
SOLUTION:Mo$ed the top talkers to
%ore power!ul Exchange '' plat!or@
C4 I Troubleshooting Too
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
52/119
C4 Troubleshooting Toohttp*FFblogsHtechnetHco%FbFexchangeFarchi$eF'F
6Fa-script-to-troubleshoot-issues-with-exchange-ai$esncHaspx ?er use!ul !or parsing
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
53/119
C4 Troubleshooting Too
4 %ore ad$anced tool is Lo! $arser
Stu4iohttp*FFblogsHtechnetHco%FbFexchangeFarchi$'F'6F'8Fintroducing-log-parser-studioHasp
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
54/119
C4 Troubleshooting Too
@>
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
55/119
Transport
Troubleshooting
Transport I Exchange ''6
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
56/119
Transport Exchange ''6logs
@A
Transport - Exchange
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
57/119
p g''8F'' logs
@8
Transport - Exchange
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
58/119
p g''8F'' logs oectivit= lo!
Dogs MT2 connection acti$it o! the outbound %essage deli$e0ueues to the destination MB; ser$er7 s%art host7 or do%ain
isabled b de!ault
$rotocol lo! isabled b de!ault on all MT2 end and Recei$e connectors
Enabled or disabled on a per connector basis Protocol logginlevel%ust be con.gured to Verbosein order to record
Circular logging is used to li%it .le sie
There is a special7 in$isible7 intra-organiation end connector (!or 9:B 9:B7 9:B E/E7 9:B Exchange ''6 rela) !o
which we can enable protocol logging onl $ia 2owerhell*Set-TransportServer server -!ntra"rgProtocol#ogging#evel Ver
@
Transport - Exchange logs
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
59/119
p g g
essa!e traci! lo!s Record detailed acti$it as the %essage%o$es within Exchange
Enabled b de!ault
On Exchange ''6 there is less in!or%atand the E$ent
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
60/119
p g g
essa!e traci! lo!s earch onl in graphical %ode in Exchan''6
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
61/119
p g g $ipelie Traci! 6Excha!e )((.
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
62/119
p
Lo!s(acti$ation i! needed) and
analsis Neto < 2iresharI %essage
routing7 latencies7 general
co%%unication5 TELNETon port @ to test MT2
co%%unication between e%ail ser$e Fueue ie@er(/et-3ueue powersh
co%%andlets) in case o! stuckA
Transport - Tools
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
63/119
p
$rocess Traci! Lo! 6$TL8 ,orks onl with Exchange ''8 and '
4llows parsing7 %onitoring and analingMessage Tracking logs
2ower!ul tool !or troubleshooting as well !or generating statistics !or %essage traX
A6
Transport - Tools
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
64/119
p
$rocess Traci! Lo! 6$TL8I use!
!or* Message Dooping
Message !ailures7 such as in eli$er tatus#oti.cations (#s)
Dist o! top %ail senders
Dist o! top %ail recipients
Top large %essage sie generators
3ueues backing up
2er!or%ance issues due to %essage loadA>
Transport - Tools
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
65/119
p
$rocess Traci! Lo! 6$TL8To parse one .le*
cscript 2rocessTrackingDogH$bs UC*J2rogra%"ilesJMicroso!tJExchangeer$erJ?>JTransportRolesJDogsJMessageTrackingJM/TR+x/U all
To parse all .les in each subdirector that werlogged esterda*
cscript 2rocessTrackingDogH$bs UC*J2rogra%"ilesJMicroso!tJExchangeer$erJ?>JTransportRolesJDogsJMessageTrackingU ' allesterda
Read %ore about it `http*FFblogsHtechnetHco%FbFexchangeFarchi$eF'F'FFupdated-process-tracking-log-ptl-tool-!or-use-with-exchange-''8-and-excha
A@
Transport I 2TD exa%ples
http://blogs.technet.com/b/exchange/archive/2011/10/21/updated-process-tracking-log-ptl-tool-for-use-with-exchange-2007-and-exchange-2010.aspxhttp://blogs.technet.com/b/exchange/archive/2011/10/21/updated-process-tracking-log-ptl-tool-for-use-with-exchange-2007-and-exchange-2010.aspx -
7/26/2019 Exchange Troubleshooting and Reporting Basics1
66/119
MTsn"ailureResultsHcs$
MTRecipienttatisticsHcs$
AA
Transport I Exa%ple
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
67/119
ISSUE:
4!ter %igrating around 8'' users toExchange ''7 there are constantl>''-@'' %essages in the E''6-E'
routing group 0ueue o! the Exchange''6 bridgeheadH
A8
Transport I Exa%ple
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
68/119
TROUBLESHOOTING: Exchange ''6 bridgehead*
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
69/119
&N&LDSIS:
Get-Receiveoector -server HUB J Ka7e07axac0tarpit
Name : Default ****E0024
MaxAcknowledgementDelay : 00:00:30
Tarpt!nter"al : 00:00:0#AG
Transport I Exa%ple
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
70/119
EA$L&N&TION:
The MaxAcknowledgementDelaypara%eter speci.es the period the transer$er delas acknowledge%ent when recei$ing %essages !ro% a host thatdoesnYt support shadow redundancH ,hen recei$ing %essages !ro% a host doesnYt support shadow redundanc7 a Microso!t Exchange er$er '' transer$er delas issuing an acknowledge%ent until it $eri.es that the %essage
been success!ull deli$ered to all recipientsH 9owe$er7 i! it takes too long to success!ul deli$er7 the transport ser$er ti%es out and issues anacknowledge%ent anwaH The de!ault $alue is 6' secondsH
The TarpitIntervalpara%eter speci.es the period o! ti%e to dela an MT2response to a re%ote ser$er that Exchange deter%ines %a be abusing theconnectionH 4uthenticated connections are ne$er delaed in this %annerH Thde!ault $alue is @ secondsH To speci! a $alue7 enter the $alue as a ti%e spanhh7''7ss7 where hQ hours7 'Q %inutes7 and sQ secondsH The $alid input 8'
Transport I Exa%ple
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
71/119
SOLUTION:
On all the 9:B ser$ers in theorganiation7 deacti$ated Tarpit!nterva& MaxAc1no*le(ge'ent8ela&7
Set-Receiveoector -I4etit= +e3ault HUBM -TarpitIterval ((:((:((
Set-Receiveoector -I4etit= +e3ault HUBM -ax&co@le4!e7etela((:((:((
icroso3t Excha!e Trasportservice ee4s top 5e restarte4 to8
Transport I Exa%ple
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
72/119
ISSUE:The Exchange '' E/E ser$er
cannot send e%ails to a certain do%I contosoHco%H
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
73/119
TROUBLESHOOTING: 4cti$ated 2rotocol logging !or the M
send connectorTook a network trace on the Edge
ser$er Reproduced the issue Collected the traces
86
Transport I Exa%ple
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
74/119
&N&LDSIS: :sed
wwwHhscriptsHco%FtoolsF9#TFdns-rerdHphp
to resol$e the
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
75/119
&N&LDSIS:
MT2 log*
B looking at the log7 we can onl sathat we !ail when we contactcontosoHco% in
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
76/119
&N&LDSIS a4 EA$L&N&TION:
#etwork trace on the Edge ser$er*
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
77/119
SOLUTION: 2rioritie co%%unication o$er
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
78/119
ISSUE: o%e users are recei$ing e%ails with
the bod in non-European charactersThese e%ails are co%ing !ro% outsid
the Exchange organiation
8
Transport I Exa%ple 6
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
79/119
TROUBLESHOOTING: 4sked !or a %essage in 2TFM/ !or%at
&N&LDSIS:The %essage is %ultipart
#oticed that the onl part o! the e%ail beidisplaed in European characters was thedisclai%er*
8G
Transport I Exa%ple 6
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
80/119
EA$L&N&TION:This can happen i! the charset o! the
%essage bod is di1erent !ro% the charseo! the disclai%er
Exchange cannot include both and needs
select one (all Exchange $ersions ha$e thesa%e beha$ior* http*FFsupportH%icroso!tHco%FkbFGAGG)
Bad luck* Exchange chooses the charset o
the disclai%er (ut!-)7 displaing it correctbut arblin the %essa e bod b doin s'
Transport I Exa%ple 6
http://support.microsoft.com/kb/916299http://support.microsoft.com/kb/916299http://support.microsoft.com/kb/916299http://support.microsoft.com/kb/916299http://support.microsoft.com/kb/916299 -
7/26/2019 Exchange Troubleshooting and Reporting Basics1
81/119
1URTHER TROUBLESHOOTING: 4sked !or pipeline tracing to be
acti$ated !or an external user /ot the result and at that %o%ent it
beca%e clear that the disclai%er wathe cause* E%ail bod !or all %essage snapshots prto the disclai%er being applied is displa
correctl
Transport I Exa%ple 6
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
82/119
SOLUTION: id !urther in$estigations on the
disclai%erTurned out that it was con.gured
incorrectl7 since it was getting appli!or %essges co%ing !ro% externalrecipients
Transport I Exa%ple >
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
83/119
ISSUE:The inheritance was cut in 4 !or one
O: 4s a result7 e%ails sent to those user
were 0ueuing up in the :nreachable3ueue on the Exchange ''8 9:Bser$er
6
Transport I Exa%ple >
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
84/119
SOLUTION: 2ut back the inheritance#ew e%ails are deli$ered to users
The :nreachable 0ueue is resub%itte
auto%aticall onl i! the old routingtables are not the sa%e as the old on Resub%it the %essages still blocked
the :nreachable 0ueue to thecategorier*
- - " " ->
Transport I Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
85/119
ISSUE: Root cause analsis (Exchange ''6 i1erent change operations had bee
done o$er the weekend7 o$erlapping
On Monda7 the noticed that the0ueue KMessages awaiting directorlookupL was growing and growing
ituation got back to nor%al onTuesda @
Transport I Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
86/119
TO Os: "ind out as accuratel as possible wh
the issue started a%e re0uire%ent !or the end o! the
downti%e Explain what happened
A
Transport I Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
87/119
TROUBLESHOOTING:
tarted b looking into the e$ent log uring the downti%e7 Expert logging
!or the Transport co%ponents was no
acti$ated so in!or%ation was notenough 4sked !or the %essage tracking logs
be!ore7 during and a!ter the issue
8
Transport I Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
88/119
&N&LDSIS:
4s this is Exchange ''67 we ha$e
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
89/119
G
Transport I Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
90/119
G'
Transport I Exa%ple @
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
91/119
&N&LDSIS:
uring the downti%e7 we ha$ebetween ' and >' %essages beingsent per %inute
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
92/119
&N&LDSIS:
B adding
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
93/119
EA$L&N&TION:
The trigger was a prox change onaturda
"ollowing this change7 the 4nti$irus
could no longer contact his updateser$er !or updates
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
94/119
Mailbox
Troubleshooting
Mailbox
2 b bl th t iti t i
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
95/119
2robabl the %ost sensiti$e part sinc
this is where the databases reside Doosing data (%ails7 appoint%ents) i
probabl the ad%inistratorVs worstnight%are
%ain issues !or this section* is%ounted database
DogFatabase growth issues
G@
Mailbox I is%ounted BCheck the database status*
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
96/119
EE:T
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
97/119
EE:T
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
98/119
,hat #OT to do i! a database is dow
EE:T
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
99/119
/rowth stthing to do* use ExMon(Exchang
:ser Monitor I works with Exchange''67 ''8 & '') on the %ailboxser$er with the issue to see i! an useis causing the growth ort on C2: (\) colu%n and see i! thesa%e user(s) is alwas on top
ort b Btes Out and check !or top talke
Mailbox I DogFatabase/rowth Be!ore %o$ing !orward here is an ExMon
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
100/119
/rowth Be!ore %o$ing !orward7 here is an ExMonexa%ple*
The user who causes the issue on the ser$er(transaction log growth) is the one consu%ing the%ost C2:H
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
101/119
/rowth
,ireshark) to identi! the client
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
102/119
/rowth
Exchange ''8F''* run the !ollowing 2owerhell
co%%and*/et-Dogontatistics select-obectuserna%e ,indows'''account identit %essa in o erationcount othero
'
Mailbox I DogFatabase/rowth 9ere is an exa%ple o! how operation rates can he
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
103/119
/ o9ere is an exa%ple o! how operation rates can hethe account is ooding the WWW account with
%ails
:ser#a%e * ,indows'''4ccount * do%ainSJ@6 %ail stor%
:ser#a%e * blank,indows'''4ccount * do%ainSJ9:BS %achine account o! a 9:B
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
104/119
%ailbox ser$er did not help7 %o$e on the9:Bs to check !or* Messages 0ueued or in retr*/et-Exchangeer$er where H
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
105/119
till ha$ing the issue a!ter all that
troubleshooting onVt !orget $TL!or both Mailbox &
9:B= till no solution in radar Call support7 tell us what ou ha$e
alread tried so that we donVt ask !oragain
'@
Mailbox Checking and repairing database integrit
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
106/119
Checking and repairing database integrit
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
107/119
'8
Mailbox
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
108/119
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
109/119
pISSUE:
Exchange er$er '' Backup so!tware cannot purge
transaction logs
'G
Mailbox I One si%pleexa%ple5TROUBLESHOOTING:
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
110/119
p
'
Mailbox I One si%pleexa%ple5TROUBLESHOOTING:
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
111/119
pTROUBLESHOOTING:* The range o! log.les JJJ/DOB4DROOTJe$iceJ9arddisk?olu%ehadowCop>J!or reco$er to co%plete success!ullH
Mailbox I One si%pleexa%ple5SOLUTION:
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
112/119
SOLUTION:
,e %anaged to .nd %issing logs inolder .le-le$el backups
Restored the %issing transaction loginto the Exchange !older
Backup was able to purge the logsa!terwards
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
113/119
3 & 4
ExMon Tool !or gathering statistics about how the Outloo
li ki ! h E h
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
114/119
clients are %aking use o! the Exchange er$er
resources such as C2: and network bandwidth
B8C-4CE"-8G>4"8BplalangQen
>
ExMon2hat Exo ca 4o:
" h h th ti th ! CP
http://www.microsoft.com/downloads/details.aspx?familyid=9A49C22E-E0C7-4B7C-ACEF-729D48AF7BC9&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?familyid=9A49C22E-E0C7-4B7C-ACEF-729D48AF7BC9&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?familyid=9A49C22E-E0C7-4B7C-ACEF-729D48AF7BC9&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?familyid=9A49C22E-E0C7-4B7C-ACEF-729D48AF7BC9&displaylang=en -
7/26/2019 Exchange Troubleshooting and Reporting Basics1
115/119
"or each user7 show the consu%ption on the ser$er !or* CP
(is1: net*or1 tra;c Measure ser$er-side C2: latenc /i$e the !ollowing in!or%ation about the clients* /se
na'e: version: !P a((ress 2rocess data sent to the Exchange ser$er b M42< clients a
exposes the user experience b showing the actual R2C(network & ser$er) latenc
/i$e so%e data about 4cti$enc traXc
2hat Exo caot 4o: Measure MT27 O,47 4?7 2O2F
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
116/119
g
er$er :ser Monitor5 5.nish the installation
-
7/26/2019 Exchange Troubleshooting and Reporting Basics1
117/119
ti%e consu%ption on the ser$er7 or in tracing %ode7 to gstatistics
KDi$eL %ode is going to be used %ostl with s%all updateinter$als7 the %ini%u% being %inute
ExMon /:< to be used !or this tpe o! %onitoring
The %axi%u% snapshotting inter$al is 6' %inutes7 which iuse!ul !or doing captures o$er longer periods o! ti%e
Traces S 6' %inutes cannot be created !ro% /: