exchange troubleshooting and reporting basics1

Upload: scribdibrahim

Post on 13-Apr-2018

263 views

Category:

Documents


0 download

TRANSCRIPT

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    1/119

    Chalk & Talk OnlineExchange Troubleshooting andRporting basics (tools and howto)Mihai Bobu

    upport Engineer

    Microso!t "rance

    #o$e%ber nd

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    2/119

    Content

    Obecti$e* +now tools to troubleshoot and pro$ide

    reporting and how to use the% withExchange

    ,hat ou will .nd*

    /eneral presentation ,hen to use what

    Tools and techni0ues !or di1erent kind o!issues

    2ractical exa%ples 3&4

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    3/119

    To begin with5

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    4/119

    /eneral 2resentation

    Scope: Exchange ''67 ''8 & ' Structure: C47 9:B7 MB; peci.c troubleshooting tools and

    techni0ues !or each role as well asco%%on ones that can be used acros

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    5/119

    Exchange er$ices

    st

    thing to do when so%ething goeswrong*

    Check i! all Exchange ser$icescon.gured in 4uto%atic %ode are5 inser$ice ?er si%ple task which can sa$e a lo

    o! troubleshooting ti%e on occasions

    @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    6/119

    Exchange er$ices

    Exchange ''6*

    A

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    7/119

    Exchange er$ices

    Exchange ''8*

    8

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    8/119

    Exchange er$ices

    Exchange ''*

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    9/119

    Exchange er$ices

    %essage and check 4pplication andste% e$ent logs

    Man possible reasons wh restart

    !ails* Docal ser$er con.gurationG

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    10/119

    Exchange er$ices

    Tool that < .nd %ost use!ul in case @Haspx

    '

    http://technet.microsoft.com/en-us/sysinternals/bb896645.aspxhttp://technet.microsoft.com/en-us/sysinternals/bb896645.aspxhttp://technet.microsoft.com/en-us/sysinternals/bb896645.aspxhttp://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    11/119

    Exchange er$ices IExa%ple ISSUE:4!ter patching7 Outlook clients could no longconnect to one speci.c C4 ser$er

    TROUBLESHOOTING: Ran Test-ServiceHealthand noticed tha

    the R2C Client 4ccess ser$ice was not

    running

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    12/119

    Exchange er$ices IExa%ple TROUBLESHOOTING:Tried to start the RC4 ser$ice but it !ailed

    with*

    #othing %ore in the E$ent logs

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    13/119

    Exchange er$ices IExa%ple TROUBLESHOOTING: Ran 2rocMon and .ltered !or 2rocess and Thread

    4cti$it onl !or the RC4 executable in the .rstplace*

    6

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    14/119

    Exchange er$ices IExa%ple TROUBLESHOOTING: 4dded "ile 4cti$it to the $iew and started lookin

    !or Results that were not uccess

    ,ent into JBin !older

    to look !or the DD5

    The .le was no longerthere

    >

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    15/119

    Exchange er$ices IExa%ple SOLUTION: Cop the DD !ro% another C4 ser$

    (sa%e 2 and R: le$el)The ser$ice could be started a!terwaTurned out that there was a proble%

    during patching while upgrading thatspeci.c DD

    @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    16/119

    Exchange er$ices IExa%ple ISSUE:There was an Exchange ser$ice no longerstartingH

    "or this exa%ple7 < used the RC4 ser$ice to illustrate the

    that < saw although another Exchange ser$ice was in$ol$edthat ti%eH

    A

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    17/119

    Exchange er$ices IExa%ple TROUBLESHOOTING: 4gain7 nothing use!ul in the E$ent logs5 "iltering step b step in 2rocMon we hit th

    issue*

    8

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    18/119

    Exchange er$ices IExa%ple ecurit tea% haddone so%e

    rein!orce%ents*

    SOLUTION: /a$e back the

    appropriate per%issions

    to #etwork er$ice and

    started the ser$ice

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    19/119

    C4

    Troubleshooting

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    20/119

    C4

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    21/119

    C4

    C4 ser$ers ha$e gotten %oreintelligent and the handle di1erenttasks (O4B distribution7 %ailbox %o$etcH)

    till7 the .rst thing that co%es into o%ind when we think o! C4 is clientconnecti$it

    2las a decisi$e role in end userexperience

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    22/119

    C4

    ,hen discussing client connecti$it7there are %ain tpes o! connection M42< connections such as Outlook

    K,ebL connections such as O,47

    4cti$enc and all the other ,eb ser$icewhich are being handled b the

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    23/119

    C4 logging* RC4 The de!ault settings !or this logging are as !ollow

    and located in the .le calledMicrosoft.Exchange.RpcClientAccess.Service.exeng 2rotocol logging is enabled b de!ault

    e!ault !older setting* "%Excha!eIstallir%#Lo!!i!#R$

    liet &ccess#" Max sie in +B that a single log .le can grow to be!ore a new on

    generated* "'()*("

    Max sie in +B that the entire director o! logs can grow to be!othe oldest log is deleted* +'(*,./"

    Dength o! ti%e in hours a log will be kept be!ore being deleted*+.)("

    Dog tpe tags to be logged* oectiscoect0 Lo!o06

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    24/119

    C4 logging* RC4 Nou can change the de!ault $alues as needed

    The R2C Client 4ccess ser$ice %ust be restarted to pick up thischange The %ost use!ul changes are the e!ault "older and the Dog Tag

    e3ault 1ol4er Setti! Nou %a want to change the location where the log .les are stored i! ou want to

    the% to a di1erent dri$e etcHLo! Ta!s to 5e lo!!e4

    9ere ou can speci! %ore details to log our other options are*

    Rops 6re7ote operatios8 Rops option o1ers ou a top le$el %ention o! the operation being per!or%ed (Dog

    Open"older7 Release7 /etContentsTable7 CreateMessage)

    OperatioSpeci9c This option shows %ore detail !or each RO2 (!or exa%ple7 et2rops on CreateMes

    operation)

    Throttli!>

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    25/119

    C4 logging* RC4 This logging can be $er help!ul to $eri! i! a use

    has connected in the past at allH The ad%inistrato%ight recei$e a co%plaint b a user who sas theare not able to connect at allH ,ith protocol loggiturned on b de!ault7 the ad%inistrator can checwhen the last ti%e was the did success!ull

    connect to the ser$er or i! the user has e$erconnected be!ore

    E$en though this logging is designed to si%pl ke

    track o! who is connecting and disconnecting7 it cpro$ide $aluable troubleshooting in!or%ation@

    C l i C l

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    26/119

    C4 logging* RC4 Exa%ple

    9ere is a trace where :ser4 is creatina new %essage and sending the%essage to :serB*

    '-'>-'TA*'8*'H'AGP7AG7G>7FoQContosoFouQExchange4d%inistrati$e /roup("N

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    27/119

    C4 logging* RC4 Exa%ple

    9ere is a trace with throttlingin!or%ation !or :ser4*

    '-'>-'TG*>6*@@HG8P777FoQContosoFouQExchange

    4d%inistrati$e /roup("N

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    28/119

    C4 logging*

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    29/119

    C4* #TDM $s +erberos Sceario:Both Outlook 4nwhere and Outlook T

    clients are unable to connectH The get pro%pted%ultiple ti%es and still cannot connectH Theconnection status window %a show KConnecting

    $ossi5le as@er:B de!ault7 #etlogon onl allo concurrent 42< calls !or authentication re0uestsour clients are connecting with #TDM it is possibthat a large nu%ber o! users could cause theseauthentication re0uests to ti%eout i! the CVs areo$erloadedH

    2hat to 4o: 2rioritie +erberos authentication o$er #TDM

    Change the MaxConcurrentAPIpara%eter on Cs and G

    C4 E l

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    30/119

    C4 I Exa%ple

    ISSUE: o%e users are not able to access an

    o! the Exchange ,eb er$ices*4utodisco$er7 O,47 a$ailabilit7 O4B

    #ot all users are ha$ing this issue

    6'

    C4 E l

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    31/119

    C4 I Exa%ple

    TROUBLESHOOTING: Reproduced the issue with an a1ecte

    user I tried to access O,47 but goterror K>'' Bad re0uestL in the web

    browser Dooked into the

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    32/119

    C4 I Exa%ple

    TROUBLESHOOTING: #eed to see i! the re0uest %ade b the us

    actuall gets up to the C4 ser$er o7 got down one le$el and checked

    9TT2ERR logs Error >'' is $isible here with a clear indica

    to the issue*'-'-' *6A*@ WWHWWH'HA AA WWHWWH@AH >>9TT2FH 2OT Fautodisco$erFautodisco$erHx%l >'' -Re0uestDength -

    6

    C4 E l

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    33/119

    C4 I Exa%ple

    EA$L&N&TION: :ser %aking the re0uest has a large

    +erberos ticket7 as he is %e%ber in S''securit groups

    RESOLUTION: 4pplied the !ollowing +Bs*http*FFsupportH%icroso!tHco%FkbF>G6@>

    http*FFsupportH%icroso!tHco%FkbF''G>6

    2lease note that the regke66

    C4 E a%ple

    http://support.microsoft.com/kb/2491354http://support.microsoft.com/kb/2020943http://support.microsoft.com/kb/2020943http://support.microsoft.com/kb/2020943http://support.microsoft.com/kb/2491354http://support.microsoft.com/kb/2491354
  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    34/119

    C4 I Exa%ple

    ISSUE: Exchange er$er '' 2TM/ re$erse prox OO" does not work in Outlook and O,

    6>

    C4 Exa%ple

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    35/119

    C4 I Exa%ple

    TROUBLESHOOTING: Reproduced the issue !ro% the userVs

    workstation Dooked into the

    >'6H> translates to D re0uired

    4sked !or a test !ro% O,4 per!or%ed dire

    on the C4 to bpass TM/7 we got success

    /ET ec r!rQowa& QOr anie 4uto%aticRe liesHslab >>6 HH6@

    C4 Exa%ple

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    36/119

    C4 I Exa%ple

    EA$L&N&TION: D is acti$ated !or both O,4 and EC2 4uto%aticReplies de%and is !orwarded to

    2ort ' Q 9TT2

    The TM/ O,4 rule is con.gured to use 9Tnot 9TT2 !or the traXc between TM/ andExchange

    SOLUTION: isabled D !or O,4 and EC27 did

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    37/119

    C4 I Exa%ple 6

    ISSUE: Exchange er$er '' 9ardware Doad-balanced C4 arra Outlook '' O4B download is not possible

    68

    C4 Exa%ple 6

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    38/119

    C4 I Exa%ple 6

    TROUBLESHOOTING: Made an atte%pt to download the O4B Checked F'@F'6

    C4 Exa%ple 6

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    39/119

    C4 I Exa%ple 6

    TROUBLESHOOTING: "orced Outlook to connect directl to the

    C4 (C4

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    40/119

    C4 I Exa%ple >

    ISSUE: 4!ter %o$ing the %ailbox !ro%

    Exchange ''8 to Exchange ''7user cannot connect an%ore

    9e gets the !ollowing error*KCannot open our de!ault e-%ail !olders7 Microso!t Excha

    is not a$ailableH Either there are network proble%s or theExchange ser$er is down !or %aintenanceHL

    >'

    C4 Exa%ple >

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    41/119

    C4 I Exa%ple >

    TROUBLESHOOTING: Dooked into the RC4 logs and !ound the

    !ollowing*'-'@-'T*'G*8H@'8P7>@>7YFoQ;NPFouQ4BC-;;;FcnQPPPPPPFcnQ?????FcnQabc Q TTTTY77O:TDOO+HE;E7>H'H>8A'H'''7Classic777ncacniptcp777'xAB4(rpc**Exception)7''*''*''H'@A''7essionropped7RpcEnd2oint* Zer$er:na$ailableException[ Conn%ust be re-established -S ZessioneadException[ The pri%ar owner logon has !ailedH ropping a

    connectionH

    '-'@-'T*'G*8HP7>@@7'YFoQ;NPFouQ4BC-;;;FcnQPPPPPPFcnQ?????FcnQabc Q TTTTY77O:TDOO+HE;E7>H'H>8A'H'''7Classic7##H##H'HG7!e'**c>*@A>G*8cca*adG\7ncacniptcpect7'7''*''*''7U

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    42/119

    C4 I Exa%ple >

    EA$L&N&TION:

    C4 Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    43/119

    C4 I Exa%ple @

    ISSUE: Exchange ''8F'' %ixed

    en$iron%entTM/ publishing rules !or 4cti$enc :sers with a certain tpe o! %obile

    phone are co%plaining that !ro% ti%to ti%e their 6

    C4 Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    44/119

    C4 I Exa%ple @

    TROUBLESHOOTING: 2ushed 4cti$enc logging at %ax on

    Exchange '' C4 ser$ers 4sked users to call 9elp esk as soo

    as the hit the issue and %ention theti%e as accuratel as possible

    id that !or one da and then got the

    logs (E$ent logs7

    C4 I Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    45/119

    C4 I Exa%ple @

    &N&LDSIS: ,hen the users reported the issue7 we ha$

    ,arning '' in 4pplication E$ent logThis log alone does not tell us what happe

    it is known to gi$e !alse positi$es5

    C4 I Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    46/119

    C4 I Exa%ple @

    &N&LDSIS: ,hen we %atch the null\!\>null\!\@\c"C\6a'''\!'\c2olic\6ae!aultThrottling2olic\@"b'8Ga-eG'>->!e>

    aGb-88ec8dG!>e\c#or% >>6 COR2J^^^^ '@H6HGHG 4pple-i2hone6CFG'H'A '' ' A> 8>

    >A

    C4 I Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    47/119

    C4 I Exa%ple @

    &N&LDSIS: o7 the error is co%ing !ro% the 9TT2 lae Dooked at the 9TT2ERR logs and saw the

    !ollowing*

    Connectionropped MExchangenc4pp2ool ,hen the i2honeFi2ad recei$es two @''

    errors consecuti$el it resets the connectiand proceeds to re-download the userVs

    entire %ailboxThis could co%e either !ro% the C4 ser$e

    >8

    C4 I Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    48/119

    C4 Exa%ple @

    SOLUTION:

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    49/119

    C4 Exa%ple A

    ISSUE: Exchange er$er ''6 BE cluster

    hosting approxH 6''' %ailboxesThe ,6,2HE;E process runs at al%o

    ''\ C2:H 4s a conse0uence74cti$enc snchroniation is slow odoes not work at all

    >G

    C4 I Exa%ple A

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    50/119

    C4 Exa%ple A

    TROUBLESHOOTING: 4sked !or the

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    51/119

    C4 Exa%ple A

    TROUBLESHOOTING:

    SOLUTION:Mo$ed the top talkers to

    %ore power!ul Exchange '' plat!or@

    C4 I Troubleshooting Too

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    52/119

    C4 Troubleshooting Toohttp*FFblogsHtechnetHco%FbFexchangeFarchi$eF'F

    6Fa-script-to-troubleshoot-issues-with-exchange-ai$esncHaspx ?er use!ul !or parsing

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    53/119

    C4 Troubleshooting Too

    4 %ore ad$anced tool is Lo! $arser

    Stu4iohttp*FFblogsHtechnetHco%FbFexchangeFarchi$'F'6F'8Fintroducing-log-parser-studioHasp

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    54/119

    C4 Troubleshooting Too

    @>

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    55/119

    Transport

    Troubleshooting

    Transport I Exchange ''6

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    56/119

    Transport Exchange ''6logs

    @A

    Transport - Exchange

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    57/119

    p g''8F'' logs

    @8

    Transport - Exchange

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    58/119

    p g''8F'' logs oectivit= lo!

    Dogs MT2 connection acti$it o! the outbound %essage deli$e0ueues to the destination MB; ser$er7 s%art host7 or do%ain

    isabled b de!ault

    $rotocol lo! isabled b de!ault on all MT2 end and Recei$e connectors

    Enabled or disabled on a per connector basis Protocol logginlevel%ust be con.gured to Verbosein order to record

    Circular logging is used to li%it .le sie

    There is a special7 in$isible7 intra-organiation end connector (!or 9:B 9:B7 9:B E/E7 9:B Exchange ''6 rela) !o

    which we can enable protocol logging onl $ia 2owerhell*Set-TransportServer server -!ntra"rgProtocol#ogging#evel Ver

    @

    Transport - Exchange logs

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    59/119

    p g g

    essa!e traci! lo!s Record detailed acti$it as the %essage%o$es within Exchange

    Enabled b de!ault

    On Exchange ''6 there is less in!or%atand the E$ent

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    60/119

    p g g

    essa!e traci! lo!s earch onl in graphical %ode in Exchan''6

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    61/119

    p g g $ipelie Traci! 6Excha!e )((.

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    62/119

    p

    Lo!s(acti$ation i! needed) and

    analsis Neto < 2iresharI %essage

    routing7 latencies7 general

    co%%unication5 TELNETon port @ to test MT2

    co%%unication between e%ail ser$e Fueue ie@er(/et-3ueue powersh

    co%%andlets) in case o! stuckA

    Transport - Tools

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    63/119

    p

    $rocess Traci! Lo! 6$TL8 ,orks onl with Exchange ''8 and '

    4llows parsing7 %onitoring and analingMessage Tracking logs

    2ower!ul tool !or troubleshooting as well !or generating statistics !or %essage traX

    A6

    Transport - Tools

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    64/119

    p

    $rocess Traci! Lo! 6$TL8I use!

    !or* Message Dooping

    Message !ailures7 such as in eli$er tatus#oti.cations (#s)

    Dist o! top %ail senders

    Dist o! top %ail recipients

    Top large %essage sie generators

    3ueues backing up

    2er!or%ance issues due to %essage loadA>

    Transport - Tools

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    65/119

    p

    $rocess Traci! Lo! 6$TL8To parse one .le*

    cscript 2rocessTrackingDogH$bs UC*J2rogra%"ilesJMicroso!tJExchangeer$erJ?>JTransportRolesJDogsJMessageTrackingJM/TR+x/U all

    To parse all .les in each subdirector that werlogged esterda*

    cscript 2rocessTrackingDogH$bs UC*J2rogra%"ilesJMicroso!tJExchangeer$erJ?>JTransportRolesJDogsJMessageTrackingU ' allesterda

    Read %ore about it `http*FFblogsHtechnetHco%FbFexchangeFarchi$eF'F'FFupdated-process-tracking-log-ptl-tool-!or-use-with-exchange-''8-and-excha

    A@

    Transport I 2TD exa%ples

    http://blogs.technet.com/b/exchange/archive/2011/10/21/updated-process-tracking-log-ptl-tool-for-use-with-exchange-2007-and-exchange-2010.aspxhttp://blogs.technet.com/b/exchange/archive/2011/10/21/updated-process-tracking-log-ptl-tool-for-use-with-exchange-2007-and-exchange-2010.aspx
  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    66/119

    MTsn"ailureResultsHcs$

    MTRecipienttatisticsHcs$

    AA

    Transport I Exa%ple

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    67/119

    ISSUE:

    4!ter %igrating around 8'' users toExchange ''7 there are constantl>''-@'' %essages in the E''6-E'

    routing group 0ueue o! the Exchange''6 bridgeheadH

    A8

    Transport I Exa%ple

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    68/119

    TROUBLESHOOTING: Exchange ''6 bridgehead*

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    69/119

    &N&LDSIS:

    Get-Receiveoector -server HUB J Ka7e07axac0tarpit

    Name : Default ****E0024

    MaxAcknowledgementDelay : 00:00:30

    Tarpt!nter"al : 00:00:0#AG

    Transport I Exa%ple

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    70/119

    EA$L&N&TION:

    The MaxAcknowledgementDelaypara%eter speci.es the period the transer$er delas acknowledge%ent when recei$ing %essages !ro% a host thatdoesnYt support shadow redundancH ,hen recei$ing %essages !ro% a host doesnYt support shadow redundanc7 a Microso!t Exchange er$er '' transer$er delas issuing an acknowledge%ent until it $eri.es that the %essage

    been success!ull deli$ered to all recipientsH 9owe$er7 i! it takes too long to success!ul deli$er7 the transport ser$er ti%es out and issues anacknowledge%ent anwaH The de!ault $alue is 6' secondsH

    The TarpitIntervalpara%eter speci.es the period o! ti%e to dela an MT2response to a re%ote ser$er that Exchange deter%ines %a be abusing theconnectionH 4uthenticated connections are ne$er delaed in this %annerH Thde!ault $alue is @ secondsH To speci! a $alue7 enter the $alue as a ti%e spanhh7''7ss7 where hQ hours7 'Q %inutes7 and sQ secondsH The $alid input 8'

    Transport I Exa%ple

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    71/119

    SOLUTION:

    On all the 9:B ser$ers in theorganiation7 deacti$ated Tarpit!nterva& MaxAc1no*le(ge'ent8ela&7

    Set-Receiveoector -I4etit= +e3ault HUBM -TarpitIterval ((:((:((

    Set-Receiveoector -I4etit= +e3ault HUBM -ax&co@le4!e7etela((:((:((

    icroso3t Excha!e Trasportservice ee4s top 5e restarte4 to8

    Transport I Exa%ple

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    72/119

    ISSUE:The Exchange '' E/E ser$er

    cannot send e%ails to a certain do%I contosoHco%H

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    73/119

    TROUBLESHOOTING: 4cti$ated 2rotocol logging !or the M

    send connectorTook a network trace on the Edge

    ser$er Reproduced the issue Collected the traces

    86

    Transport I Exa%ple

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    74/119

    &N&LDSIS: :sed

    wwwHhscriptsHco%FtoolsF9#TFdns-rerdHphp

    to resol$e the

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    75/119

    &N&LDSIS:

    MT2 log*

    B looking at the log7 we can onl sathat we !ail when we contactcontosoHco% in

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    76/119

    &N&LDSIS a4 EA$L&N&TION:

    #etwork trace on the Edge ser$er*

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    77/119

    SOLUTION: 2rioritie co%%unication o$er

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    78/119

    ISSUE: o%e users are recei$ing e%ails with

    the bod in non-European charactersThese e%ails are co%ing !ro% outsid

    the Exchange organiation

    8

    Transport I Exa%ple 6

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    79/119

    TROUBLESHOOTING: 4sked !or a %essage in 2TFM/ !or%at

    &N&LDSIS:The %essage is %ultipart

    #oticed that the onl part o! the e%ail beidisplaed in European characters was thedisclai%er*

    8G

    Transport I Exa%ple 6

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    80/119

    EA$L&N&TION:This can happen i! the charset o! the

    %essage bod is di1erent !ro% the charseo! the disclai%er

    Exchange cannot include both and needs

    select one (all Exchange $ersions ha$e thesa%e beha$ior* http*FFsupportH%icroso!tHco%FkbFGAGG)

    Bad luck* Exchange chooses the charset o

    the disclai%er (ut!-)7 displaing it correctbut arblin the %essa e bod b doin s'

    Transport I Exa%ple 6

    http://support.microsoft.com/kb/916299http://support.microsoft.com/kb/916299http://support.microsoft.com/kb/916299http://support.microsoft.com/kb/916299http://support.microsoft.com/kb/916299
  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    81/119

    1URTHER TROUBLESHOOTING: 4sked !or pipeline tracing to be

    acti$ated !or an external user /ot the result and at that %o%ent it

    beca%e clear that the disclai%er wathe cause* E%ail bod !or all %essage snapshots prto the disclai%er being applied is displa

    correctl

    Transport I Exa%ple 6

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    82/119

    SOLUTION: id !urther in$estigations on the

    disclai%erTurned out that it was con.gured

    incorrectl7 since it was getting appli!or %essges co%ing !ro% externalrecipients

    Transport I Exa%ple >

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    83/119

    ISSUE:The inheritance was cut in 4 !or one

    O: 4s a result7 e%ails sent to those user

    were 0ueuing up in the :nreachable3ueue on the Exchange ''8 9:Bser$er

    6

    Transport I Exa%ple >

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    84/119

    SOLUTION: 2ut back the inheritance#ew e%ails are deli$ered to users

    The :nreachable 0ueue is resub%itte

    auto%aticall onl i! the old routingtables are not the sa%e as the old on Resub%it the %essages still blocked

    the :nreachable 0ueue to thecategorier*

    - - " " ->

    Transport I Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    85/119

    ISSUE: Root cause analsis (Exchange ''6 i1erent change operations had bee

    done o$er the weekend7 o$erlapping

    On Monda7 the noticed that the0ueue KMessages awaiting directorlookupL was growing and growing

    ituation got back to nor%al onTuesda @

    Transport I Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    86/119

    TO Os: "ind out as accuratel as possible wh

    the issue started a%e re0uire%ent !or the end o! the

    downti%e Explain what happened

    A

    Transport I Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    87/119

    TROUBLESHOOTING:

    tarted b looking into the e$ent log uring the downti%e7 Expert logging

    !or the Transport co%ponents was no

    acti$ated so in!or%ation was notenough 4sked !or the %essage tracking logs

    be!ore7 during and a!ter the issue

    8

    Transport I Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    88/119

    &N&LDSIS:

    4s this is Exchange ''67 we ha$e

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    89/119

    G

    Transport I Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    90/119

    G'

    Transport I Exa%ple @

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    91/119

    &N&LDSIS:

    uring the downti%e7 we ha$ebetween ' and >' %essages beingsent per %inute

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    92/119

    &N&LDSIS:

    B adding

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    93/119

    EA$L&N&TION:

    The trigger was a prox change onaturda

    "ollowing this change7 the 4nti$irus

    could no longer contact his updateser$er !or updates

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    94/119

    Mailbox

    Troubleshooting

    Mailbox

    2 b bl th t iti t i

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    95/119

    2robabl the %ost sensiti$e part sinc

    this is where the databases reside Doosing data (%ails7 appoint%ents) i

    probabl the ad%inistratorVs worstnight%are

    %ain issues !or this section* is%ounted database

    DogFatabase growth issues

    G@

    Mailbox I is%ounted BCheck the database status*

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    96/119

    EE:T

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    97/119

    EE:T

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    98/119

    ,hat #OT to do i! a database is dow

    EE:T

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    99/119

    /rowth stthing to do* use ExMon(Exchang

    :ser Monitor I works with Exchange''67 ''8 & '') on the %ailboxser$er with the issue to see i! an useis causing the growth ort on C2: (\) colu%n and see i! thesa%e user(s) is alwas on top

    ort b Btes Out and check !or top talke

    Mailbox I DogFatabase/rowth Be!ore %o$ing !orward here is an ExMon

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    100/119

    /rowth Be!ore %o$ing !orward7 here is an ExMonexa%ple*

    The user who causes the issue on the ser$er(transaction log growth) is the one consu%ing the%ost C2:H

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    101/119

    /rowth

    ,ireshark) to identi! the client

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    102/119

    /rowth

    Exchange ''8F''* run the !ollowing 2owerhell

    co%%and*/et-Dogontatistics select-obectuserna%e ,indows'''account identit %essa in o erationcount othero

    '

    Mailbox I DogFatabase/rowth 9ere is an exa%ple o! how operation rates can he

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    103/119

    / o9ere is an exa%ple o! how operation rates can hethe account is ooding the WWW account with

    %ails

    :ser#a%e * ,indows'''4ccount * do%ainSJ@6 %ail stor%

    :ser#a%e * blank,indows'''4ccount * do%ainSJ9:BS %achine account o! a 9:B

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    104/119

    %ailbox ser$er did not help7 %o$e on the9:Bs to check !or* Messages 0ueued or in retr*/et-Exchangeer$er where H

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    105/119

    till ha$ing the issue a!ter all that

    troubleshooting onVt !orget $TL!or both Mailbox &

    9:B= till no solution in radar Call support7 tell us what ou ha$e

    alread tried so that we donVt ask !oragain

    '@

    Mailbox Checking and repairing database integrit

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    106/119

    Checking and repairing database integrit

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    107/119

    '8

    Mailbox

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    108/119

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    109/119

    pISSUE:

    Exchange er$er '' Backup so!tware cannot purge

    transaction logs

    'G

    Mailbox I One si%pleexa%ple5TROUBLESHOOTING:

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    110/119

    p

    '

    Mailbox I One si%pleexa%ple5TROUBLESHOOTING:

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    111/119

    pTROUBLESHOOTING:* The range o! log.les JJJ/DOB4DROOTJe$iceJ9arddisk?olu%ehadowCop>J!or reco$er to co%plete success!ullH

    Mailbox I One si%pleexa%ple5SOLUTION:

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    112/119

    SOLUTION:

    ,e %anaged to .nd %issing logs inolder .le-le$el backups

    Restored the %issing transaction loginto the Exchange !older

    Backup was able to purge the logsa!terwards

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    113/119

    3 & 4

    ExMon Tool !or gathering statistics about how the Outloo

    li ki ! h E h

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    114/119

    clients are %aking use o! the Exchange er$er

    resources such as C2: and network bandwidth

    B8C-4CE"-8G>4"8BplalangQen

    >

    ExMon2hat Exo ca 4o:

    " h h th ti th ! CP

    http://www.microsoft.com/downloads/details.aspx?familyid=9A49C22E-E0C7-4B7C-ACEF-729D48AF7BC9&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?familyid=9A49C22E-E0C7-4B7C-ACEF-729D48AF7BC9&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?familyid=9A49C22E-E0C7-4B7C-ACEF-729D48AF7BC9&displaylang=enhttp://www.microsoft.com/downloads/details.aspx?familyid=9A49C22E-E0C7-4B7C-ACEF-729D48AF7BC9&displaylang=en
  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    115/119

    "or each user7 show the consu%ption on the ser$er !or* CP

    (is1: net*or1 tra;c Measure ser$er-side C2: latenc /i$e the !ollowing in!or%ation about the clients* /se

    na'e: version: !P a((ress 2rocess data sent to the Exchange ser$er b M42< clients a

    exposes the user experience b showing the actual R2C(network & ser$er) latenc

    /i$e so%e data about 4cti$enc traXc

    2hat Exo caot 4o: Measure MT27 O,47 4?7 2O2F

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    116/119

    g

    er$er :ser Monitor5 5.nish the installation

  • 7/26/2019 Exchange Troubleshooting and Reporting Basics1

    117/119

    ti%e consu%ption on the ser$er7 or in tracing %ode7 to gstatistics

    KDi$eL %ode is going to be used %ostl with s%all updateinter$als7 the %ini%u% being %inute

    ExMon /:< to be used !or this tpe o! %onitoring

    The %axi%u% snapshotting inter$al is 6' %inutes7 which iuse!ul !or doing captures o$er longer periods o! ti%e

    Traces S 6' %inutes cannot be created !ro% /: