fmw 11g diagnostic framework
TRANSCRIPT
-
7/25/2019 FMW 11g Diagnostic Framework
1/38
FMW Diagnostic Framework
-
7/25/2019 FMW 11g Diagnostic Framework
2/38
22
Diagnostic Framework (DFW) Background
Database group has built a generic diagnosticframework to be adopted by all Oracle products
Primary goal of this framework is to reduce time toresolution of bugs/issues
Aids in detecting, diagnosing, and resolving problems F ! has adopted the concepts and parts of the
generic framework, and added "ava/F ! specifice#tensions
$argeted problems are critical errors caused by codebugs, metadata corruption, customer data corruption,deadlocked threads, and inconsistent state
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
3/38
**
DFW Goals
First+failure diagnosis imiting damage and interruptions after a problem is
detected -educing problem diagnostic and resolution time .implifying customer interaction with Oracle .upport
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
4/38
DFW Tools and Tec nologies
Automatic capture of diagnostic data upon first failure .tandardi0ed log formats Diagnostic rules
1ncident detection log filter 1ncident packaging service 1P.3 1ntegration with !eb ogic Diagnostics Framework
! DF3 4 .upport !orkbench 1.. viewer
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
5/38
55
DFW !co"e and rele#ance to la$ered "roducts
DF! is applicable for 6 Oracle !eb ogic .erver with "-F ayered Fusion iddleware products that run on top of
Oracle !eb ogic .erver such as 6 Oracle 71 Oracle .OA .uite Oracle 7P4 Process anager Oracle 7A
4tc8) %omponents such as O9. and !eb %ache are
currently not covered)
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
6/38
::
De%initions Pro&lem 6 A problem is a critical error in the product) 4ach problem
has a uni;ue problem id and a key)
< Problem key 6 A te#t string that includes an error code
Incident 6 An incident is a single occurrence of a problem)
< Incident package ' An incident package package3 is a collection ofdata about incidents for one or more problems)
< Incident Packing Service 6 4nables creating and manipulating anincident package)
< Incident flood control 6 A flood controlled incident is one which doesnot generate an incident dump to prevent information overload)
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
7/38==
De%initions (contd )
We& ogic Diagnostic Framework (W DF) ' ! DFconsists of a set of components that work together to collect, archive,and access diagnostic information about a !eb ogic .erver instanceand its applications)
Har#ester' 9arvester in the ! DF component responsible forcollecting metrics from specified bean attributes)
Watc es 6 A watch is a rule which monitors weblogic server runtime
beans or log messages for a specified condition)
*oti%ication 6Once the watch condition is satisfied an associatednotification can be raised using . $P, .> P, " ?, or " . )
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
8/38@@
De%initions (contd )
+utomatic Diagnostic ,e"ositor$ (+D,) 6 is a file+based repository for diagnostic data such as traces, dumps, the alertlog, health monitor reports, and more)
+D,-I + $he AD- %ommand 1nterpreter AD-%13 is a utility thatenables you to investigate problems, and package and upload first+failure diagnostic data to Oracle .upport, all within a command+lineenvironment) AD-%1 also enables you to view the names of the tracefiles in the AD-, and to view the alert log with ? tags stripped, withand without content filtering)
W !T ' DF! has ! .$ commands for ;uerying Problems/1ncidentsacross ! . servers, view incident dump files, create manual incidents,e#ecute dump manually etc
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
9/38
DFW -om"onents
+utomatic Diagnostic ,e"ositor$ .a#a Diagnostic Dum"s .a#a Diagnostics Data /0tractor
Diagnostic Framework M&eans Diagnostic Framework (+D,) -ommand ine
Inter%ace (- I)
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
10/38('('
+utomatic Diagnostic ,e"ositor$ 1 +D,
.tores diagnostic data trace, dumps, metadata3 for
all Oracle products inside a single file+systemdirectory hierarchy AD- data is highly structured
etadata e)g) problem/incidents3 stored in
relationsDumps for incidents are performed in separate files
4ach !eb ogic .erver stores diagnostic data insubdirectories of its own home directory within the
AD-
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
11/38((((
!am"le +D, Director$ !tructure %or We& ogic
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
12/38(2(2
+D, Home 2 +D, Base
+D, ome + is the root directory for all diagnosticdata for a particular instance of an Oracleproduct/component)
+D, Base + AD- homes for differentproducts/components are grouped together under thesame root directory called the AD- base)
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
13/38(*(*
.a#a Diagnostic Dum"s
Perform targeted diagnostic data collection .imple "ava interface 1mplemented by components and applications -egistered at runtime %an be run manually and automatically Out Of $he 7o# Diagnostic dumps include6
$hreads 4#ecution %onte#t all active ones3 Active 9$$P re;uests %lass histogram D . etrics ogs by 4%1D3 ogs by timestamp up to +/B 5 minutes3 ! DF .erver 1mage
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
14/38((
.a#a Diagnostic Dum"s la$out
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
An incident on !eb ogic .erver may include the
following dumps6>ame6 Cvm)threadsDump Files6 Cvm threads( i()dmp
>ame6 dms)metricsDump Files6 dms metrics2 i()dmp
>ame6 odl);uicktraceDump Files6 odl ;uicktrace* i()dmp
>ame6 odl)logsDump Arguments6 ecidE'c a5'bdcb e (e(6+2f2@ 25f6(*2@ 'c@fb*6+@'''+'''''''''''''a dDump Files6 odl logs i()dmp
>ame6 odl)logsDump Arguments6 timestampE(*(:=2@=5=' 'Dump Files6 odl logs5 i()dmp
>ame6 wls)imageDump Files6 diagnostic image Admin.erver 2'(( ' 22 2( 5 (=)0ip
-
7/25/2019 FMW 11g Diagnostic Framework
15/38(5(5
.a#a Diagnostics Data /0tractor (DD/)
Active in each !eb ogic .erver
-ule driven incident detection and creation 1ncident detection
"ava)util)logging log filter ! DF Diagnostic odule " ? notifications
1ncident creation Default set of dumps to e#ecute -ule based dump e#ecution
"ava AP1 for creating incidents athers incident meta data correlation keys, component info,
etc)3 Flood control .upport for creating manual incidents
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
16/38(:(:
.a#a Diagnostics Data /0tractor (-ontd )
Out Of $he 7o# rules detect 1>%1D4>$ 4--O- level logmessages
Out Of $he 7o# ! DF !atch/>otification %onfiguration6 Out of memory errors
Gncaught e#ceptions $hread deadlocks .tuck threads
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
17/38(=(=
Management Beans (MBeans) 3 .M4
DF! provides 7eans that can be used to configurethe Diagnostic Framework
%an configure an ! DF !atch and >otification tocreate an incident
4nable or disable the detection of incidents throughthe log files
4nable or disable flood control and setting parametersfor flood control
.i0e management to configure ma# si0e for allincidents
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
18/38
(@(@
Management Beans
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
19/38
((
+D,-I -ommand3 ine 5tilit$
AD- %ommand 1nterpreter
Gtility that enables users to investigate problems, andpackage
A list of common AD-%1 commands is included in asubse;uent slide)
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
20/38
2'2'
W !T and DFW
listProblems 3Problem 1d Problem Hey( 74A+('('2' I9$$PJ
list1ncidents idEK(L31ncident 1d Problem Hey 1ncident $ime2 74A+('('2' I9$$PJ Fri Feb 2: (*6 26'( PD$ 2'('( 74A+('('2' I9$$PJ $ue Feb 2* ':6(=6* PD$ 2'('
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
21/38
2(2(
W !T 2 DFW
show1ncident idEM(M31ncident 1d6 (Problem 1d6 (Problem Hey6 74A+('('2' I9$$PJ
1ncident $ime6 $ue Feb 2* ':6(=6* PD$ 2'('4rror essage 1d6 74A+('('2'4#ecution %onte#t6 ''''14#;GvyAhH75"N 4yf(AfdC:'''' iFlood %ontrolled6 false
Dump Files 6 dms ecidct#( i()dmp Cvm threads2 i()dmpdms metrics* i()dmp odl logs i()dmp odl logs5 i()dmpdiagnostic image Admin.erver 2'(' '2 2* ': (= 2)0ipreadme)t#t
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
22/38
2222
W !T 2 DFW
e#ecuteDump nameEMdms)metricsM,outputFileEM/tmp/dumpout)t#tM, idEM(M3
Dump file dms metrics( i()dmp added to incident (
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
Incident reation 6 og %ilter
-
7/25/2019 FMW 11g Diagnostic Framework
23/38
2*%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
Incident -reation 6 og %ilter
loada""lication,ules1register dum"s
. 5 17Dlogging 0ml
Incident Filter
create
log message
in#oke
DD/ loadDe%aultrules
,egister De%aultdum"s
-reate incident8
+D,
create incident
/0ecutedum"s
/0ecutedum"s
Write message
!er#er log
-
7/25/2019 FMW 11g Diagnostic Framework
24/38
22
Incident -reation Generated &$ W DF Watc*oti%ication
Fusion iddleware configures a ! DF Diagnosticsodule F !DF!
1t contains a set of !atch and >otification rules fordetecting a specific set of critical errors and creatingan incident for each occurrence of those errors)
7y default it contains the following watch conditions6 Deadlock .tuck$hread Gnchecked4#ception such as >ullPointer4#ception,
.tackOverflow4rror, or OutOf emory4rror %ustom !atch conditions can be added
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
25/38
2525
Incident -reation Generated &$ W DF Watc*oti%ication
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
Managed !er#er
Diagnostic,ules
Watc*oti%ication
W DF DiagnosticImage Bean
DFW Diagnostics Data /0tractor +D,
+""lication Dum" -om"onent Dum"
9
: ;
=
?
,egister
*oti%$
/0ecute /0ecute
In#oke
,egister
Write
Write
-reateIncident
@
-
7/25/2019 FMW 11g Diagnostic Framework
26/38
2:2:
Incident -reation *oti%ication &$ W DF Watc*oti%ication
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
27/38
2@2@
Incident Packaging !er#ice
Package and intelligently gather 1ncident data -elated incidents packaged based on6
4%1D $imestamp %omponent
Prompt the user to dump additional data 1P. actions3 1ntrusive dumps %onfiguration info $est cases
1nitiate and package -DA report
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
28/38
*'*'
/M !u""ort Work&enc
anage reported problems .ystem alerted or user+reported anage life+cycle and history of problems found .ingle target view, group view, stack view, enterprise view
Present patches and workarounds when available %reate .ervice -e;uests Packages data using 1P. Gpload incident information and test cases to Oracle
Provide guided resolution for non+bug problems Automatically track and synchroni0e .ervice -e;ueststatus from Oracle
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
Work&enc 3 Domain !ummar$
-
7/25/2019 FMW 11g Diagnostic Framework
29/38
*(
Work&enc 3 Domain !ummar$
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
Work&enc 3 Pro&lems
-
7/25/2019 FMW 11g Diagnostic Framework
30/38
*2
Work&enc 3 Pro&lems
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
W k& 3 P &l D il
-
7/25/2019 FMW 11g Diagnostic Framework
31/38
****
Work&enc 3 Pro&lem Details
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
Work&enc 3 Incident Details
-
7/25/2019 FMW 11g Diagnostic Framework
32/38
**
Work&enc 3 Incident Details
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
33/38
*5*5
Work&enc 3 Package
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
34/38
*:*:
+D,-I -ommands
%ommon AD-%1 commands include6
94 P .4$ 7A.4 .4$ 9O 4PA$9 .9O! A 4-$ .9O! P-O7 4 .9O! 1>%1D4>$ .9O! $-A%4F1 4 1P. %-4A$4 PA%HA 4 1P. 4>4-A$4 PA%HA 4 PG- 4
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
35/38
*=*=
Manual Incident creation and "ackaging
create1ncident message1dEM D.+5'5''M,descriptionEMsample incidentM31ncident 1d6 ( Problem 1d6 Problem Hey6 D.+5'5'' I A>GA J
1ncident $ime6 2*rd February 2'(' ((6556 5 $ 4rror essage 1d6 D.+5'5''Flood %ontrolled6 false
! 9O 4/wlserver (')*/server/adr/adrci .et AD- 7A.4 and AD- 9O 4PA$9 variables 1P. %-4A$4 PA%HA 4 1>%1D4>$ (
%reated package ( based on incident id (, correlation level typical 1P. 4>4-A$4 PA%HA 4 ( in /tmp
enerated package ( in file /tmp/74A**=!eb 2'(''22*(*2*(5 %O ()0ip,mode complete
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
36/38
*@*@
,D+ 2 DFW
,emote Diagnostic +ssistant is a command line diagnostic tool whichprovides a comprehensive picture of the middleware environment)
1f the diagnostic dump information captured in the incident package
proves to be insufficient for analy0ing the root cause, an -DA reportmay be generated as additional input for support)
%opyright & 2'(( Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
37/38
**
+dditional ,e%erences DFW inks
DF! ProCect page DF! .upport 1nternals DF! %ustomer Documentation DF! 1nternal Dump Documentation DF! Functional specification DF! !eb ogic .erver port functional specification DF! Developers guide DF! "avaDoc AD- %ustomer Documentation
FMW Diagnostics
Diagnostics .py Application F ! Diagnostics Ora$weet roup KfmwdiagL F ! Diagnostics %onnect roup ailing list6 fmwdiag ww
%opyright & 2'(' Oracle and/or its affiliates) All rights reserved)
-
7/25/2019 FMW 11g Diagnostic Framework
38/38