-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
1/41
IBM Software Group
IBM Corporation
Tivoli Access Manager problem determination
using logging and tracing features
Jenny Totterdel l - EMEA Securi ty Suppo rt jen ny_to tter dell@u k.ibm .com
mailto:[email protected]:[email protected] -
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
2/41
IBM Software Group | Tivoli software
Topics covered in this workshop
Installation and Configuration Logs
Serviceability logs
Trace Logging
WebSEAL HTTP Trace Logging
Debugging Java Runtime Issues
GSKit Traces
Must Gather Information for Support
Capturing Core Files
System_status script
Question/Answer Session
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
3/41
IBM Software Group | Tivoli software
Log Files
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
4/41
IBM Software Group | Tivoli software
Installation Logs If the easy installation programs are used, the log files are written to the temp directory
Windows - %TEMP% (e.g. C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp)
UNIX - typically /tmp or /var/tmp
Component Installation log file name
Policy server msg__ammgr_install.log
Policy proxy server msg__amproxy_install.log
Authorization server msg__amacld_install.log
Runtime msg__amrte_install.log
Java runtime msg__amjrte_install.log
ADK msg__amadk_install.log
Web Portal Manager msg__amwpm_install.log
WebSEAL msg__amweb_install.log
WebSEAL Application Development Kit msg__amwebadk_install.log
Plug-in for Web Servers msg__amwpiismp_install.log
WebSphere Application Svr & BEA WebLogic Svr integration support msg__amismp.log
Attribute retrieval service msg__amars_install.log
Tivoli Identity Manager Provisioning Fast Start msg__ampfs_install.log
IBM Tivoli Directory Server msg__ldaps_install.log
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
5/41
IBM Software Group | Tivoli software
Configuration Logs
Messages generated during the configuration process are stored
within Tivoli Access Manager configuration log files.
Component Installation log file name
Base msg__config.logWeb Portal manager msg__amwpmcfg.log & amwpmcfg1.log
Java runtime environment msg__PDJrteCfg1.log
WebSEAL msg__amweb_config.log
Plug-in for Web Servers msg__pdwpicfg.log
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
6/41
IBM Software Group | Tivoli software
Serviceability Logs
Examples of serviceability logs: msg__pdmgrd_utf8.log
msg__webseald-default.log
Message logging (i.e. Error/Warning/Informational logging) is enabled by default
Default log locations
UNIX: /var/PolicyDirector/log/
Windows:pd_dir \log\
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
7/41
IBM Software Group | Tivoli software
Message Format
A message consists of:
Date
Message Number (unique 32-bit decimal or hexadecimal value)
Process Name
Priority (e.g. WARNING)
Component information (including file name)
A message identifier (ID) and message text.
Example of a failed login captured in WebSEAL server message log
(/var/pdweb/log/msg__webseald-default.log):
2005-07-20-05:54:36.655+00:00I----- 0x132120DDwebsealdWARNING ias authsvc pdauthn.cpp 1435 0x00002526
HPDIA0221W Authentication for user testuser failed. Youhave used an invalid user name, password or clientcertificate
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
8/41
IBM Software Group | Tivoli software
Types of Messages
Notice (Notice_verbose)
Does not directly require action, such as information about running state
Warning
Results may not be as desired but the program continues to function
normally.
Error
The product continues to function, but some services or functionality might
not be available
Fatal
Unrecoverable error, the process encountering the error usually terminates
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
9/41
IBM Software Group | Tivoli software
Message Examples:
Notices:
2005-08-09-09:07:31.814+00:00I----- 0x1354A0A0 pdmgrd NOTICE ivc generalivmgrd.cpp 743 0x00000001 Server startup
Server startup message
Warning:
2003-10-31-23:09:45.457+00:00I----- 0x38CF0131 webseald WARNING wwa server listen-ssl.c 167 0x00000044 The 'ssl_writechunk' routine failed for 'gsk_secure_soc_write', errno
= 406 This error is common and normal for webseal and ssl, which is why they're reported as warnings.
Mainly they are due to network connectivity or the customer hitting the "stop" button on their browsers.The reason you see several messages with the same timestamp is because the browsers tend to openmultiple simultaneous connections. Losing the network or hitting the "stop" button will cancel allsimultaneous connections.
406 is a GSKit return code GSK_ERROR_IO
Error:
2003-07-08-12:59:07.032+00:00I----- 0x1354A0B6 pdmgrd ERROR ivc generalLDAPClient.cpp 212 0x00000001 LDAP initialization failed: ira_rgy_init('tarsus', 636,'cn=ivmgrd/master,cn=SecurityDaemons,secAuthority=Default', ***) = 113, 202
Connection to LDAP failed.
Fatal:
2004-12-09-14:42:32.391+01:00I----- 0x14C010A4 pdmgrd FATAL mgr generale:\am510\src\ivmgrd\ivmgrd.cpp 252 0x00000ba4 HPDMG0164E The Policy Server couldnot be started (0x14c01420).
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
10/41
IBM Software Group | Tivoli software
Message ID FormatThe message ID consists of 10 alphanumeric characters, where the sequence is
XXXYY####Z:
XXX is the product identifier, including the following product codes:
Code Subsystem
HPD Base
DPW/HPW WebSEAL
AWD Plug-in for IBM WebSphere Edge Server
AWL BEA WebLogic Server integration
AWX WebSphere Application Server Integration
AMZ Plug-in for Web Servers
YY is the subsystem code
#### is a unique message id.Z is the severity code indicator, including the following indicators:
Severity Code Description
I Informational message.
W Warning message.
E Error message.
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
11/41
IBM Software Group | Tivoli software
Routing Message Logging
Routing of serviceability messages is controlled by the routing file.
The contents of the routing file enables control of
Whether message logging is on or off for each class of messages (FATAL, ERROR,
WARNING, NOTICE, or NOTICE_VERBOSE)
Where the message log output for each class of messages is to be directed
If message output is being directed to a file, how many files for each class of messages
should be used, and how many messages should be placed in each file
The routing files for each component are
pdmgrd_routing for the Policy Server
pdacld_routing for the Authorization Server
pdmgrproxyd_routing for the Policy Proxy Server
/opt/pdweb/etc/routing for WebSEAL
/opt/PolicyDirector/etc/routing for Runtime
PDJlog.properties for Java apps
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
12/41
IBM Software Group | Tivoli software
Routing File Message Logging Entry
The format of a routing file entry that controls message logging is:
severity:destination:location {[;destination:location] ...}
[;GOESTO:other_severity]
Default configuration for FATAL and ERROR messages:
Unix FATAL:STDOUT:-
;UTF8FILE:/var/PolicyDirector/log/msg__pdmgrd_utf8.log:644:ivmgr:ivmgr
ERROR:STDOUT:-
;UTF8FILE:/var/PolicyDirector/log/msg__pdmgrd_utf8.log:644:ivmgr:ivmgr
Windows:
FATAL:STDERR:-;FILE:C:/PROGRA~1/Tivoli/POLICY~1/log/msg__fatal.log
ERROR:STDERR:-;FILE:C:/PROGRA~1/Tivoli/POLICY~1/log/msg__error.log
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
13/41
IBM Software Group | Tivoli software
WebSEAL Logs
WebSEAL maintains three conventional HTTP log files that record
activity rather than messages:
request.log
logs HTTP requests, such as information on URLs that have been
requested and information on the client (e.g. IP address).
agent.log records contents of the User_Agent:header in the HTTP request. Includes
data about the client browser, such as architecture or version number
referer.log
records the Referer:header of the HTTP request. Records the document
that contained the link to the requested document.
By default, these log files are located under the following directory:
UNIX: /var/pdweb/www/log/
Windows: C:\Program Files\Tivoli\PDWeb\www\log\
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
14/41
IBM Software Group | Tivoli software
Request.log
Every response sent back by TAM is recorded with a one-line entry inthe request.log
Format: host - authuser [date] request status bytes
host Specifies the IP address of the requesting machine.
authuserIdentity information of the user. The value unauth is used for an
unauthenticated user.dateSpecifies the date and time of the request.
request Specifies the first line of the request as it came from the client.
statusSpecifies the HTTP status code sent back to the requesting machine.
bytesSpecifies the number of bytes sent back to the requesting machine.
130.15.1.90- lmalone [30/Aug/2005: 10:24:11 +0100]"GET /jct/images/IBMLogo.gif HTTP/1.1" 2001979
130.15.1.90- lmalone[30/Aug/2005: 10:24:13 +0100]"GET /jct/images/IBMLogo.gif HTTP/1.1" 3040
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
15/41
IBM Software Group | Tivoli software
Traces
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
16/41
IBM Software Group | Tivoli software
Trace Logging
Unlike message logging, trace logging (or tracing) is not enabled bydefault.
Enabled using routing/properties files or pdadmin
Useful for
Recreateable problems
Issues short lived in duration
Blade Startup Failures (including during configuration)
Checking LDAP Return Codes
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
17/41
IBM Software Group | Tivoli software
Enabling TracingRouting File
Can trace all components, or limit the scope
General format for routing file tracing statement:
component :subcomponent.debuglevel:destination :attributes
Examples:
Entries in /opt/PolicyDirector/etc/pdmgrd_routing (TAM 5.1)Trace all components for the Policy Server at highest trace level
*:*.9:TEXTFILE.10.10000:/var/PolicyDirector/log/trace__%ld.log
Trace the Policy Server's LDAP client calls/LDAP Server return codes
ivc:ira.9:TEXTFILE.10.10000:/var/PolicyDirector/log/trace__
pdmgrd_ira.log
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
18/41
IBM Software Group | Tivoli software
Enabling TracingTrace Command
Can be activated dynamically using the command:
pdadmin> server task server_name trace set component level
List possible dynamic trace points:
pdadmin sec_master> server task server_name trace list
Particularly useful pdadmin traces:
pdweb.debug, pdweb.snoop, pd.ivc.ira
Examples:
pdadmin> server task webseald-instance trace set pdweb.debug 2
file path=/tmp/pdweb.debug.out
pdadmin> server task webseald-instance trace show
pdweb.debug 2
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
19/41
IBM Software Group | Tivoli software
Trace Logging Example
Tracing an authentication failure, WebSEAL using auth-using-
compare=yes
Snippet from msg__webseald-default.log:
2005-07-20-07:55:29.772+00:00I----- 0x132120DD webseald
WARNING ias authsvc pdauthn.cpp 1435 0x00002728
HPDIA0221W Authentication for user testuser failed. You
have used an invalid user name, password or client
certificate.
Enable pd.ivc.ira tracing using:
pdadmin> server task default-webseald-amaix51 trace setpd.ivc.ira 9 file path=/tmp/pdweb.ira.out
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
20/41
IBM Software Group | Tivoli software
Trace Logging Example (continued)
Portion of pd.ivc.ira trace output: 2005-07-20-07:55:29.757+00:00I----- thread(4) trace.pd.ivc.ira:8
/project/am510/build/am510/src/ivrgy/ira_auth.c:1417: CII ENTRY:
ira_auth_passwd_compare() dn: cn=testuser,o=ibm,c=us
2005-07-20-07:55:29.757+00:00I----- thread(4) trace.pd.ivc.ira:7
/project/am510/build/am510/src/ivrgy/ira_entry.c:3053:
ira_ldap_compare_s() DN: cn=testuser,o=ibm,c=us Attr: userPassword
2005-07-20-07:55:29.758+00:00I----- thread(4) trace.pd.ivc.ira:7
/project/am510/build/am510/src/ivrgy/ira_ldap.c:757:
ira_ldap_compare_s(): No timeout - calling ldap_compare_s
2005-07-20-07:55:29.759+00:00I----- thread(4) trace.pd.ivc.ira:7
/project/am510/build/am510/src/ivrgy/ira_ldap.c:767:
ira_ldap_compare_s: Returning LDAP rc x5
2005-07-20-07:55:29.759+00:00I----- thread(4) trace.pd.ivc.ira:7
/project/am510/build/am510/src/ivrgy/ira_entry.c:3060: LDAP rc: x5
2005-07-20-07:55:29.759+00:00I----- thread(4) trace.pd.ivc.ira:8
/project/am510/build/am510/src/ivrgy/ira_auth.c:1427: CII EXIT
ira_auth_passwd_compare()with rc: 0x00000031 LDAP_ERROR x5 "Acompare operation returned false.".
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
21/41
IBM Software Group | Tivoli software
WebSEAL HTTP Trace Logging
pdweb.debug
Advantages:
Smallest trace files available within webseal
HTTP headers in plain text, with time stamp showing arrival/sent
Disadvantages
Only traces HTTP headers
Does not trace responses from WebSEAL or show WebSEAL user or client IP
address
pdweb.snoop
Advantages:
Includes message bodies, responses from WebSEAL and client IP addresses
Decrypts HTTPS traffic Disadvantages
Large trace files (4-5 chars per byte)
Messages are hex encoded (get ascii value for none control char)
Does not show WebSEAL user (unless iv_user header is sent to jnc)
packets do not correspond to network frames in network trace
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
22/41
IBM Software Group | Tivoli software
WebSEAL HTTP Trace Logging
Starting traces
pdadmin>server task webseald-instance trace set
pdweb.debug 2 file path=/var/pdweb/log/debug.log
pdadmin>server task webseald-instance trace set
pdweb.snoop 9 file path=/var/pdweb/log/snoop.out
Stopping traces
pdadmin>server task webseald-instance trace set
pdweb.debug 0
pdadmin>server task webseald-instance trace set
pdweb.snoop 0
Path and Filename Issues for the traces
Follow local Operating System Rules
DYNURL mapping shown in traces
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
23/41
IBM Software Group | Tivoli software
Pdweb.debug and Pdweb.snoop Overview
Typical request breaks down to 4 parts
Browser ===>PD
PD ===>BackEnd
PD
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
24/41
IBM Software Group | Tivoli software
Pdweb.debug Example2 0 0 5 - 0 8 - 0 9 - 1 4 : 0 4 : 5 7 . 8 7 8 - 0 5 : 0 0 I - - - - - t h r e a d ( 4 ) t r a c e . p d w e b . d e b u g : 2
/ p r o j e c t / a m w e b 5 1 0 / b u i l d / a m w e b 5 1 0 / s r c / p d w e b / w a n d / w a n d / l o g . c : 3 0 9 : - - - - - - - - - - - - - - - - - B r o w s e r = = = > P D - - - - - - - - - - - - - - - - -T h r e a d _ I D : 1 3 3 2 6G E T / t e s t / H T T P / 1 . 1 H o s t : l i n u x U s e r - A g e n t : M o z i l l a / 5 . 0 ( X 1 1 ; U ; L i n u x i 6 8 6 ; e n - U S ; r v : 1 . 0 . 1 ) G e c k o / 2 0 0 2 0 9 0 3 A c c e p t :t e x t / x m l , a p p l i c a t i o n / x m l , a p p l i c a t i o n / x h t m l + x m l , t e x t / h t m l ; q = 0 . 9 , t e x t / p l a i n ; q = 0 . 8 , v i d e o / x -m n g , i m a g e / p n g , i m a g e / j p e g , i m a g e / g i f ; q = 0 . 2 , t e x t / c s s , * / * ; q = 0 . 1 A c c e p t - L a n g u a g e : e n - u s , e n ; q = 0 . 5 0 A c c e p t - E n c o d i n g : g z i p ,d e f l a t e , c o m p r e s s ; q = 0 . 9 A c c e p t - C h a r s e t : I S O - 8 8 5 9 - 1 , u t f - 8 ; q = 0 . 6 6 , * ; q = 0 . 6 6 K e e p - A l i v e : 3 0 0 C o n n e c t i o n : k e e p - a l i v eA u t h o r i z a t i o n : * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 0 0 5 - 0 8 - 0 9 - 1 4 : 0 4 : 5 7 . 8 9 6 - 0 5 : 0 0 I - - - - - t h r e a d ( 4 ) t r a c e . p d w e b . d e b u g : 2/ p r o j e c t / a m w e b 5 1 0 / b u i l d / a m w e b 5 1 0 / s r c / p d w e b / w a n d / w a n d / l o g . c : 3 0 9 : - - - - - - - - - - - - - - - - - P D = = = > B a c k E n d - - - - - - - - - - - - - - - - -
T h r e a d _ I D : 1 3 3 2 6G E T / H T T P / 1 . 1 v i a : H T T P / 1 . 1 l i n u x : 4 4 3 u s e r - a g e n t : M o z i l l a / 5 . 0 ( X 1 1 ; U ; L i n u x i 6 8 6 ; e n - U S ; r v : 1 . 0 . 1 ) G e c k o / 2 0 0 2 0 9 0 3i v _ s e r v e r _ n a m e : d e f a u l t - w e b s e a l d - l i n u x a c c e p t - c h a r s e t : I S O - 8 8 5 9 - 1 , u t f - 8 ; q = 0 . 6 6 , * ; q = 0 . 6 6 h o s t : l i n u x . n e t : 8 0 8 0 a c c e p t :t e x t / x m l , a p p l i c a t i o n / x m l , a p p l i c a t i o n / x h t m l + x m l , t e x t / h t m l ; q = 0 . 9 , t e x t / p l a i n ; q = 0 . 8 , v i d e o / x -m n g , i m a g e / p n g , i m a g e / j p e g , i m a g e / g i f ; q = 0 . 2 , t e x t / c s s , * / * ; q = 0 . 1 k e e p - a l i v e : 3 0 0 c o n n e c t i o n : c l o s e a c c e p t - l a n g u a g e : e n - u s ,e n ; q = 0 . 5 0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 0 0 5 - 0 8 - 0 9 - 1 4 : 0 4 : 5 7 . 9 2 8 - 0 5 : 0 0 I - - - - - t h r e a d ( 4 ) t r a c e . p d w e b . d e b u g : 2/ p r o j e c t / a m w e b 5 1 0 / b u i l d / a m w e b 5 1 0 / s r c / p d w e b / w a n d / w a n d / l o g . c : 3 0 9 : - - - - - - - - - - - - - - - - - P D < = = = B a c k E n d - - - - - - - - - - - - - - - - -T h r e a d _ I D : 1 3 3 2 6 H T T P / 1 . 1 2 0 0 O K c o n t e n t - t y p e : t e x t / h t m l l a s t - m o d i f i e d : W e d , 0 6 N o v 2 0 0 2 1 3 : 0 6 : 4 7 G M T d a t e : T u e , 0 9 A u g2 0 0 5 1 9 : 0 4 : 5 7 G M T e t a g : " 2 1 3 7 c - 1 2 5 4 - 3 d c 9 1 3 e 7 " c o n t e n t - l e n g t h : 4 6 9 2 a c c e p t - r a n g e s : b y t e s c o n n e c t i o n : c l o s e s e r v e r :
I B M _ H T T P _ S E R V E R / 1 . 3 . 2 6 . 2 A p a c h e / 1 . 3 . 2 6 ( U n i x ) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 0 0 5 - 0 8 - 0 9 - 1 4 : 0 4 : 5 7 . 9 2 9 - 0 5 : 0 0 I - - - - - t h r e a d ( 4 ) t r a c e . p d w e b . d e b u g : 2/ p r o j e c t / a m w e b 5 1 0 / b u i l d / a m w e b 5 1 0 / s r c / p d w e b / w a n d / w a n d / l o g . c : 3 0 9 : - - - - - - - - - - - - - - - - - B r o w s e r < = = = P D - - - - - - - - - - - - - - - - -T h r e a d _ I D : 1 3 3 2 6H T T P / 1 . 1 2 0 0 O K p 3 p : C P = " N O N C U R O T P i O U R N O R U N I " c o n t e n t - t y p e : t e x t / h t m l l a s t - m o d i f i e d : W e d , 0 6 N o v 2 0 0 2 1 3 : 0 6 : 4 7 G M Tt r a n s f e r - e n c o d i n g : c h u n k e d d a t e : T u e , 0 9 A u g 2 0 0 5 1 9 : 0 4 : 5 7 G M T e t a g : " 2 1 3 7 c - 1 2 5 4 - 3 d c 9 1 3 e 7 " a c c e p t - r a n g e s : b y t e s x - o l d -c o n t e n t - l e n g t h : 4 6 9 2 s e r v e r : I B M _ H T T P _ S E R V E R / 1 . 3 . 2 6 . 2 A p a c h e / 1 . 3 . 2 6 ( U n i x )- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
25/41
IBM Software Group | Tivoli software
Pdweb.snoop Example One
Webseal (9.168.13.15) opens up a socket to the Application Server (9.1.131.27)----------------------------------------
2005-08-08-09:47:36.050+02:00I----- thread(263) trace.pdweb.snoop.jct:1/project/amweb510/build/amweb510/src/pdwebrte/webcore/amw_snoop.cpp:100:
----------------------------------------
Thread 251; fd 58; local 9.168.13.15:62113; remote 9.1.131.27:4482
Socket opened.
A GET is performed on IBMabcLogo.gif
----------------------------------------
2005-08-08-09:47:36.053+02:00I----- thread(263) trace.pdweb.snoop.jct:1/project/amweb510/build/amweb510/src/pdwebrte/webcore/amw_snoop.cpp:159:
----------------------------------------
Thread 251; fd 58; local 9.168.13.15:62113; remote 9.1.131.27:4482
Sending 2652 bytes
0x0000 4745 5420 2e2f 7465 7374 2f69 6d61 6765 GET./test/images
0x0010 732f 4942 4d61 6263 4c6f 676f 2e67 6966 /IBMabcLogo.gif.
0x0020 4854 5450 2f31 2e30 2020 7669 613a 2048 HTTP/1.0..via:.H
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
26/41
IBM Software Group | Tivoli software
Pdweb.snoop Example Two2005-08-07-13:09:31.588-05:00I----- thread(3) trace.pdweb.snoop.jct:1
/project/amweb510/build/amweb510/src/pdwebrte/webcore/amw_snoop.cpp:159:
----------------------------------------
Thread 67586; fd 6; local 192.168.220.130:32895; remote 192.168.220.130:8080
Sending 60 bytes
0x0000 4845 4144 202f 2048 5454 502f 312e 310d HEAD./.HTTP/1.1.
0x0010 0a68 6f73 743a 206c 696e 7578 2e6e 6574 .host:.linux.net
0x0020 3a38 3038 300d 0a63 6f6e 6e65 6374 696f :8080..connectio
0x0030 6e3a 2063 6c6f 7365 0d0a 0d0a n:.close....
2005-08-07-13:09:31.589-05:00I----- thread(3) trace.pdweb.snoop.jct:1
/project/amweb510/build/amweb510/src/pdwebrte/webcore/amw_snoop.cpp:133:----------------------------------------
Thread 67586; fd 6; local 192.168.220.130:32895; remote 192.168.220.130:8080
Receiving 275 bytes
0x0000 4854 5450 2f31 2e31 2032 3030 204f 4b0d HTTP/1.1.200.OK.
0x0010 0a44 6174 653a 2053 756e 2c20 3037 2041 .Date:.Sun,.07.A
0x0020 7567 2032 3030 3520 3138 3a30 393a 3331 ug.2005.18:09:31
0x0030 2047 4d54 0d0a 5365 7276 6572 3a20 4942 .GMT..Server:.IB
0x0040 4d5f 4854 5450 5f53 4552 5645 522f 312e M_HTTP_SERVER/1.
0x0050 332e 3236 2e32 2020 4170 6163 6865 2f31 3.26.2..Apache/1
0x0060 2e33 2e32 3620 2855 6e69 7829 0d0a 4c61 .3.26.(Unix)..La
0x0070 7374 2d4d 6f64 6966 6965 643a 2057 6564 st-Modified:.Wed
0x0080 2c20 3036 204e 6f76 2032 3030 3220 3133 ,.06.Nov.2002.13
0x0090 3a30 363a 3437 2047 4d54 0d0a 4554 6167 :06:47.GMT..ETag
0x00a0 3a20 2232 3133 3763 2d31 3235 342d 3364 :."2137c-1254-3d
0x00b0 6339 3133 6537 220d 0a41 6363 6570 742d c913e7"..Accept-
0x00c0 5261 6e67 6573 3a20 6279 7465 730d 0a43 Ranges:.bytes..C
0x00d0 6f6e 7465 6e74 2d4c 656e 6774 683a 2034 ontent-Length:.4
0x00e0 3639 320d 0a43 6f6e 6e65 6374 696f 6e3a 692..Connection:
0x00f0 2063 6c6f 7365 0d0a 436f 6e74 656e 742d .close..Content-
0x0100 5479 7065 3a20 7465 7874 2f68 746d 6c0d Type:.text/html.
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
27/41
IBM Software Group | Tivoli software
Java Issues
IBM S ft G | Ti li ft
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
28/41
IBM Software Group | Tivoli software
PDJrte Configuration
Verify the pdjrte has been configured properly
Created in
/PolicyDirector
/PolicyDirector/PD.propertiescontains key-value pairs used by the TAM java runtime
/PolicyDirector/PDJLog.propertiescontains key-value pairs used by Java Logging
/PolicyDirector/PDCA.ksCA certificate keystore. Used in subsequent calls to pdmgrd(ie. SvrSslCfg)
Added in /lib/ext/
PD.jaradmin and authorization java classes
ibmjcefw.jarjava cryptography extension
ibmjsse.jarjava secure sockets implementation ibmjcaprovider.jar, US_export_policy.jar, local_policy.jarcryptography
ibmpkcs.jar, ibmpkcs11.jarpublic key cryptography standard support
jaas.jarjava authentication and authorization service
US_export_policy.jar
local_policy.jar
IBM S ft G | Ti li ft
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
29/41
IBM Software Group | Tivoli software
Debugging Common Java Runtime Issues
Expired certificates in keystore files
Check WebSphere logs for errors
Enable WAS security trace
Viewing certificates in keystore using keytool
keytoollistvkeystore -storetype JCEKS
How to refresh the certificate
java com.tivoli.pd.jcfg.SvrSslCfgaction replcertadmin_id
-admin_pwd -cfg_file
IBM S ft G | Ti li ft
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
30/41
IBM Software Group | Tivoli software
Debugging Common Java Runtime Issues (contd)
SvrSslCfg not found
Multiple JREs on system?
Outdated TAM JRTE
WAS 5.0.2 must be configured with TAM 5.1 Java Runtime
Incorrect administrator name or password specified
Caused by incompatibility between PD.jar file shipped with WAS and one
shipped with TAM.
To resolve, copy PD.jar file from /java/export/pdjrte/ to
/java/jre/lib/ext directory
Embedded WAS Support Issues
IBM S ft G | Ti li ft
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
31/41
IBM Software Group | Tivoli software
Tracing and Messaging
PDJLog.properties is a wrapper to the java logging facility
Configuration of logging is done via:
/PolicyDirector/PDJLog.properties
Log files created: trace_amj.log
msg__amj_fatal.log
msg__amj_error.log
msg__amj_warning.log
msg__amj_notice.logmsg__amj_noticeverbose.log
IBM Software Group | Tivoli software
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
32/41
IBM Software Group | Tivoli software
Enabling Tracing and Logging
To enable logging:
Edit the PDJLog.properties:
For all components, specify:
baseGroup.PDJTraceLogger.isLogging = true
For individual components, specify:
baseGroup.PDJadminTraceLogger.isLogging = true
baseGroup.PDJauthzTraceLogger.isLogging = true
IBM Software Group | Tivoli software
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
33/41
IBM Software Group | Tivoli software
GSKit Traces
IBM Software Group | Tivoli software
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
34/41
IBM Software Group | Tivoli software
GSkit Trace
To enable the trace, perform the following steps:
Specify the file in which the trace data is to be stored with the environment
variable GSK_TRACE_FILE. Reference the following example:
export GSK_TRACE_FILE=/tmp/mytracefile
Re-create the error.
The system will append a ".1" to the file name and then accumulate
about 25 megabytes of trace data. It will then close the
"/tmp/mytracefile.1" file, open a "/tmp/mytracefile.2" file which
accumulate 25 more megabytes of trace information. It will then close
that one, erase the first file, and start over.
The trace files are binary
IBM Software Group | Tivoli software
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
35/41
IBM Software Group | Tivoli software
System Data
IBM Software Group | Tivoli software
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
36/41
IBM Software Group | Tivoli software
Must Gather Information for Support
The following should be known before calling support
Platform for each component (O/S level including patches)
TAM Version and Fixpack Level from all machine not just the failing machine
Appropriate Log and Configuration files
If core, provide senddata output
If windows failure, provide dr watson
User Registry and version (IDS, Sun, eDirectory)
Integration with other products
IBM Software Group | Tivoli software
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
37/41
IBM Software Group | Tivoli software
Capturing Core Files
Senddata ScriptWhat does it capture
core
daemon binary
libs.tar
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
38/41
IBM Software Group | Tivoli software
Core Files cont.
Also on AIX it is possible to use the AIX Command snapcore which
does not require dbx.
IBM Software Group | Tivoli software
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
39/41
IBM Software Group | Tivoli software
System_status Script
Sample of Information Gathered
O/S and patch levels
Resource and Environment data (Memory, disk space, environment
variables, locales, ulimits)
Network information (/etc/hosts, ip address, network devices, aliases)
TAM Configuration (configuration files, daemon build levels)
TAM Log files
TAM data
ACLs, Users, POPs, Groups, Junctions, ObjectSpace, Servers, Password
Policy, GSO data,
LDAP Data Schema definitions, suffix data
DB2 Data
Instances, db connectivity to databases, table searches
IBM Software Group | Tivoli software
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
40/41
IBM Software Group | Tivoli software
System_status script (cont)
This script can be run with multiple options
system_status.ksh
Anonymous pdadmin and Anonymous LDAP Requests
system_status.ksh -D 'cn=root' -w 'cn=root_password'
Anonymous pdadmin but LDAP authentication with
cn=root/cn=root_password
system_status.ksh -a 'sec_master' -p 'master_password'
pdadmin authentication with sec_master/master_password, but
Anonymous LDAP bind
system_status.ksh -a 'sec_master' -p 'master_password' -D 'cn=root' -w
'cn=root_password'
Pdadmin and LDAP authentication
Resulting file will be in the format of hostname-mm-dd-yy_hh-mm-ss
This script can take up to and hour or more to run
IBM Software Group | Tivoli software
-
8/10/2019 Tivoli Access Manager problem determination using logging and tracing features
41/41
IBM Software Group | Tivoli software
Questions