am 3.2: technical support troubleshooting and new …copying all or part of this manual, or dist...

Nove

AM 3.2: Technical Support Troubleshooting and New FeaturesLecture

www.novel l .comNovell Training Services

AT T L I V E 2 0 1 2 L A S V E G A S

N I Q 1 8

ll, Inc. Copyright 2012-ATT LIVE-1-HARDCOPY PERMITTED. NO OTHER PRINTING, COPYING, OR DISTRIBUTION ALLOWED.

Novel

Legal Notices

Novell, Inc., makes no representations or warranties with respect to the contents or use of this documentation, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to revise this publication and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes.

Further, Novell, Inc., makes no representations or warranties with respect to any software, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to make changes to any and all parts of Novell software, at any time, without any obligation to notify any person or entity of such changes.

Any products or technical information provided under this Agreement may be subject to U.S. export controls and the trade laws of other countries. You agree to comply with all export control regulations and to obtain any required licenses or classification to export, re-export or import deliverables. You agree not to export or re-export to entities on the current U.S. export exclusion lists or to any embargoed or terrorist countries as specified in the U.S. export laws. You agree to not use deliverables for prohibited nuclear, missile, or chemical biological weaponry end uses. See the Novell International Trade Services Web page (http://www.novell.com/info/exports/) for more information on exporting Novell software. Novell assumes no responsibility for your failure to obtain any necessary export approvals.

Copyright © 2012 Novell, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the publisher.

Novell, Inc., has intellectual property rights relating to technology embodied in the product that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more of the U.S. patents listed on the Novell Legal Patents Web page (http://www.novell.com/company/legal/patents/) and one or more additional patents or pending patent applications in the U.S. and in other countries.

Novell, Inc.

404 Wyman Street, Suite 500

Waltham, MA 02451

U.S.A.

www.novell.com

Online Documentation: To access the latest online documentation for this and other Novell products, see the Novell Documentation Web page (http://www.novell.com/documentation).

Novell Trademarks

For Novell trademarks, see the Novell Trademark and Service Mark list (http://www.novell.com/company/legal/trademarks/tmlist.html).

Third-Party Materials

All third-party trademarks are the property of their respective owners.

Copying all or part of this manual, or distributing such copies, is strictly prohibited.To report suspected copying, please call 1-800-PIRATES.

Version 12

l, Inc. Copyright 2012-ATT LIVE-1-HARDCOPY PERMITTED. NO OTHER PRINTING, COPYING, OR DISTRIBUTION ALLOWED.

http://www.novell.com/info/exports/

http://www.novell.com/company/legal/patents/

http://www.novell.com/documentation

http://www.novell.com/company/legal/trademarks/tmlist.html

Troubleshooting NetIQ Access Manager 3.2™

Neil CashellTechnical Support [email protected]

Novell, Inc. Copyright 2012-ATT LIVE-1-HARDCOPY PERMITTED. NO OTHER PRINTING, COPYING, OR DISTRIBUTION ALLOWED.

© 2011 NetIQ Corporation. All rights reserved.2

Agenda

• General Networking troubleshooting tools• Access Manager specific troubleshooting tools• Access Manager protected resource flow• Access Manager log settings and log files• Case study• Additional reading

– Hidden slides (Access Gateway Service)


3 © 2011 NetIQ Corporation. All rights reserved.

Networking Tools

• netstat -patune –connection and stat info• tcpdump/wireshark• netcat• tcp stats:

‒ general tcp/udp stats /proc/net/snmp

• Ethtool (-S, -K TSO)• ipsysctl TCP settings• iptables (-t nat -nvL) – make sure firewall not blocking

data; redirecting ports; masquerading



Networking Tools

• CurrPorts (Windows port monitor)‒ http://www.nirsoft.net/utils/cports.html

• PsTools (Windows process tools)‒ http://technet.microsoft.com/en-us/sysinternals/bb896649


http://www.nirsoft.net/utils/cports.html

http://technet.microsoft.com/en-us/sysinternals/bb896649


Export options‒ Complete setup via ambkup.sh‒ Access Gateway via the device -> Export option

‒ http://www.novell.com/documentation/novellaccessmanager/adminguide/index.html?page=/documentation/novellaccessmanager/adminguide/data/ba9dh2r.html

‒ Policy information‒ http://www.novell.com/documentation/novellaccessmanager/adminguide/index.

html?page=/documentation/novellaccessmanager/adminguide/data/b5pm021.html

‒ LDAP browser and browse to following

Generic Novell® Access Manager™ Troubleshooting Tools (cont.)



Certificates and keystores‒ openssl s_client -connect idpcluster.lab.novell.com:8443

CONNECTED(00000003)depth=1 /OU=Organizational CA/O=linuxlab5_treeverify error:num=19:self signed certificate in certificate chainverify return:0---Certificate chain 0 s:/CN=idpcluster.lab.novell.com i:/OU=Organizational CA/O=linuxlab5_tree 1 s:/OU=Organizational CA/O=linuxlab5_tree i:/OU=Organizational CA/O=linuxlab5_tree

‒ keytool -list -keystore /var/opt/novell/novlwww/devman.keystore -v

Your keystore contains 1 entryAlias name: tomcatCreation date: 13-Dec-2011Entry type: keyEntryCertificate chain length: 2Certificate[1]:Owner: O=novell, OU=accessManager, CN=linuxlab5Issuer: O=linuxlab5_tree, OU=Organizational CA:Certificate[2]:Owner: O=linuxlab5_tree, OU=Organizational CAIssuer: O=linuxlab5_tree, OU=Organizational CA:




Certificates and keystores‒ JAVA “javax.net.debug=ssl” option to add SSL handshake debug messages

‒ Add 'JAVA_OPTS="${JAVA_OPTS} -Djavax.net.debug=ssl"' to tomcat5.conf

‒ http-8443-Processor9, WRITE: TLSv1 Handshake, length = 32Thread-78, WRITE: TLSv1 Application Data, length = 52Thread-78, called close()Thread-78, called closeInternal(true)Thread-78, SEND TLSv1 ALERT: warning, description = close_notifyThread-78, WRITE: TLSv1 Alert, length = 18Thread-2677, handling exception: java.net.SocketException: Socket closed%% Invalidated: [Session-1264, SSL_RSA_WITH_RC4_128_MD5]Thread-2677, SEND TLSv1 ALERT: fatal, description = unexpected_messageThread-2677, WRITE: TLSv1 Alert, length = 18Thread-2677, Exception sending alert: java.net.SocketException: Socket closedThread-2677, called closeSocket()http-8443-Processor9, READ: TLSv1 Change Cipher Spec, length = 1http-8443-Processor9, READ: TLSv1 Handshake, length = 32*** Finished




IDP config 'Logging' TAB configuration




AC general logs from 'Auditing' TAB




Network layout informationFirewalls/L4 may pose Connectivity/State problems

LAN analyzer (Wireshark, TCPDump)‒ Trace traffic between browser, proxy, IDP and

authentication servers‒ Loopback interface

Performance analysis tools on dependencies‒ http://www.novell.com/communities/node/7063/elapsed-time-416

(LDAP performance on eDirectory)‒ HTTP common or extended logs (Web server performance)



http://www.novell.com/communities/node/7063/elapsed-time-416


NIDP/NESP Monitor or Statistic logging‒ /opt/novell/nids(nesp)/lib/webapp/WEB-INF

‒ urn:novell:nidp:monitor:anyaccess




Statistic logging (Auditing → Device Health → Device → Statistics)




Configuration reader‒ /opt/novell/devman/bin/amdiagcfg.sh and browser!




Access Gateway Overview

IdentityServer

AccessGateway

Identity Store

Apache or IIS webserver configured

to accept header-basedauthentication

1. User Accesses protected resource2. User is redirected to Identity Server and is presented with an http login form requesting their username and password3. The Identity Server verifies the username and password against the Identity Store4. Once the user's identity is validated, the Access Gateway retrieves the user's common name and password5. The Access Gateway injects the username and password into the authentication header and allows access to the encrypted Web content

3

51

2

4



Access Gateway/ESP Flow

1

2

3

4

5

6

7

8

9

1011

12

13

141516

Client Browser External website AG Service Provider Identity Provider

Respond with requestfor Liberty session

Redirect to login page with Liberty<AuthnRequest

The AGW requests metadata

The IDP requests metadata

IDP creates an authentication EntryRedirect browser toSP with Artifact

The SP sends the artifact to the IDPThe IDP responds withthe list of attributes overthe SOAP backchannelSession information

The IDP sends login page

User has access toProtected resource

User enterscredentials

User tries to accessProtected resource



Liberty Authentication Request

• Make sure the AuthnRequest includes the appropriate information (http://www.projectliberty.org/liberty/content/download/2197/14625/file/draft-liberty-idff-protocols-schema-1.2-errata-v3.0.pdf – section 3.2!)

‒ ProviderID matches SP metadata entry

‒ Contract matches

‒ Time matches‒ https://idpcluster.lab.novell.com:8443/nidp/idff/sso?RequestID=idNTXycnsP7cfmrq5o.k8za-

yuIus&MajorVersion=1&MinorVersion=2&IssueInstant=2007-09-24T11%3A41%3A29Z&ProviderID=https%3A%2F%2Fwww.neilagesp.net%3A443%2Fnesp%2Fidff%2Fmetadata&RelayState=https%3A%2F%2Fwww.neilagesp.net%3A443%2FLAGBroker%3F%2522http%3A%2F%2Fwww.mylag.com%2Fservlets-examples%2F%2522&consent=urn%3Aliberty%3Aconsent%3Aunavailable&ForceAuthn=false&IsPassive=false&NameIDPolicy=onetime&ProtocolProfile=http%3A%2F%2Fprojectliberty.org%2Fprofiles%2Fbrws-art&AuthnContextStatementRef=secure%2Fname%2Fpassword%2Furi



Liberty Authentication Request (cont.)

• Confirm that contract can be executed‒ Local Contract com.novell.nidp.authentication.AuthenticationContract@ded4ba

https://idpcluster.lab.novell.com:8443/nidp/idff/sso com.novell.nidp.authentication.ContractExecutionState@13805c9<amLogEntry> 2012-03-24T14:13:37Z VERBOSE NIDS Application: Executing authentication method Introductions </amLogEntry><amLogEntry> 2012-03-24T14:13:37Z VERBOSE NIDS Application: Authentication method Introductions failed. </amLogEntry><amLogEntry> 2012-03-24T14:13:37Z VERBOSE NIDS Application: Session has consumedauthentications: false </amLogEntry><amLogEntry> 2012-03-24T14:13:37Z VERBOSE NIDS Application: Executing authentication method Secure Name/Password - Form </amLogEntry>

• Confirm that artifact sent back‒ <amLogEntry> 2012-03-24T14:13:42Z INFO NIDS Application: AM#500105018:

AMDEVICEID#D5AF8CA5FBDB5813: AMAUTHID#BA7213D5E240018DD2F5FB38A4C37C1A: Responding to AuthnRequest with artifact AAOCkf3sRbgL1kSiTxccEVUvvBGYJO30dM1xkwe8y4gwRXYV9UfDf52J </amLogEntry>



Liberty Authentication Response (cont.)

• Confirm that assertion request received from SP‒ <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-

ENV:Body><samlp:Request xmlns:samlp="urn:oasis:names:tc:SAML:1.0:protocol" xmlns:lib="urn:oasis:names:tc:SAML:1.0:protocol" xmlns:saml="urn:oasis:names:tc:SAML:1.0:assertion" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" IssueInstant="2012-03-24T14:13:21Z" MajorVersion="1" MinorVersion="1"RequestID="idQCXo90QeOxtVF7Re1tSfK-F5o4"><samlp:AssertionArtifact>AAOCkf3sRbgL1kSiTxccEVUvvBGYJO30dM1xkwe8y4gwRXYV9UfDf52J</samlp:AssertionArtifact></samlp:Request></SOAP-ENV:Body></SOAP-ENV:Envelope>

• Confirm assertion response sent to SP (with assertion)

‒ <amLogEntry> 2012-03-24T14:13:42Z NIDS Trace: Method: BaseHandler.sendSOAPResponse() Thread: http-0%2F0.0.0.0-8443-Processor4SOAP EndpointResponse: <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"> <SOAP-ENV:Body> <samlp:Response InResponseTo="idQCXo90QeOxtVF7Re1tSfK-jF5o4" IssueInstant="2012-03-24T14:13:42Z" MajorVersion="1" MinorVersion="1" Recipient="https://www.neilagesp.net:443/nesp/idff/metadata" ResponseID="idtz8AISJfSnxQX60j0-cESUbdMrY" xmlns:lib="urn:liberty:iff:2003-08" xmlns:samlp="urn:oasis:names:tc:SAML:1.0:protocol"> <samlp:Status> <samlp:StatusCode Value="samlp:Success"/> </samlp:Status> <saml:Assertion AssertionID="id7-m97u9xYZGWWzTZpqdoc7A.NSc" InResponseTo="idbiFOuDVt9UPHvfa9QLZ8puR7uuk" IssueInstant="2012-03-24T14:13:42Z" Issuer="https://idpcluster.lab.novell.com:8443/nidp/idff/metadata" MajorVersion="1" MinorVersion="2" .................................



Assume authentication headers used for SSO to origin web servers

Access Gateway Group(Load Balanced by L4 Switch)

Access Manager Advanced OverviewExisting Session with Web Single-Sign-On and Access Gateway Cluster

IdentityServer

1

Identity Store

Web Servers

6

AG2

AG1Web

Browser

1. User Accesses protected resource on AG1 and the browser presents AG1 a valid Access Gateway session cookie created earlier by AG2 for this user session.

2. AG1 doesn't have a session for the user so it asks the other AGs in the Group to see if they have a session for the user3. AG2 responds claiming ownership for the user session

4. AG1 asks AG2 for the policy and user data required for the user to access the protected resource5. AG2 requests policy and user data from the Identity Server (if it isn't cached)6. The Identity Server gets the user data from the Identity Store (if it isn't cached)

9. AG1 processes the policy and user data and allows access to the protected resource

7. Identity Server responds to AG2 with the policy and user data8. AG2 responds to AG1 with the policy and user data

22, 32, 3, 4

55, 7

2, 3, 4, 8

9

Assume User already had active session on AG2.



Identity Server Group(Load Balanced by L4 Switch)

IDS1

IDS2

Access Gateway Advanced OverviewNew session with Web Single Sign-On and Identity Server Cluster

AccessGateway

1

Identity Store

Web Servers

2

AG1

1. User accesses protected resource on AG1

2. AG1 redirects users browser to IDS1 which presents a web login form requesting a username and passwd

3. IDS1 verifies the username and password against the Identity Store

4. IDS1 redirects browser back to AG1 with auth artifact

5. AG1 requests from IDS2 an authentication artifact validation and policy/user data required to access the protected resource

6. IDS2 doesn't have a session for the user so it asks the other IDSes in the Group if they have a session for the user and IDS1 responds claiming ownership for the session

7. IDS2 forwards request to IDS1

8. IDS1 validates artifact and obtains user data from Identity Store

9. IDS1 responds to IDS2 with the artifact validation and policy/user data

10. IDS2 responds to AG1 with the artifact validation and policy/user data

11. AG1 creates AG session for the new user, processes policies and user data, and allows access to the protected resource

4

5

33, 8

11


66, 76, 7, 9

5, 10



Troubleshooting Clustering Tools

• Verify persistence enabled on Load Balancer• Enable Via HTTP header

‒ Returns deviceID of Access Gateway• IDP/ESP Statistics output

‒ Proxied versus non Proxied requests• Identify ESP/IDP sessionID on client HTTP headers

‒ Search catalina and AG log files for sessionID• Check for 'must proxy' string in catalina

‒ Confirmation that request has been proxied



AGA/AGS Troubleshooting Tools

• netcat localhost 2300 on LAG‒ view proxy console

• OS tools TOP/Netstat/'PS -eLf'‒ check process utilisation, memory and conn usage

• HTTP header and data viewer‒ STRACE on IE or Firefox httpfox plugin

‒ Fiddler (http://www.fiddler2.com/fiddler2/) - incl. RTT

• viewinfo.* files from unsupported directory‒ Decode HTTP headers on back end

• Diff tools e.g. Beyond Compare (rewriting issues)• Curl (view IDP metadata, simulate HTTP req)


http://www.fiddler2.com/fiddler2/


AG Troubleshooting Tools (cont.)

• TCPDUMP output (incl. Loopback on Linux for TCP 8181/9009)



LAG Troubleshooting Files

• /var/log/ics_dyn.log - verbosity of message depends on

‒ /etc/laglogs.conf file settings

LOG_LEVEL=7 (default 5)DEBUG_SOAP_MESSAGE=1 (default 0)DEBUG_HTTP_HEADERS=1 (default 0)DEBUG_HTTP_RESPONSE=1 (default 0)

• /var/novell/.~newInstall‒ remove file => Clears cache

• Touch files from /var/novell/ and /tmp/‒ Coredumps, remove IP address checks for example



LAG Troubleshooting Files (cont.)

/var/log/laghttpheadersdecodes http headers of requests/responses on all channels

Sending request to webserver for browser request '98'-------------------------------------------------------------------------GET /images/classifieds/quicksearch/updates.png

HTTP/1.1Host: www.unison.ieReferer: http://www.unison.ie/Accept-Language: en-usAccept-Encoding: gzip, deflateUser-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows

NT 5.1)Cookie:

Unison_User=83.141.112.214.109131171028663164; Via: 1.1 www.mylag.com (Access Gateway 3.0.0-83)

Headers received from webserver for request '98'------------------------------------------------------------------Date: Fri, 26 Jan 2008 14:54:15 GMTServer: Apache/1.3.34 (Debian) PHP/4.4.2-1.1

mod_perl/1.29Last-Modified: Mon, 22 Jan 2007 11:23:29 GMTAccept-Ranges: bytesContent-Length: 1932Content-Type: image/png




/var/log/lagsoapmessages‒ log-level setting available via /etc/laglogs.conf‒ Decodes all SOAP backchannel messages for auth and policy

interaction‒ Get user, roles, contract and timeout details during auth‒ Get personal policy info for formfill, II and authorization

‒ <SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><NIDPSetSession XLibid="00000200930224c625b125a639540dda7192bb24fbfcd794" hardExpire="899" id="552382333C8BE989D7F39E1993D30B33" softExpire="584"><storetype="ldap"><dn>cn=ncashell,o=novell</dn></store><authentications><contracts><contract>name/password/uri</contract></contracts></authentications><roles/></NIDPSetSession></SOAP-ENV:Body></SOAP-ENV:Envelope>




/var/log/ics_dyn.log‒ proxy specific logs‒ Unique format

‒ <time>:<host>:<component>:<DeviceID>:<AuthID>:<EventID><mesg>

‒ Component determined by string 5045xxxx‒ where '5' is the log level (never changes!)

‒ '045' represents the LAG component ID

‒ 'xxxx' represents the LAG subgroup ... for example

'0100' -> multihoming'0400' -> Authentication'0600' -> Identity Injection'1100' -> Rewriting'1200' -> SOAP backchannel




/var/log/ics_dyn.logFeb 18 13:39:46 lag129 : AM#504503000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#681: Process request 1 'lag129.lab.novell.com:/formfill/sybase.html' [147.2.36.148:2134 -> 147.2.16.129:443]Feb 18 13:39:46 lag129 : AM#504517000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#681: Search success for /formfill/sybase.html (0xa5cf96e4:0xa598b7a4:64)Feb 18 13:39:46 lag129 : AM#504504000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#681: protected-resourceFeb 18 13:39:46 lag129 : AM#504504000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#681: Got valid Cookie[1984350736 196608 3530491756 1573825269 147.2.36.148 0.3 CIP:147.2.36.148] COOKIE_VALIDATIONFeb 18 13:39:47 lag129 : AM#504507000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#1F45C624E8EF324AC9A92FA39E20B22F: AMEVENTID#681:Scheduling Formfill, policies matched 1Feb 18 13:39:47 lag129 : AM#504503000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#681: Sending request to origin server 147.2.16.154:80 (c24cb1a1.c24cb1a1)Feb 18 13:39:47 lag129 : AM#504509000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#681: Received response from origin server, status = 200 (147.2.16.154:80)Feb 18 13:39:47 lag129 : AM#504507000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#1F45C624E8EF324AC9A92FA39E20B22F: AMEVENTID#681:Content-Type () Formfill is interested in this response.Feb 18 13:39:47 lag129 : AM#504507000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#1F45C624E8EF324AC9A92FA39E20B22F: AMEVENTID#681:FFResDS:0xa59ff824 Processing responseFeb 18 13:39:47 lag129 : AM#504507000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#1F45C624E8EF324AC9A92FA39E20B22F: AMEVENTID#681:FF Sending GetAttribute soaprequest:5987 to eSP.(1F45C624E8EF324AC9A92FA39E20B22F)Feb 18 13:39:49 lag129 : AM#504512000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#5987: backchannel receivedResp (app a5fe24a4 FF ) (5987)[seg:0xa4b87de0:0xa58c4a00:1125]Feb 18 13:39:49 lag129 : AM#504507000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#1F45C624E8EF324AC9A92FA39E20B22F: AMEVENTID#681:ffCacheDataEvent:: data:0xa5a46824 start FormfillFeb 18 13:39:49 lag129 : AM#404517000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#681: FF Adjusting content length by 314, original entitySize 8440 (0)Feb 18 13:39:49 lag129 : AM#504507000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#1F45C624E8EF324AC9A92FA39E20B22F: AMEVENTID#681:Completed Formfill processing.(hit)Feb 18 13:39:49 lag129 : AM#504520000: AMDEVICEID#ag-7AA324FFCBA4D4E: AMAUTHID#0: AMEVENTID#681: Browser req/resp[1185635, 1185637, 1185639] [timeToResp:2 respDuration:2] curTime:1185639 FinishTransmit [auth:0 acl:0 II:0] [rewrite 0 :1185637 11856371185637] [origin: 1185637, 1185637, 1185637,1185637 retry:0 0]



Troubleshooting Files (cont.)

• /var/opt/novell/nam/logs/idp(nesp)/tomcat/catalina.out‒ eSP logs for communication with proxy and IDP

‒ eSP inherits IDP logging settings ('Application, Liberty, Web Service Provider/Consumer)

‒ Used to troubleshoot import, authentication and policy issues

‒ Can search for JSESSIONID, Policy ID or threadID (Processor string)

‒ Display IDP/ESP statistics‒ Performance issues running out of threads (maxThreads,

Xmx, LDAPLoadThreshold)‒ http://www.novell.com/communities/node/9321/how-configure-access-

gateway-embedded-service-provider-reduce-access-gateway-load-and-impr



Troubleshooting Policies

• Define what policies get evaluated on IDP and AG• Make sure Web Services Consumer/Provider

components set to DEBUG• Understand policy setup (condition versus action)• Understand proxy generated soap request to ESP

‒ ESP may need to go to user store with poor configurations

• Understand tools to confirm proxy received info‒ HTTP headers (II), STRACE (FF), LAN traces (loopback)



Role based Policy – IDP server logs



Authz/II/FF based Policy – AG logs• EventID used to track policy 'Evaluation' in catalina log file

‒ Retrieve EventID from Via HTTP header

• Authorization policy contains conditions and actions‒ Log entries on LAG similar to the IDP role entries

• Formfill and Identity Injection policies‒ GetAttributeRequest SOAP request to ESP‒ Important to send attributes in Assertion for performance


Troubleshooting Case Study:Single sign-on to back-end app fails with Identity Injection



Policy Case Study–Background

• Customer enabled an Identity Injection policy to apply to a protected resource policy added the:

‒ username and password to the basic auth header‒ user's e-mail address to the X-Mail HTTP header‒ user's certificate to the X-userCertificate HTTP header

• After applying the policy and logging in to the Linux Access gateway protected resource, the user could not SSO to the back-end Web server

‒ authentication failed, error messages were returned from theback-end application

‒ No valid user certificate sent



Policy Case Study–Troubleshooting

Get policy and where policy applied (get screenshot) of protected resources and export of policy)



Policy Case Study–Troubleshooting

• View protected resources with amdiagcfg.sh output‒ Policies enabled and configured correctly

• Enable logs for policies ‒ Must understand where in the policy flow the request is

failing (Web server, Proxy server, eSP, IDP, user store)?



Policy Case Study–Log Analysis• Check browser HTTP headers for cookies (LAG/ESP)

• Locate event ID from LAGHTTPHeaders ouput

• Search ICS_DYN log for eventID and policy activation

Feb 5 10:49:31 www : AM#504506000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#98514F39F4D2A2D5A8638C25560765A5: AMEVENTID#31: IdInjection enabled for the protected resourceFeb 5 10:49:31 www : AM#504506000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#98514F39F4D2A2D5A8638C25560765A5: AMEVENTID#31: IIRdata:a9d35704 cnt:2 processSearchMatch (ds:a99ecd44)Feb 5 10:49:31 www : AM#504506000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#98514F39F4D2A2D5A8638C25560765A5: AMEVENTID#31: idCache miss. (key<43KO7M0O-9719-280O-200M-5772M447KL4IPCZQX03a36c6c0a=00000000930223500d7f35546deb348a87c859e198514F39F4D2A2D5A8638C25560765A5>)Feb 5 10:49:31 www : AM#504506000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#98514F39F4D2A2D5A8638C25560765A5: AMEVENTID#31: II:a9d35704 Sending EVAL Request 5715 policyId 43KO7M0O-9719-280O-200M-N5772M447KL4Feb 5 10:49:31 www : AM#504512000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#0: AMEVENTID#15: processSoapRequests - size 6 processed 1, deleted 3 (3, conFail 0 conTimeout 0) 0 (0)Feb 5 10:49:31 www : AM#504515000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#0: AMEVENTID#56: CSocket 0xa99bd624:56 connectInProgress [0.0.0.0:0 0.0.0.0:8080] defaultNagleFeb 5 10:49:31 www : AM#504515000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#0: AMEVENTID#0: Connection Established with peer 127.0.0.1:8080 (src 127.0.0.1:0)Feb 5 10:49:31 www : AM#504512000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#0: AMEVENTID#5715: sent soapRequest 5715 app a99ecd88 IISCacheCreateWrked for pool Xerc 20000 (6)nFeb 5 10:49:31 www : AM#504512000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#0:AMEVENTID#5715: backchannel receivedResp (app a99ecd88 II ) (5715)[seg:0xa8b87de0:0x586aa048:16131]Feb 5 10:49:31 www : AM#504506000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#98514F39F4D2A2D5A8638C25560765A5: AMEVENTID#31: Received response for IdInjection EVAL requestFeb 5 10:49:31 www : AM#504506000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#98514F39F4D2A2D5A8638C25560765A5: AMEVENTID#31: Injecting AUTH_HEADERFeb 5 10:49:31 www : AM#504506000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#98514F39F4D2A2D5A8638C25560765A5: AMEVENTID#31: Injecting CUSTOM_HEADERFeb 5 10:49:31 www : AM#504506000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#98514F39F4D2A2D5A8638C25560765A5: AMEVENTID#31: Injecting (X-mail)Feb 5 10:49:31 www : AM#504506000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#98514F39F4D2A2D5A8638C25560765A5: AMEVENTID#31: Injecting CUSTOM_HEADERFeb 5 10:49:31 www : AM#504506000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#98514F39F4D2A2D5A8638C25560765A5: AMEVENTID#31: Injecting (X-ClientCert)Feb 5 10:49:31 www : AM#504503000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#0: AMEVENTID#31: connecting to webserver 147.2.16.154:80 c24cb1a1 noPersist . (policy:1:2)Feb 5 10:49:31 www : AM#504515000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#0: AMEVENTID#56: CSocket 0xa99cda24:56 connectInProgress [147.2.16.159:0 147.2.16.159:80] Feb 5 10:49:31 www : AM#504515000: AMDEVICEID#ag-BC7465BE319FA86: AMAUTHID#0: AMEVENTID#0: Connection Established with peer (147.2.16.154:80)



Policy Case Study–Log Analysis

Check AG Catalina.out log for policy evaluate<amLogEntry> 2011-02-05T10:49:31Z INFO NIDS Application: AM#501101050: AMDEVICEID#esp-7AA324FFCBA4D4ED: PolicyID#43KO7M0O-9719-280O-200M-N5772M447KL4: NXPESID#5715: Evaluating policy </amLogEntry><amLogEntry> 2011-02-05T10:49:31Z INFO NIDS Application: AM#501103050: AMDEVICEID#esp-7AA324FFCBA4D4ED: AMAUTHID#98514F39F4D2A2D5A8638C25560765A: 43KO7M0O-9719-280O-200M-N5772M447KL4: NXPESID#5715: AGIdentityInjection Policy Trace: ~~RL~1~~~~Rule Count: 1~~Success(67) ~~RU~RuleID_1239275044815~IdentityInjection~DNF~~0:3~~Success(67) ~~PA~ActionID_1265966514254~~InjectAuthHeader~uid~uid(1):CredentialProfile(7010:):NEPXurn~3Anovell~3Acredentialprofile~3ASecret~5Bcp~3AName~3D~22LDAPCredentials~22~5D~2Fcp~3AEntry~5Bcp~3AName~3D~22UserName~22~5D:~Ok:ttl -1~Success(0)~~PA~ActionID_1265966514254~~InjectAuthHeader~password~pwd(1):CredentialProfile(7010:):NEPXurn~3Anovell~3Acredentialprofile~3A2005ret~5Bcp~3AName~3D~22LDAPCredentials~22~5D~2Fcp~3AEntry~5Bcp~3AName~3D~22UserPassword~22~5D:~Ok~Success(0) ~~PC~ActionID_1265966514254~~Document=(ou=xpemlPEP,ou=mastercdn,ou=ContentPublisherContainer,ou=Partition,ou=PartitionsContainer,ou=VCDN_Root,ou=accessManagerContainer,o=novell:romaContentCollectionXMLDoc),Policy=(IdentityInjection),Rule=(1::RuleID_1239275044815),Action=(InjectAuthHeader::ActionID_1265966514254)~~~~Success(0) ~~PA~ActionID_1254471149303~~Inject Custom Header~Xmail~Value(2):LdapAttribute(6647:):NEPXurn~3Anovell~3Aldap~3A200602~2Fldap~3AUserAttribute~40~40~40~40WSCQLDAPToken~40~40~40~40~2FUserAttribute~5B~40ldap~3AtargetAttribute~3D~22mail~22~5D:~Ok:ttl -1~Success(0) ~~PC~ActionID_1254471149303~~Document=(ou=xpemlPEP,ou=mastercdn,ou=ContentPublisherContainer,ou=Partition,ou=PartitionsContainer,ou=VCDN_Root,ou=accessManagerContainer,o=novell:romaContentCollectionXMLDoc),Policy=(IdentityInjection),Rule=(1::RuleID_1239275044815),Action=(InjectCustomHeader::ActionID_1254471149303)~~~~Success(0) ~~PA~ActionID_1261572496536~~InjectCustomHeader~XClientCert~Value(2):LdapAttribute(6647:):NEPXurn~3Anovell~3Aldap~3A200602~2Fldap~3AUserAttribute~40~40~40~40WSCQLDAPToken~40~40~40~40~2FUserAttribute~5B~40ldap~3AtargetAttribute~3D~22userCertificate~22~5D:~Ok:ttl -1~Success(0) ~~PC~ActionID_1261572496536~~Document=(ou=xpemlPEP,ou=mastercdn,ou=ContentPublisherContainer,ou=Partition,ou=PartitionsContainer,ou=VCDN_Root,ou=accessManagerContainer,o=novell:romaContentCollectionXMLDoc),Policy=(IdentityInjection),Rule=(1::RuleID_1239275044815),Action=(InjectCustomHeader::ActionID_1261572496536)~~~~Success(0) </amLogEntry>




Check AG catalina.out log for parameter values and return codesQuery Response: <ldap:QueryResponse(urn:novell:ldap:2006-02)>:ns=urn:novell:ldap:2006-02 nspfx=ldap itemIdRef=exss80bmcyk3x timeStamp=2007-02-05T10:49:30Z <ldap:Status(urn:novell:ldap:2006-02)>:code=ldap:OK <ldap:Data(urn:novell:ldap:2006-02)>: itemIdRef: NEPXurn~3Anovell~3Aldap~3A2006-02~2Fldap~3AUserAttribute~40~40~40~40WSCQLDAPToken~40~40~40~40~2FUserAttribute~5B~40ldap~3AtargetAttribute~3D~22mail~22~5D

<ldap:UserAttribute(urn:novell:ldap:2006-02)>: Id: exss80521py4a Target Attribute: mail

<ldap:Value(urn:novell:ldap:2006-02)>: Value: *****

Method: com.novell.nidp.liberty.wsc.WSC.getDataWithoutInteraction()(Thread: http-8080-Processor3): Completed Request. Response: WSCResponse: Status: All Success WSCQResponseEntry: WSCQLDAPToken: Model Entry: UserAttribute Unique Id: NEPXurn~3Anovell~3Aldap~3A2006-02~2Fldap~3AUserAttribute~40~40~40~40WSCQLDAPToken~40~40~40~40~2FUserAttribute~5B~40ldap~3AtargetAttribute~3D~22userCertificate~22~5D Select String: /UserAttribute[@ldap:targetAttribute="userCertificate"]

Status: OK Location Cookie: com.novell.nidp.liberty.wsc.WSCResourceOffering Value: <ldap:UserAttribute(urn:novell:ldap:2006-02)>: Id: exss80z7w0v4i Target Attribute: userCertificate // missing "Value: *****" field




• Catalina.out file shows values returned but masked (!)• Check AG Loopback interface for values returned

‒ Tcpdump -i any -s 0 -w IIValues.cap port 8080‒ See values for all requested attributes BUT ldap

UserCertificate is blank




Check IDP log for userCertificate parameter values<ldap:Query(urn:novell:ldap:2006-02)>:ns=urn:novell:ldap:2006-02 nspfx=ldap id=exss814edf549 itemId=exss814f5d44a <ldap:ResourceID(urn:novell:ldap:2006-02)>: Text: http://idpcluster.lab.novell.com:8080/nidp/?rsid%3D147.2.16.109%26sess%3D9C1CD281A9B0B6B68D8F65EE10B09A0F%26ugid%3D810de4119743d711a8d400c04fb1d4e2%26tpid%3Dhttp%3A%2F%2Fwww.mylag.com%3A80%2Fnesp%2Fidff%2Fmetadata%26auth%3DLDAPLDAPV.1.0%26svc%3Durn%3Anovell%3Aldap%3A2006-02%26ulid%3DnbYvdXIvClJdw7bimcu%2B55jOvOqVxr3jPVwIAA%3D%3D%26OB%3Dfalse <ldap:QueryItem(urn:novell:ldap:2006-02)>:id=exss814f1jf4b itemId=NEPXurn~3Anovell~3Aldap~3A2006-02~2Fldap~3AUserAttribute~40~40~40~40WSCQLDAPToken~40~40~40~40~2FUserAttribute~5B~40ldap~3AtargetAttribute~3D~22userCertificate~22~5D includeCommonAttributes=false <ldap:Select(urn:novell:ldap:2006-02)>:Select String: /UserAttribute[@ldap:targetAttribute="userCertificate"]

<ldap:QueryResponse(urn:novell:ldap:2006-02)>:ns=urn:novell:ldap:2006-02 nspfx=ldap itemIdRef=exss814f5d44a timeStamp=2007-02-05T10:49:31Z <ldap:Status(urn:novell:ldap:2006-02)>:code=ldap:OK <ldap:Data(urn:novell:ldap:2006-02)>: itemIdRef: NEPXurn~3Anovell~3Aldap~3A2006-02~2Fldap~3AUserAttribute~40~40~40~40WSCQLDAPToken~40~40~40~40~2FUserAttribute~5B~40ldap~3AtargetAttribute~3D~22userCertificate~22~5D

<ldap:UserAttribute(urn:novell:ldap:2006-02)>: Id: exss80z7w0v4i Target Attribute: userCertificate <Neil> No value returned!




Check LDAP traffic with User store for userCertificate request/response



Policy Case Study–Solution

• Confirmed that LDAP sent requested info to IDP • Confirmed that IDP sent the AG a resulting NULL for

the requested attribute• Concluded that IDP did not handle response from

LDAP correctly‒ No values displayed

• Identified issue with IDP server's inability to handle base64 encoded format of data returned

‒ Bug in Novell® Access Manager™



Key Component Folders (Linux)

• Apache‒ /opt/novell/apache2 (apache binaries/certs)‒ /etc/opt/novell/{apache2,amlogging,ag,activemmq,tomcat7} (config)‒ /opt/novell/accessgateway (AGS binaries incl. install logs) ***‒ /opt/novell/activemq (ActiveMQ binaries, J2EE env)‒ /opt/novell/nam/mag/webapps/agm‒ /opt/novell/nam/mag/webapps/nesp‒ /opt/novell/devman/jcc



Configuration Files (Linux)• Apache

‒ /etc/opt/novell/ag/‒ /etc/opt/novell/apache2/conf‒ /etc/opt/novell/apache2/conf/vhosts.d/

• Gateway Manager‒ /opt/novell/nam/mag/webapps/agm/WEB-INF/agm.properties

• JCC‒ /opt/novell/devman/jcc/conf/settings.properties

• Audit/Logging‒ /etc/logevent.conf‒ /etc/opt/novell/amlogging/config/log4j.xml



Configuration Files (Windows)

• For windows, all key components are deployed under single folder “C:\Program Files\Novell\”

• There are corresponding files for every component mentioned above under the above path


Troubleshooting Apache based Access Gateway



AG Architecture



AG Troubleshooting Logs• /var/log/novell-apache2/rcnovell-apache2.out

‒ Apache startup messages (N/A for Windows)

• /var/opt/novell/amlogging/logs/ags_error.log‒ NAM specific Apache startup messages and configuration updates

• /var/log/novell-apache2/access_log or extended_log ‒ Uses CommonLog or ExtendedLog module from apache

• /var/opt/novell/nam/logs/nesp/tomcat/catalina.out‒ Authentication, Policy, clustering and SOAP traffic logs



AG Logs - error_log• /var/log/novell-apache2/error_log

‒ Httpd logs for GET/Response traffic from browsers here. Most of the logs will be here.

‒ General apache errors so google (beware of what you focus on)



AG Tools – Advanced Logging options• Advanced options

‒ Adds custom NAM level logging to error_log for each request‒ /var/log/novell-apache2/error_log

‒ Can directly modify httpd.conf with these lines at the end and restart‒ /etc/opt/novell/apache2/conf/ directory



AG Tools – NAGVia Advanced Option• Using Via Header to track EventID in logs

‒ Browser side Via header includes eventID for specific request

‒ Useful for tracking request info in server side Proxy and ESP logs

[Thu Mar 22 21:31:33 2012] ../mod_auth_liberty.c(267): AMEVENTID#47: req: /neil/formfill/phpinfo.php (phpinfo.php/php) 47. [Thu Mar 22 21:31:33 2012] AM#504600000: AMDEVICEID#ag-F19ED707F71F50C9: AMAUTHID#: AMEVENTID#47: Requ: GET https://www.proxy163.com/neil/formfill/phpinfo .php matched service:neil (149.44.133.124:2659->164.99.185.172:443) [Thu Mar 22 21:31:33 2012] AM#504600000: AMDEVICEID#ag-F19ED707F71F50C9: AMAUTHID#157ED3F07A5854F0F7EBB6430AD6131D: AMEVENTID#47: local user. [Thu Mar 22 21:31:33 2012] AM#504600100: AMDEVICEID#ag-F19ED707F71F50C9: AMAUTHID#157ED3F07A5854F0F7EBB6430AD6131D: AMEVENTID#47: Restricted URL [Thu Mar 22 21:31:33 2012] AM#504600000: AMDEVICEID#ag-F19ED707F71F50C9: AMAUTHID#157ED3F07A5854F0F7EBB6430AD6131D: AMEVENTID#47: matched PR:neil [Thu Mar 22 21:31:33 2012] AM#504600000: AMDEVICEID#ag-F19ED707F71F50C9: AMAUTHID#157ED3F07A5854F0F7EBB6430AD6131D: AMEVENTID#47: Configured contract name :name/password/uri push Activity to agscd MatchedContract: name/password/uri [Thu Mar 22 21:31:33 2012] AM#504600005: AMDEVICEID#ag-F19ED707F71F50C9: AMAUTHID#157ED3F07A5854F0F7EBB6430AD6131D: AMEVENTID#47: sending ACL eval req



AG Tools – NAGGlobalOptions

‒ AppendProviderID: Appends Provider ID to log/audit entries‒ DebugFormFill: Enables printing of FormFill debug messages to the error

log and depending on the form size (< 4k, consequence of the form being streamed) X-Mag headers

‒ DebugHeaders: The X-Mag header is added to responses‒ ESP_Busy_Threshold: Proxy starts sending errors to the browser if eSP's

response time is more than the specified value. ‒ NoUrlNormalize: URL's are not normalized (effects PBMH, PR, FF, and RW

matching)‒ mangleCookies: Invalidates web server cookies on logout‒ InPlaceSilent:enables SSO to certain Web sites that require the login page

to remain as is without any modifications to its structure‒ RewriterEncodePath: When injecting paths in document on PBMH strip

accelerators, multi-byte char (UTF-8) paths will be url encoded (%XX)



AG Tools : X-MAG headers• Short descriptions about the processing path.



AG Tools - Server Status

‒ Http://localhost:8181/server-status‒ Gives web based real time statistics from httpd & mag module‒ Number of free network slots that can serve (idle servers)‒ Server generation‒ Uptime‒ Traffic data



AG Tools - Server Status (cont)



AG Tools - Server Status (cont)‒ What each slot status

‒ Waiting, writing, reading, Logging, Keepalive, Closing

‒ Load statistics (connections & requests per second)‒ Last served request url‒ Service statistics (balancer details)‒ ESP statistics

‒ Speed Limit: number of milliseconds to wait before marking esp as bad

‒ Actual ESP speed (avg of 100 response time)

‒ How many tickets (busted)

‒ How many credits it gets for each failed request

‒ Users currently connected‒ Cache statistics



Troubleshooting slow performance

• HTTPWatch output with timestamps• LAN traces with private keys• Check whether ESP is slow (from server-status or enable

access log in server.xml)• Check whether backend is slow• Enable extended logs and check response time logged• X-mag header FP4 gives the response time for each

request.



Troubleshooting Identity Injection• Confirm X-MAG headers and Via eventID

[Wed Apr 11 18:03:39 2012] AM#504600006 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D01917825E18794C231F29E: AMEVENTID#7: status:200 GET http://nam32app-vm.lab.novell.com/formfill/phpinfo.php <01001100952c859be154910448048a2eca59cdc4> X-Mag: 45B6586EB94FC2A7;ca59cdc4;7;usrLkup->0;usrBase->0;LocUsr;ConfigII->120;configACL->186;NoPol;ConfigFF->251;formfill-pr;Contract-valid->251;usrPr->252;Allow->252;aud->252;nam32vm-pxy-srvc;EvalII->296;CHd;AH;QS;FP2->297;WS=a1b14cc2;default;FP4->305;C005;

• Confirm HTTP headers sent (error_log or httpheaders)Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: GET /formfill/phpinfo.php?X-Roles=authenticated HTTP/1.1Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: Host: ncsles10.lab.novell.comApr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101:Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-client-IP: 149.44.133.155Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Auth-Cont: name/password/uriApr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-mail: [email protected] Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: Authorization: Basic bmNhc2hlbGw6bm92ZWxsApr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-For: 149.44.133.155Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-Host: ncsles10.lab.novell.comApr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-Server: nam32app-vm.lab.novell.com

• Confirm ESP evaluation<amLogEntry> 2012-04-11T17:03:39Z DEBUG NIDS Application: AM#501103050: AMDEVICEID#esp-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D01917825E18794C231F29E: PolicyID#N885856P-48PP-9NN7-1K15-N8O6NOP040L2: NXPESID#7: AGIdentityInjection Policy Trace: ~~RL~1~~~~Rule Count: 4~~Success(67) ~~RU~RuleID_1333023361576~Identity-Inj-All-Pol~DNF~~0:1~~Success(67) ~~PA~ActionID_1333023364340~~Inject Auth Header~uid~uid(1):CredentialProfile(7010:):NEPXurn~3Anovell~3Acredentialprofile~3A2005-03~2Fcp~3ASecrets~2Fcp~3ASecret~2Fcp~3AEntry~40~40~40~40WSCQSSToken~40~40~40~40~2Fcp~3ASecrets~2Fcp~3ASecret~5Bcp~3AName~3D~22LDAPCredentials~22~5D~2Fcp~3AEntry~5Bcp~3AName~3D~22UserName~22~5D:~Ok:ttl -1~Success(0) ~~PA~ActionID_1333023364340~~Inject AuthHeader~password~pwd(1):CredentialProfile(7010:):NEPXurn~3Anovell~3Acredentialprofile~3A2005-03~2Fcp~3ASecrets~

• 2Fcp~3ASecret~2Fcp~3AEntry~40~40~40~40WSCQSSToken~40~40~40~40~2Fcp~3ASecrets~2Fcp~3ASecret~5Bcp~3AName~3D~22LDAPredentials~22~5D~2Fcp~3AEntry~5Bcp~3AName

• ~3D~22UserPassword~22~5D:~Ok~Success(0)


mailto:[email protected]


Troubleshooting Formfill issues• STRACE output to look at the form details

‒ Could get LAN traces with private keys

• Confirm policy evaluated in the catalina log file‒ Search for 'AGFormFill Policy Trace' or 'NXPESID#EventIDNumber'

• Enable 'NAGGlobalOptions DebugFormFill=on' Advanced Option

‒ X-mag header FP4 gives the response time for each request

[Wed Apr 11 17:43:08 2012] AM#504600006 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D01917825E18794C231F29E: AMEVENTID#7: status:200 GET http://nam32app-vm.lab.novell.com/formfill/bootcamp.htm <01001100952c859be154910448048a2eca59cdc4> X-Mag:45B6586EB94FC2A7;ca59cdc4;7;usrLkup->0;usrBase->0;LocUsr;NoPol;ConfigII->116;NoPol;configACL->174;ConfigFF->236;EvalFF->247;formfill-bootcamp-pr;Contract-valid->247;mastercdnFormfill-Pol-bootcamp3310;FF4GUD->267;FillSilent;Match FormName;Match;username;Miss;title;Match;password;Miss;ldap;FF4End->267;FP4->267;

• Enable NAGGlobalOptions InPlaceSilent


MAG case study- Users receive looping error on

browser acessing protected resource



Problem Description• New application rolled out behind cluster of MAGs• Some users erroring with following browser message

• Seems to appear when domains were switched‒ Going to ESP proxy did not show issue for same user



Problem Description• No problem bypassing the L4 but going through the L4, we

saw the following symptoms with cookies



Troubleshooting steps• Enabled ModVia header and ran Apache in debug mode• From HTTP headers were able to identify the EventID

(notes)• Followed eventID to the Apache error_log

[Thu Apr 12 08:36:57 2012] ../mod_auth_liberty.c(267): AMEVENTID#132: req: /secure/auth/l/acct/summary_accounts.aspx (summary_accounts.aspx/aspx) 132.[Thu Apr 12 08:36:57 2012] AM#504600000: AMDEVICEID#ag-6A06500C95167F96: AMAUTHID#: AMEVENTID#132: Requ: GET https://testaccel.wisconsinpublicservice.com/secure/auth/l/acct/summary_accounts.aspx matched service:testaccel_wisconsinpublicservice_com (10.120.200.98:50518->10.2.57.16:443)[Thu Apr 12 08:36:57 2012] AM#304600404: AMDEVICEID#ag-6A06500C95167F96: AMAUTHID#: AMEVENTID#132: IP mismatch. IP in Cookie:\n\x02\x03D. clientAddr:\nx\xc8b[Thu Apr 12 08:36:57 2012] AM#504600000: AMDEVICEID#ag-6A06500C95167F96: AMAUTHID#: AMEVENTID#132: sending status code:302[Thu Apr 12 08:36:57 2012] AMEVENTID#132: GET https://testaccel.wisconsinpublicservice.com/secure/auth/l/acct/summary_accounts.aspx <010018000a0203441ce48 de576b1c13526809bed> X-Mag: 6A06500C95167F96;0bbeaf67;132;usrLkup->0;usrLkup->0;usrBase->0;FP4->0;



Resolution• Cookie validation process failed on IP address• Based on error message in logs, the L4 had SNAT enabled

‒ Source IP address was changing‒ Reason why bypassing L4 worked

• Enabled Advanced option “NAGGlobalOptions NAGErrorOnIPMismatch=on”‒ Disables IP address check on cookie

• Previously enabled on the LAG but migration steps to convert touch files to Advanced options not carried out


Demo tracking user session from login to logout



Additional Reading• Avoiding performance issues with Java

‒ http://www.novell.com/communities/node/9321/how-configure-access-gateway-embedded-service-provider-reduce-access-gateway-load-and-impr

• Troubleshooting 100101044/43 errors‒ http://www.intl.novell.com/communities/node/2297/troubleshoo

ting-100101043-and-100101044-errors-access-manager

• Troubleshooting SAML ‒ http://www.intl.novell.com/communities/node/2303/configuring-

and-troubleshooting-saml-11-novell-access-manager

• Troubleshooting SSLVPN‒ http://www.intl.novell.com/communities/node/3071/troubleshoo

ting-sslvpn

• SSLVPN Architecture‒ http://www.intl.novell.com/communities/node/2974/ssl-vpn-

architecture



Additional Reading

• Troubleshooting formfill issues‒ http://www.novell.com/support/php/search.do?

cmd=displayKC&docType=kc&externalId=7002780&sliceId=1&docTypeID=DT_TID_1_1&dialogID=39679063&stateId=0%200%2039677453

• SAML cool solutions on Concur (1.1), GoogleApps (2.0 IDP), Shibboleth (2.0 SP)

• LAG/MAG differences• Error codes from the product documentation

‒ http://www.novell.com/documentation/novellaccessmanager/pdfdoc/errorcodes/errorcodes.pdf


Best Practices



Admin Console best practices‒ Place internally and not in DMZ‒ Avoid mapping Liberty personal or employee profile attributes,

Secret Store attributes or or using persistent federations



IDP Best practices‒ Memory allocation for Java heap (tomcat7.conf)‒ MaxThreads (web.xml)‒ Cipher settings (server.xml)‒ MaxHTTPHeaders (server.xml)‒ Secure/HTTPOnly flag enabled on cookies‒ Persistence on load balancer‒ Disable write operations on LDAP server to avoid replication


+1 713.548.1700 (Worldwide)888.323.6768 (Toll-free)[email protected]

Worldwide Headquarters1233 West Loop South Suite 810 Houston, TX 77027 USA

http://community.netiq.com



mailto:[email protected]

http://www.netiq.com/

http://community.netiq.com/

http://www.facebook.com/netiq

http://www.twitter.com/netiq

http://www.linkedin.com/groups?gid=2745833

http://community.netiq.com/

This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. NetIQ Corporation may make improvements in or changes to the software described in this document at any time.

Copyright © 2011 NetIQ Corporation. All rights reserved.

ActiveAudit, ActiveView, Aegis, AppManager, Change Administrator, Change Guardian, Compliance Suite, the cube logo design, Directory and Resource Administrator, Directory Security Administrator, Domain Migration Administrator, Exchange Administrator, File Security Administrator, Group Policy Administrator, Group Policy Guardian, Group Policy Suite, IntelliPolicy, Knowledge Scripts, NetConnect, NetIQ, the NetIQ logo, PSAudit, PSDetect, PSPasswordManager, PSSecure, Secure Configuration Manager, Security Administration Suite, Security Manager, Server Consolidator, VigilEnt, and Vivinet are trademarks or registered trademarks of NetIQ Corporation or its subsidiaries in the United States.


Troubleshooting NetIQ Access Manager 3.2™

Neil CashellTechnical Support [email protected]


22


3© 2011 NetIQ Corporation. All rights reserved.Networking Tools•netstat -patune –connection and stat info•tcpdump/wireshark•netcat•tcp stats:‒general tcp/udp stats /proc/net/snmp•Ethtool (-S, -K TSO)•ipsysctl TCP settings•iptables (-t nat -nvL) – make sure firewall not blocking data; redirecting ports; masquerading


4© 2011 NetIQ Corporation. All rights reserved.Networking Tools•CurrPorts (Windows port monitor)‒http://www.nirsoft.net/utils/cports.html•PsTools (Windows process tools)‒http://technet.microsoft.com/en-us/sysinternals/bb896649


3.2 /opt/novell/nam/adminconsole/conf/server.xml:

<Connector NIDP_Name="devman" port="8444" maxThreads="200" minSpareThreads="5" enableLookups="false" acceptCount="100" scheme="https" secure="true" disableUploadTimeout="true" URIEncoding="utf-8" clientAuth="true" sslProtocol="tls" sslImplementationName="com.novell.socket.DevManSSLImplementation" keystoreFile="/var/opt/novell/novlwww/devman.keystore" keystorePass="6F9DFCB045EA7D18" SSLEnabled="true" address="147.2.16.135" />

3.1 /etc/opt/novell/tomcat5/server.xml:

Single instance of tomcat


/opt/novell/nam/idp/conf/tomcat7.conf




Access Gateway Group(Load Balanced by L4 Switch)

Access Manager Advanced OverviewExisting Session with Web Single-Sign-On and Access Gateway Cluster

IdentityServer

1

Identity Store

Web Servers

6

AG2

AG1Web

Browser

1. User Accesses protected resource on AG1 and the browser presents AG1 a valid Access Gateway session cookie created earlier by AG2 for this user session.

2. AG1 doesn't have a session for the user so it asks the other AGs in the Group to see if they have a session for the user3. AG2 responds claiming ownership for the user session

4. AG1 asks AG2 for the policy and user data required for the user to access the protected resource5. AG2 requests policy and user data from the Identity Server (if it isn't cached)6. The Identity Server gets the user data from the Identity Store (if it isn't cached)

9. AG1 processes the policy and user data and allows access to the protected resource

7. Identity Server responds to AG2 with the policy and user data8. AG2 responds to AG1 with the policy and user data

22, 32, 3, 4

55, 7

2, 3, 4, 8

9

Assume User already had active session on AG2.



Identity Server Group(Load Balanced by L4 Switch)

IDS1

IDS2

Access Gateway Advanced OverviewNew session with Web Single Sign-On and Identity Server Cluster

AccessGateway

1

Identity Store

Web Servers

2

AG1

1. User accesses protected resource on AG1

2. AG1 redirects users browser to IDS1 which presents a web login form requesting a username and passwd

3. IDS1 verifies the username and password against the Identity Store

4. IDS1 redirects browser back to AG1 with auth artifact

5. AG1 requests from IDS2 an authentication artifact validation and policy/user data required to access the protected resource

6. IDS2 doesn't have a session for the user so it asks the other IDSes in the Group if they have a session for the user and IDS1 responds claiming ownership for the session

7. IDS2 forwards request to IDS1

8. IDS1 validates artifact and obtains user data from Identity Store

9. IDS1 responds to IDS2 with the artifact validation and policy/user data

10. IDS2 responds to AG1 with the artifact validation and policy/user data

11. AG1 creates AG session for the new user, processes policies and user data, and allows access to the protected resource

4

5

33, 8

11


66, 76, 7, 9

5, 10


Via HTTP header reported in ieHTTPHeader output:

Via: 1.1 nam32app-vm.lab.novell.com (Access Gateway-ag-45B6586EB94FC2A7-183)


#define MULTI_HOME "01" //Multi-homing #define SM "02" //Service Manager#define REQ_PRO "03" //Request Processing#define AUTH "04" //Authentication #define AUTHOE "05" //Authorization #define II_L "06" //ID-Injection#define FF "07" //Form Fill#define CACHE "08" //Cache#define RES_PRO "09" //Response Processing#define REWR "11" //Rewriting#define SOAP_CH "12" //Soap Channel #define VCC_L "13" //VCC#define VM "14" //VM#define CM "15" //Connection Manager



Role based Policy – IDP server logs

● Policy Completion Codes

00 – Success, policy completed 64 – Policy rendered Permit action65 – Policy rendered Deny action67 – Policy rendered No Action68 – Condition returned false69 – Condition returned true70 – Condition Error...did not evaluate correctly

● Failed case (user title!=NTS)

<amLogEntry> 2012-04-10T14:53:12Z INFO NIDS Application: AM#500199050: AMDEVICEID#DB7471BE99DE2C40: AMAUTHID#B09F628189A32F1B5619C0E6504739D3: IDP RolesPep.evaluate(), policy trace: ~~RL~1~~~~Rule Count: 1~~Success(67) ~~RU~RuleID_1334056657971~ManagerRole~DNF~~1:1~~Success(67) ~~CS~1~~ANDs~~2~~False(68) ~~CO~1~SelectedLdapGroup(66455):hidden-param:hidden-value:~ldap-group-is-member-of~LdapGroup(6645):no-param:hidden-value:~~~True(69) ~~CO~2~LdapAttribute(6647):NEPXurn~3Anovell~3Aldap~3A20062~2Fldap~3AUserAttribute~40~40~40~40WSCQLDAPToken~40~40~40~40~2FUserAttribute~5B~40ldap~3AtargetAttribute~3D~22title~22~5D:hiddenvalue:~com.novell.nxpe.condition.NxpeOperator@string-equals~(0):hidden-param:hidden-value:~~~False(68) </amLogEntry>

<amLogEntry> 2012-04-10T14:53:12Z INFO NIDS Application: AM#500105013: AMDEVICEID#DB7471BE99DE2C40: AMAUTHID#B09F628189A32F1B5619C0E6504739D3: Authenticated user cn=ncashell,o=novell in User Store Camenbert with roles "authenticated". </amLogEntry>


34© 2011 NetIQ Corporation. All rights reserved.Authz/II/FF based Policy – AG logs•EventID used to track policy 'Evaluation' in catalina log file‒Retrieve EventID from Via HTTP header •Authorization policy contains conditions and actions‒Log entries on LAG similar to the IDP role entries•Formfill and Identity Injection policies‒GetAttributeRequest SOAP request to ESP‒Important to send attributes in Assertion for performance


Troubleshooting Case Study:Single sign-on to back-end app fails with Identity Injection


46© 2011 NetIQ Corporation. All rights reserved.Key Component Folders (Linux)•Apache‒/opt/novell/apache2 (apache binaries/certs)‒/etc/opt/novell/{apache2,amlogging,ag,activemmq,tomcat7} (config)‒/opt/novell/accessgateway (AGS binaries incl. install logs) ***‒/opt/novell/activemq (ActiveMQ binaries, J2EE env)‒/opt/novell/nam/mag/webapps/agm‒/opt/novell/nam/mag/webapps/nesp‒/opt/novell/devman/jcc


47© 2011 NetIQ Corporation. All rights reserved.Configuration Files (Linux)•Apache‒/etc/opt/novell/ag/‒/etc/opt/novell/apache2/conf‒/etc/opt/novell/apache2/conf/vhosts.d/•Gateway Manager‒/opt/novell/nam/mag/webapps/agm/WEB-INF/agm.properties•JCC‒/opt/novell/devman/jcc/conf/settings.properties•Audit/Logging‒/etc/logevent.conf‒/etc/opt/novell/amlogging/config/log4j.xml

Config files pushed from AC: - /opt/novell/nam/mag/webapps/agm/WEB-INF/config/current - /opt/novell/nam/mag/webapps/agm/WEB-INF/config/pending

Config file for agm: - /opt/novell/nam/mag/webapps/agm/WEB-INF/agm.properties


48© 2011 NetIQ Corporation. All rights reserved.Configuration Files (Windows)•For windows, all key components are deployed under single folder “C:\Program Files\Novell\”•There are corresponding files for every component mentioned above under the above path


Troubleshooting Apache based Access Gateway


51© 2011 NetIQ Corporation. All rights reserved.AG Troubleshooting Logs•/var/log/novell-apache2/rcnovell-apache2.out‒Apache startup messages (N/A for Windows) • /var/opt/novell/amlogging/logs/ags_error.log‒NAM specific Apache startup messages and configuration updates•/var/log/novell-apache2/access_log or extended_log ‒Uses CommonLog or ExtendedLog module from apache•/var/opt/novell/nam/logs/nesp/tomcat/catalina.out‒Authentication, Policy, clustering and SOAP traffic logs


52© 2011 NetIQ Corporation. All rights reserved.AG Logs - error_log•/var/log/novell-apache2/error_log‒Httpd logs for GET/Response traffic from browsers here. Most of the logs will be here.‒General apache errors so google (beware of what you focus on)

Adding debug startup flag adds more messages in here

The error messages look similar to the following:

[<time and date stamp>] [warn] Init: SSL server IP/port conflict:dbmhnsnetid.dsm.cit.novell.com:443 (C:/ProgramFiles/Novell/apache/conf/vhosts.d/dbmhNS-NetID.conf:18) vs.magwin1430external.dsm.cit.novell.com:443 (C:/ProgramFiles/Novell/apache/conf/vhosts.d/magMaster.conf:18)

[<time and date stamp>] [warn] Init: SSL server IP/port conflict:magdbmheguide.dsm.cit.novell.com:443 (C:/ProgramFiles/Novell/apache/conf/vhosts.d/dbmhMagEguide.conf:18) vs.magwin1430external.dsm.cit.novell.com:443 (C:/ProgramFiles/Novell/apache/conf/vhosts.d/magMaster.conf:18)

You can ignore these errors because the Access Gateway Service knows how to handle the traffic and send the packets to the correct proxy service.


53© 2011 NetIQ Corporation. All rights reserved.AG Tools – Advanced Logging options•Advanced options‒Adds custom NAM level logging to error_log for each request‒/var/log/novell-apache2/error_log‒Can directly modify httpd.conf with these lines at the end and restart‒/etc/opt/novell/apache2/conf/ directory







54© 2011 NetIQ Corporation. All rights reserved.AG Tools – NAGVia Advanced Option•Using Via Header to track EventID in logs‒Browser side Via header includes eventID for specific request‒Useful for tracking request info in server side Proxy and ESP logs[Thu Mar 22 21:31:33 2012] ../mod_auth_liberty.c(267): AM EVENTID#47: req: /neil/formfill/phpinfo.php (phpinfo.php/php) 47. [Thu Mar 22 21:31:33 2012] AM#504600000: AMDEVICEID #ag-F19ED707F71F50C9: AMAUTHID#: AMEVENTID#47: Requ: GET https://www.proxy163.com/n eil/formfill/phpinfo .php matched service:neil (149.44.133.124:2659->164.99.1 85.172:443) [Thu Mar 22 21:31:33 2012] AM#504600000: AMDEVICEID #ag-F19ED707F71F50C9: AMAUTHID#157ED3F07A5854F0F7EBB6430AD6131D: A MEVENTID#47: local user. [Thu Mar 22 21:31:33 2012] AM#504600100: AMDEVICEID #ag-F19ED707F71F50C9: AMAUTHID#157ED3F07A5854F0F7EBB6430AD6131D: A MEVENTID#47: Restricted URL [Thu Mar 22 21:31:33 2012] AM#504600000: AMDEVICEID #ag-F19ED707F71F50C9: AMAUTHID#157ED3F07A5854F0F7EBB6430AD6131D: A MEVENTID#47: matched PR:neil [Thu Mar 22 21:31:33 2012] AM#504600000: AMDEVICEID #ag-F19ED707F71F50C9: AMAUTHID#157ED3F07A5854F0F7EBB6430AD6131D: A MEVENTID#47: Configured contract name :name/password/uri push Activity to agscd MatchedContra ct: name/password/uri [Thu Mar 22 21:31:33 2012] AM#504600005: AMDEVICEID #ag-F19ED707F71F50C9: AMAUTHID#157ED3F07A5854F0F7EBB6430AD6131D: A MEVENTID#47: sending ACL eval req







55© 2011 NetIQ Corporation. All rights reserved.AG Tools – NAGGlobalOptions ‒AppendProviderID: Appends Provider ID to log/audit entries‒DebugFormFill: Enables printing of FormFill debug messages to the error log and depending on the form size (< 4k, consequence of the form being streamed) X-Mag headers‒DebugHeaders: The X-Mag header is added to responses‒ESP_Busy_Threshold: Proxy starts sending errors to the browser if eSP's response time is more than the specified value. ‒NoUrlNormalize: URL's are not normalized (effects PBMH, PR, FF, and RW matching)‒mangleCookies: Invalidates web server cookies on logout‒InPlaceSilent:enables SSO to certain Web sites that require the login page to remain as is without any modifications to its structure‒RewriterEncodePath: When injecting paths in document on PBMH strip accelerators, multi-byte char (UTF-8) paths will be url encoded (%XX)


57© 2011 NetIQ Corporation. All rights reserved.AG Tools - Server Status‒Http://localhost:8181/server-status‒Gives web based real time statistics from httpd & mag module‒Number of free network slots that can serve (idle servers)‒Server generation‒Uptime‒Traffic data


58© 2011 NetIQ Corporation. All rights reserved.AG Tools - Server Status (cont)

http://httpd.apache.org/docs/2.0/mod/mod_status.html

The number of worker serving requests The number of idle worker The status of each worker, the number of requests that worker has performed and the total number of bytes

served by the worker (*) A total number of accesses and byte count served (*) The time the server was started/restarted and the time it has been running for Averages giving the number of requests per second, the number of bytes served per second and the

average number of bytes per request (*) The current percentage CPU used by each worker and in total by Apache (*) The current hosts and requests being processed (*)


59© 2011 NetIQ Corporation. All rights reserved.AG Tools - Server Status (cont)‒What each slot status‒Waiting, writing, reading, Logging, Keepalive, Closing‒Load statistics (connections & requests per second)‒Last served request url‒Service statistics (balancer details)‒ESP statistics‒Speed Limit: number of milliseconds to wait before marking esp as bad‒Actual ESP speed (avg of 100 response time)‒How many tickets (busted)‒How many credits it gets for each failed request‒Users currently connected‒Cache statistics



Troubleshooting slow performance

• HTTPWatch output with timestamps• LAN traces with private keys• Check whether ESP is slow (from server-status or enable

access log in server.xml)• Check whether backend is slow• Enable extended logs and check response time logged• X-mag header FP4 gives the response time for each

request.



Troubleshooting Identity Injection• Confirm X-MAG headers and Via eventID

[Wed Apr 11 18:03:39 2012] AM#504600006 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D01917825E18794C231F29E: AMEVENTID#7: status:200 GET http://nam32app-vm.lab.novell.com/formfill/phpinfo.php <01001100952c859be154910448048a2eca59cdc4> X-Mag: 45B6586EB94FC2A7;ca59cdc4;7;usrLkup->0;usrBase->0;LocUsr;ConfigII->120;configACL->186;NoPol;ConfigFF->251;formfill-pr;Contract-valid->251;usrPr->252;Allow->252;aud->252;nam32vm-pxy-srvc;EvalII->296;CHd;AH;QS;FP2->297;WS=a1b14cc2;default;FP4->305;C005;

• Confirm HTTP headers sent (error_log or httpheaders)Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: GET /formfill/phpinfo.php?X-Roles=authenticated HTTP/1.1Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: Host: ncsles10.lab.novell.comApr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko/20100101:Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-client-IP: 149.44.133.155Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Auth-Cont: name/password/uriApr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-mail: [email protected] Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: Authorization: Basic bmNhc2hlbGw6bm92ZWxsApr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-For: 149.44.133.155Apr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-Host: ncsles10.lab.novell.comApr 11 18:03:39 mag32app-vm httpd: Sending to webserver for ID:7:12: X-Forwarded-Server: nam32app-vm.lab.novell.com

• Confirm ESP evaluation<amLogEntry> 2012-04-11T17:03:39Z DEBUG NIDS Application: AM#501103050: AMDEVICEID#esp-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D01917825E18794C231F29E: PolicyID#N885856P-48PP-9NN7-1K15-N8O6NOP040L2: NXPESID#7: AGIdentityInjection Policy Trace: ~~RL~1~~~~Rule Count: 4~~Success(67) ~~RU~RuleID_1333023361576~Identity-Inj-All-Pol~DNF~~0:1~~Success(67) ~~PA~ActionID_1333023364340~~Inject Auth Header~uid~uid(1):CredentialProfile(7010:):NEPXurn~3Anovell~3Acredentialprofile~3A2005-03~2Fcp~3ASecrets~2Fcp~3ASecret~2Fcp~3AEntry~40~40~40~40WSCQSSToken~40~40~40~40~2Fcp~3ASecrets~2Fcp~3ASecret~5Bcp~3AName~3D~22LDAPCredentials~22~5D~2Fcp~3AEntry~5Bcp~3AName~3D~22UserName~22~5D:~Ok:ttl -1~Success(0) ~~PA~ActionID_1333023364340~~Inject AuthHeader~password~pwd(1):CredentialProfile(7010:):NEPXurn~3Anovell~3Acredentialprofile~3A2005-03~2Fcp~3ASecrets~

• 2Fcp~3ASecret~2Fcp~3AEntry~40~40~40~40WSCQSSToken~40~40~40~40~2Fcp~3ASecrets~2Fcp~3ASecret~5Bcp~3AName~3D~22LDAPredentials~22~5D~2Fcp~3AEntry~5Bcp~3AName

• ~3D~22UserPassword~22~5D:~Ok~Success(0)



Troubleshooting Formfill issues• STRACE output to look at the form details

‒ Could get LAN traces with private keys

• Confirm policy evaluated in the catalina log file‒ Search for 'AGFormFill Policy Trace' or 'NXPESID#EventIDNumber'

• Enable 'NAGGlobalOptions DebugFormFill=on' Advanced Option

‒ X-mag header FP4 gives the response time for each request

[Wed Apr 11 17:43:08 2012] AM#504600006 AMDEVICEID#ag-45B6586EB94FC2A7: AMAUTHID#18D8F38E3D01917825E18794C231F29E: AMEVENTID#7: status:200 GET http://nam32app-vm.lab.novell.com/formfill/bootcamp.htm <01001100952c859be154910448048a2eca59cdc4> X-Mag:45B6586EB94FC2A7;ca59cdc4;7;usrLkup->0;usrBase->0;LocUsr;NoPol;ConfigII->116;NoPol;configACL->174;ConfigFF->236;EvalFF->247;formfill-bootcamp-pr;Contract-valid->247;mastercdnFormfill-Pol-bootcamp3310;FF4GUD->267;FillSilent;Match FormName;Match;username;Miss;title;Match;password;Miss;ldap;FF4End->267;FP4->267;

• Enable NAGGlobalOptions InPlaceSilent


MAG case study- Users receive looping error on browser acessing protected resource



Problem Description• New application rolled out behind cluster of MAGs• Some users erroring with following browser message

• Seems to appear when domains were switched‒ Going to ESP proxy did not show issue for same user



Problem Description• No problem bypassing the L4 but going through the L4, we

saw the following symptoms with cookies



Troubleshooting steps• Enabled ModVia header and ran Apache in debug mode• From HTTP headers were able to identify the EventID

(notes)• Followed eventID to the Apache error_log

[Thu Apr 12 08:36:57 2012] ../mod_auth_liberty.c(267): AMEVENTID#132: req: /secure/auth/l/acct/summary_accounts.aspx (summary_accounts.aspx/aspx) 132.[Thu Apr 12 08:36:57 2012] AM#504600000: AMDEVICEID#ag-6A06500C95167F96: AMAUTHID#: AMEVENTID#132: Requ: GET https://testaccel.wisconsinpublicservice.com/secure/auth/l/acct/summary_accounts.aspx matched service:testaccel_wisconsinpublicservice_com (10.120.200.98:50518->10.2.57.16:443)[Thu Apr 12 08:36:57 2012] AM#304600404: AMDEVICEID#ag-6A06500C95167F96: AMAUTHID#: AMEVENTID#132: IP mismatch. IP in Cookie:\n\x02\x03D. clientAddr:\nx\xc8b[Thu Apr 12 08:36:57 2012] AM#504600000: AMDEVICEID#ag-6A06500C95167F96: AMAUTHID#: AMEVENTID#132: sending status code:302[Thu Apr 12 08:36:57 2012] AMEVENTID#132: GET https://testaccel.wisconsinpublicservice.com/secure/auth/l/acct/summary_accounts.aspx <010018000a0203441ce48 de576b1c13526809bed> X-Mag: 6A06500C95167F96;0bbeaf67;132;usrLkup->0;usrLkup->0;usrBase->0;FP4->0;

GET /secure/auth/l/acct/summary_accounts.aspx HTTP/1.1Accept: image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/x-ms-application,

application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*

Accept-Language: en-usUser-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR

1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; InfoPath.2; MS-RTC LM 8)

Accept-Encoding: gzip, deflateHost: testaccel.dublinpublicservice.comConnection: Keep-AliveCookie: s_prevPage=HOME%20PAGE%20(default.aspx)%20%3C%20wps; s_nr=1334239048759-

New; s_cc=true; s_sq=%5B%5BB%5D%5D; IPCZQX03a36c6c0a=010022000a020344516a67eab052290026809bed

HTTP/1.1 302 FoundDate: Thu, 12 Apr 2012 13:57:29 GMTServer: ApacheSet-Cookie: IPCZQX03a36c6c0a=0; expires=Thu, 01-Jan-1970 00:00:00 GMT; path=/;

domain=.dublinpublicservice.comP3p: CP="NOI"Location: https://testaccel.dublinpublicservice.com/secure/auth/l/acct/summary_accounts.aspxContent-Length: 269Keep-Alive: timeout=300, max=98Via: 1.1 tesp.novelltestgroup.com (Access Gateway-ag-45B6586EB94FC2A7-132)Connection: Keep-AliveContent-Type: text/html; charset=iso-8859-1



Resolution• Cookie validation process failed on IP address• Based on error message in logs, the L4 had SNAT enabled

‒ Source IP address was changing‒ Reason why bypassing L4 worked

• Enabled Advanced option “NAGGlobalOptions NAGErrorOnIPMismatch=on”‒ Disables IP address check on cookie

• Previously enabled on the LAG but migration steps to convert touch files to Advanced options not carried out

GET /secure/auth/l/acct/summary_accounts.aspx HTTP/1.1Accept: image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/x-ms-application,

application/x-ms-xbap, application/vnd.ms-xpsdocument, application/xaml+xml, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*

Accept-Language: en-usUser-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR

1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0C; .NET4.0E; InfoPath.2; MS-RTC LM 8)

Accept-Encoding: gzip, deflateHost: testaccel.dublinpublicservice.comConnection: Keep-AliveCookie: s_prevPage=HOME%20PAGE%20(default.aspx)%20%3C%20wps; s_nr=1334239048759-

New; s_cc=true; s_sq=%5B%5BB%5D%5D; IPCZQX03a36c6c0a=010022000a020344516a67eab052290026809bed

HTTP/1.1 302 FoundDate: Thu, 12 Apr 2012 13:57:29 GMTServer: ApacheSet-Cookie: IPCZQX03a36c6c0a=0; expires=Thu, 01-Jan-1970 00:00:00 GMT; path=/;

domain=.dublinpublicservice.comP3p: CP="NOI"Location: https://testaccel.dublinpublicservice.com/secure/auth/l/acct/summary_accounts.aspxContent-Length: 269Keep-Alive: timeout=300, max=98Via: 1.1 tesp.novelltestgroup.com (Access Gateway-ag-45B6586EB94FC2A7-132)Connection: Keep-AliveContent-Type: text/html; charset=iso-8859-1


Demo tracking user session from login to logout


Best Practices


72© 2011 NetIQ Corporation. All rights reserved.Admin Console best practices‒Place internally and not in DMZ‒Avoid mapping Liberty personal or employee profile attributes, Secret Store attributes or or using persistent federations


73© 2011 NetIQ Corporation. All rights reserved.IDP Best practices‒Memory allocation for Java heap (tomcat7.conf)‒MaxThreads (web.xml)‒Cipher settings (server.xml)‒MaxHTTPHeaders (server.xml)‒Secure/HTTPOnly flag enabled on cookies‒Persistence on load balancer‒Disable write operations on LDAP server to avoid replication


+1 713.548.1700 (Worldwide)888.323.6768 (Toll-free)[email protected] Headquarters1233 West Loop South Suite 810 Houston, TX 77027 USAhttp://community.netiq.com 74© 2011 NetIQ Corporation. All rights reserved.


This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. NetIQ Corporation may make improvements in or changes to the software described in this document at any time.Copyright © 2011 NetIQ Corporation. All rights reserved.ActiveAudit, ActiveView, Aegis, AppManager, Change Administrator, Change Guardian, Compliance Suite, the cube logo design, Directory and Resource Administrator, Directory Security Administrator, Domain Migration Administrator, Exchange Administrator, File Security Administrator, Group Policy Administrator, Group Policy Guardian, Group Policy Suite, IntelliPolicy, Knowledge Scripts, NetConnect, NetIQ, the NetIQ logo, PSAudit, PSDetect, PSPasswordManager, PSSecure, Secure Configuration Manager, Security Administration Suite, Security Manager, Server Consolidator, VigilEnt, and Vivinet are trademarks or registered trademarks of NetIQ Corporation or its subsidiaries in the United States.


am 3.2: technical support troubleshooting and new …copying all or part of this manual, or dist...

Documents