Toronto, June 6-7 2016

IBM Connections Troubleshooting or

"get the cow off the ice"Nico Meisenzahl & Christoph Stoettner

panagenda

PLATINUM&SPOTLIGHTSPONSORS

GOLDSPONSORS

SILVERSPONSORS

BRONZESPONSORS

Christoph Stoettner

• Senior Consultant at panagenda• IBM Notes / Domino since 1999• IBM Connections since version 2.5 /

2009• Many years of experience in:

• Migrations• Administrationandinstallation• Performanceanalysis

• Joined panagenda in 2015 focusing in:• IBMConnectionsdeploymentandoptimization• IBMConnectionsmonitoring

• Husband of one & father of two, Bavarian

@stoeps linkedin.com/in/christophstoettnerwww.stoeps.de christophstoettner +491738588719christoph.stoettner@panagenda.com

Nico Meisenzahl

• Consultant at panagenda• IBM Notes / Domino since 2008• IBM Connections since version 3.0

/ 2010• Many years of experience in:

• Consulting• Migrations&Administration

• Joined panagenda in 2016 focusing in:• IBMConnectionsConsulting• ICSdeployment&optimization

@nmeisenzahl linkedin.com/in/nicomeisenzahlmeisenzahl.org nico.meisenzahl +491707355081nico.meisenzahl@panagenda.com

Agenda

• Howto: Troubleshooting• Troubleshooting…

• Connectionsitself• Backend(DB2,TDI,WebSphere,SSO)• Optionaladd-ons(Docs,FEB,Cognos,CCM)

• Tools

Toronto, June 6-7 2016

Howto: Troubleshooting

Reproduce the error

• Reproducible and/or periodically?• A sequence error?• Client-side or server-side problem?• Analyze the root cause

Be aware of the big picture

• Client-side problems• DebugindifferentBrowsers(IE,FF,Chrome)• DoNOT useaserverIE

• Server-side: IBM Connections is based on many components• debugon“highlevel”first• getanoverviewwhichbackendserviceiscausingtheerror

Configuration changes

• Changes in…• Connectionsconfiguration• Backend(WebSphere,Database,HTTP)• Firewallornetwork• OS,hardwareorVMTip:Eventhesmallestconfigurationchangecanhavebigconsequences!

Analyze log files and browser

• Analyse log files• Atom.io,Notepad++or less/tail• Baretail or tail –f• ELKstack

• Tools for client-side problems• FirebugorDeveloper-Tools• BurpSuiteorFiddler

Analyze root cause

• Find a hint inside the log• NetworktimeoutorDNS• SQLerrors• LDAPerrors• Syntaxerrorsinconfigurationfiles„xxx-config.xml“

• ErrorstackTip: In a clustered environment, start and analyze only one Node (if possible)

Find support

• Knowledge Center http://goo.gl/HFzTmv• Troubleshooting Tips https://goo.gl/mU1EQ9• IBM Connections Forum http://goo.gl/CVvQCU• Community Blogs and/or Chats• Fix Central• PMR

Toronto, June 6-7 2016

Troubleshooting

Get the cow off the ice

How to do this?

Agenda

WebSphere Application Server (IBM Connections) logs• SystemOut.log• SystemErr.log• trace.log (if tracing is enabled)• Log path:

• <wasroot>/profiles/<profilename>/logs/<servername>/

Analyze WAS log files

• Time stap: 24h time stamp with milli-seconds • Thread id: eight character hexadecimal value• Short name: typically java class name• Event type: one character only (E, W, I,…)• Message identifier: String based on component• Message: Some information

WAS Event types

• F - Fatal message• E - Error message• W - Warning message• A - Audit message• I - Informational message• C - Configuration message• D - Detail message• O - Messages that are written directly to System.out

by an application• R - Messages that are written directly to System.err by

an application• Z - Place holder to indicate type was not recognized

WAS Message identifier

• Prefix by Application or Server (CLFRW)• Specific application code (0042)• Event Type (I)

Read trace stack

• First line displays key information• “Caused by” displays root cause

Environment information

• First log lines on server startup• WebSphereversion• OSversion,Processid• Installationpath

Enable tracing

• Enable tracing using ISC• Runtimeorconfigurationonly

• Define tracing based on• Appprefix/errorstack• Mustgather(PMR)

Search issues

• http(s)://<fqdn>/search/serverStatus• Displayindex,seedlists,loginformation• Dataisdisplayedforone nodeonly

• Usingnodefqdn toaccessdifferentnodes

Debug Search

• Search queries (runtime)• com.ibm.connections.search.index.searching.*=all

• Crawling & seedlists• com.ibm.connections.search.index.indexing.*=all:com.ibm.connections.search.seedlist.*=all:com.ibm.connections.httpClient.*=all

Recreate Search Index

• SearchService.startBackgroundIndex()• Crawlsseedlists• Extractsthefilecontent• Createindex

Tip: Use „all_configured“ to index all apps

CLFRW0394E: Search indexing ofservices ...• Search index not ready

• interruptionatindexcreation• CLFRW0283E:Searchhasencounteredaproblemwhilecrawling

• CLFRW0027E:ErrorIndexingcomponent<app>forsearch

• INDEX.READYfilenotpresent• Recreateandenabletracing

com.ibm.connections.directory.services.exception.DSOutOfServiceException• Access Connections not possible• DSX not working• Check SSO settings

• Domain• LtpaToken Cookiename

Tip: WebSphere FP sometimes resets SSO domain

EJPVJ9284E: Unable to get the groups from the directory for the user…• User was not able to access Connections

anymore• WAS LDAP bind user had no read access

to one of the groups the user was member

Debug wsadmin

• Enable trace within wsadmin session• AdminControl.trace(‘com.ibm.*=all’)• <wasroot>/profiles/<profilename>/logs/wsadmin.traceout

Database connections

• Check datasouces• ISC– Resources– JDBC– Datasources• Checklogsformoreinformation

• DB2 server log• <instanceroot>/sqllib/db2dump/

• db2diag.log• db2diag.xxx.log(logrotation,youshouldenablethis!)

Tip: Oracle users have password expiration enabled by default!

HTTP Server (IHS & Plugins) logs

• IBM HTTP Server• <installroot>/logs/

• error_log• access_log• basedonconfiguration

• WebSphere AppServer Plugins• <installroot>/logs/<webserver>/http_plugin.log

HTTP 404 not found

• Outdated Plugin configuration• RestartIHS

• WAS Plugin configuration issue• http_plugin.log

HTTP 404 not found

• AppServer or App down• Network issue• http_plugin.log

HTTP 500 Internal Server Error

• Unexpected error• http_plugin.log

• Configuration issue• WAS Root certificate not trusted or missing• SSL certificate expired

SDI / TDI logs

• <tdisol>/log/ibmdi.log• TDIlogfile

• <tdisol>/employee.*• Filesincludeallchangedusers(adds,update,delete,error,skip)

• <tdisol>/syncupdates/*• Temporaryfileswithinthesync• Includingdatabasedumpandldiff• sync_updates_clean_temp_files=false(default:true)

• profiles_tdi.propertiesTip: Check lock file

Analyze SDI logs

• Error code prefix• CLFRN:Profile&Usersynchronization• CTGDIS:TDIitself

• Error code suffix• I,E,W,…

Debug SDI

• Profile&Usersynchronization(<tdisol>/etc/profiles_tdi.properties)• source_ldap_debug=true• debug_update_profile=true• debug_collect=true

• TDIissues(<tdisol>/etc/log4j.properties)• log4j.rootCategory=DEBUG,Default

Cognos BI logs

• Cognos BI• SystemOut.log &trace.log• <installroot>/logs/cogserver.log

• Cognos Transformer• <installroot>/logs/cogserver.log

• PowerCube build• <installroot>/metricsmodel/trxschelog.log• <userhome>/Transformer/Logs/*.log(winonly)

BMT-MD-6003 No connection to the data source …• PowerCubes not created yet• Check

• Cronjobs orscheduledjobs• trxschelog.log

Debug Cognos BI & Metrics

• Communication between Cognos BI and Connections Metrics• SonataHttpUsage=all:SonataHttpHeader=all:SonataHttpBody=all:com.ibm.connections.httpClient.*=all:com.ibm.connections.metrics.*=all

• Connections Metrics Servlet• com.ibm.connections.metrics.cognos.servlet.*=all

FileNet logs

• SystemOut.log & trace.log• FileNet logs

• <wasprofile>/<servername>/p8_server_error.log• <wasprofile>/<servername>/p8_server_trace.log• <wasprofile>/<servername>/pesvr_system.log• <wasprofile>/<servername>/pesvr_trace.log

FileNet urls

• http://<fqdn>/FileNet/Engine• Serverstatus,version,sonata/waltzversion

• http://<fqdn>/P8CE/Health• Healthchecksforauthentication,storesanddatabase

• http://<fqdn>/dm• FNCSversionandconfiguration

Debug CCM Widget

• Widget issues• com.ibm.quickr.communitylibrary.*=all:com.ibm.lconn.widgets.service.*=all:com.ibm.lconn.widgets.actions.*=all

• Authentication issues• com.ibm.connections.directory.services.*=all:com.ibm.connections.directory.services.*=all:com.ibm.connections.httpClient.*=all

Debug FileNet using ACCE

• http(s)://<fqdn>/acce

Debug FileNet using JVM Properties

• Add generic JVM properties• -Dlog4j.configuration=file:<path>/log4j.xml-DskipTLC=true

• Copy & customize sample log4j.xml• <ContentEngineRoot>/config/samples• 20subsystems(db,engine,security,search,…)

Docs/Viewer logs & urls

• SystemOut.log & trace.log• http(s):<fqdn>/vsanity/check• http(s):<fqdn>/sanity/check?app=all&querytype=report• http(s):<fqdn>/*/version.txt

Debug LTPA between Domino & WAS

• Debug on Domino side (notes.ini)• Debug_SSO_Trace_Level=2• Webauth_verbose_trace=1• WebSess_verbose_trace=1• Debug_outfile=<logfilepath>

• Debug on WebSphere• com.ibm.ws.security.ltpa.*=all

Debug Kerberos

• Configuration• com.ibm.ws.security.spnego.*=all:com.ibm.ws.security.*=all:com.ibm.issw.spnegoTAI.*=all:com.ibm.security.krb5.*=all

• Runtime• com.ibm.connections.httpClient.*=all:com.ibm.connections.directory.services.*=all:com.ibm.websphere.wim.*=all:com.ibm.ws.wim.*=all

• Fiddler & BurpSuite

Toronto, June 6-7 2016

Tools

Analyze logs

• Analyze logs live• Baretail• tail–f

• View logs • Atom.io,Notepad++• less,tail• ELKStack

• Elasticsearch,Logstash,Kibana• SmallDockerdeployment

Analyze Client-side

• Browser• Firebug /DeveloperTools

• Intercepting proxies• Fiddler• BurpSuite

• VMs with different IE versions• WithoutGPO• https://www.modern.ie/en-us/virtualization-tools

SQL Clients

• db2 command • IBM Datastudio• Dbeaver, DBVisualizer, squirrelSQLTip: Database write access is not supported!

LDAP Clients

• ldapsearch command• Softerra LDAP Browser• Apache Directory Studio

Analyze Network

• Wireshark• tcpdump

If you ask the * admin:There is never a problem within the *!

J*= Network, SAN, VM, ...

Useful Blogs• http://ibmconnections.com• http://turtleblog.info• http://portal2portal.blogspot.de• https://www.urspringer.de• http://socialconnections.info• http://blog.robertfarstad.com• http://www.curiousmitch.com• http://www.ramsit.com/category/blog• http://techblog.gis-ag.info• https://milanmatejic.wordpress.com• http://ibmdocs.com• http://domino.elfworld.org• https://dontforgetthe0.com

• http://dilf.me.uk/socialshazza• http://www.stoeps.de• http://scripting101.org• http://meisenzahl.org• http://martin.leyrer.priv.at• http://kbild.ch• http://www.notesgoddess.net• http://www.dominodiva.com• http://notesbusters.com• https://rob59blog.wordpress.com• http://connections101.info• http://brandlrainer.blogspot.de• https://collaborationben.com

Thank you very much for your attention!

panagenda GmbH – MakeYourDataWorkforYou

Lahnstr. 17● 64646 Heppenheim (Germany)

Skype:christophstoettner ● Cell:+49173 8588719

E-Mail:christoph.stoettner@panagenda.com

ChristophStoettnerSeniorConsultant

panagenda GmbH – MakeYourDataWorkforYou

Lahnstr. 17● 64646 Heppenheim (Germany)

Skype:nico.meisenzahl ● Cell:+49170 7355081

E-Mail:nico.meisenzahl@panagenda .com

NicoMeisenzahlConsultant

PLATINUM&SPOTLIGHTSPONSORS

GOLDSPONSORS

SILVERSPONSORS

BRONZESPONSORS