diagagent troubleshooting

SAP Solution Manager 7.0 EhP1

End-to-End DiagnosticsDiagnostics agent troubleshootingVersion 1.5, June 2009

END-TO-END DIAGNOSTICSDIAGNOSTICS AGENT TROUBLESHOOTING

Typographic Conventions Icons

Icon Meaning

Caution

Example

Note

Recommendation

Syntax

Type Style RepresentsExampleText

Words or characters quotedfrom the screen. Theseinclude field names, screentitles, pushbuttons labels,menu names, menu paths,and menu options.

Cross-references to otherdocumentation.

Exampletext

Emphasized words orphrases in body text, graphictitles, and table titles.

EXAMPLETEXT

Technical names of systemobjects. These include reportnames, program names,transaction codes, tablenames, and key concepts ofa programming languagewhen they are surroundedby body text, for example,SELECT and INCLUDE.

Exampletext

Output on the screen. Thisincludes file and directorynames and their paths,messages, names ofvariables and parameters,source text, and names ofinstallation, upgrade anddatabase tools.

Exampletext

Exact user entry. These arewords or characters that youenter in the system exactlyas they appear in thedocumentation.

<Exampletext>

Variable user entry. Anglebrackets indicate that youreplace these words andcharacters with appropriateentries to make entries in thesystem.

EXAMPLETEXT

Keys on the keyboard, forexample, F2 or ENTER.

END-TO-END DIAGNOSTICSDIAGNOSTICS AGENT TROUBLESHOOTING

Contents1 Introduction ......................................................................................................................................... 42 Pre-requisites ...................................................................................................................................... 5

2.1 OS support ................................................................................................................................... 52.2 JDK compatibility......................................................................................................................... 62.3 Managed System.......................................................................................................................... 82.4 SLD production system............................................................................................................... 9

3 Agent installation strategy................................................................................................................ 103.1 Standard environment............................................................................................................... 103.2 HA/virtualization environment .................................................................................................. 10

4 Agent installer.................................................................................................................................... 134.1 SAPinst Documentation ............................................................................................................ 134.2 Troubleshooting with SAPinst .................................................................................................. 134.3 OS configuration checks........................................................................................................... 144.4 JVM parameters......................................................................................................................... 144.5 Startup scripts ........................................................................................................................... 15

5 Agent connectivity ............................................................................................................................ 185.1 Managing system....................................................................................................................... 185.2 Managed system........................................................................................................................ 215.3 Full Qualified Name resolution issue ....................................................................................... 245.4 SAP router.................................................................................................................................. 245.5 Self-Monitoring .......................................................................................................................... 25

6 Agent Maintenance............................................................................................................................ 266.1 Upgrade SAP Kernel (saposcol update)................................................................................... 266.2 Configuration ............................................................................................................................. 26

6.2.1 Change Path of the JDK....................................................................................................... 276.2.2 Change JVM Parameters .................................................................................................... 276.2.3 Change the virtual host name............................................................................................... 276.2.4 Usage of smdsetup script..................................................................................................... 286.2.5 Change Agent log level ........................................................................................................ 31

6.3 Managing J2EE user.................................................................................................................. 327 Tracking agent issues....................................................................................................................... 33

7.1 Deprecated SAP notes .............................................................................................................. 337.2 OSS message pre-requisites..................................................................................................... 337.3 Agent crash investigation ......................................................................................................... 347.4 Thread dump.............................................................................................................................. 357.5 Activate P4 logs ......................................................................................................................... 367.6 Telnet SMD server commands.................................................................................................. 37

8 Scalability........................................................................................................................................... 378.1 Managing java stack settings.................................................................................................... 37

End-to-End Diagnostics 4Diagnostic agent troubleshooting guide

©SAP AG 2009

1 IntroductionThe goal of this guide is to help Solution Manager Users troubleshoot failures encountered when installing,configuring and using the Diagnostics Agents.

It contains the description of the resolution of the most common known issues related to the DiagnosticsAgents. It gives also information, tooltips and procedure to isolate and identify the root cause of an issuerelated to the Diagnostics Agent.

This guide is separated in different areas that correspond to the different phases of the Solution Manager lifecycle. The chapters of this document follow a logical order corresponding to the Diagnostics Agent’s life cycle.However, in order to successful troubleshoot a Diagnostics Agent’s issue, you should be familiar with all partsof this document.

This guide covers the following area: Pre-requisites: Helps users to check that all pre-requisites are fulfilled in order to complete a correct

installation of the Diagnostics Agents. Agent Installation Strategy: Answers to the most frequent questions rose during the planning phase of

Diagnostics Agent installations. Agent Installer: Lists all common issues encountered during the installation of the Diagnostics Agents. Agent Connectivity: Help users to understand and troubleshoot the connection mechanisms flow

between:o The Diagnostics and the Managing System.o The Diagnostics agent and the Managed systems.

It includes also the main resolution procedures related to network configuration issues. Agent Maintenance: Describes the procedure and common errors related to the operation of the

Diagnostics Agent. It concerns the administration of the Diagnostics Agents including a description ofthe smdsetup script.

Tracking Agent Issues: Explains where to find information related to the Diagnostics Agent activities. Itexplains how to interpret this information in order to establish a correct diagnostics whentroubleshooting an Agent issue.

Scalability: Contains scalability and performance notes related to the Diagnostics Agent.


©SAP AG 2009

2 Pre-requisites

2.1 OS support

The following table describes the list of different OS supported by the Diagnostics agent:

OS Diagnostics Agent 7.00 / 7.01 Diagnostics Agent 7.11Server 2003on IA32 32bit X X

Server 2003on IA64 64bit X XWindows

Server 2003 on x6464bit X X

5.3 64bit on Power4, 5 X X

AIX6.x 64bit on Power

4, 5 X (SAP Kernel > 133) X

11.31 on IA6464bit X X

HP11.11,

PA-RISC 64bit X X

Solaris 10 on SPARC64bit X X

IA32 32bit X XIA64 64bit X X

x86_64 64bit X XPower 64bit X X

Linux

SLES9 only onzSeries 64bit X X

I5/OS V5R3, V5R4 64bit X X

Z/OS 1.6, 1.7, 1.8, 1.964bit - -

TRU64 - -


©SAP AG 2009

2.2 JDK compatibilityThe following table describes the list of different JDK supported by the Diagnostics agent for each platform:

OS Diagnostics Agent 7.00 / 7.01 Diagnostics Agent 7.11Server 2003on IA32 32bit SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SAPJVM5 last version

Server 2003on IA64 64bit SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SAPJVM5 last versionWindows

Server 2003 on x6464bit SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SAPJVM5 last version

5.3 64bit on Power4, 5 IBM 1.4.2 SR12 caix64142-20080923 SAPJVM5 last version

AIX6.x 64bit on Power

4, 5 IBM 1.4.2 SR12 caix64142-20080923 SAPJVM5 last version

11.31 on IA6464bit HP 1.4.2.15 to 1.4.2.172 or 1.4.2.21+ SAPJVM5 last version

HP11.11,

PA-RISC 64bit HP 1.4.2.15 to 1.4.2.172 or 1.4.2.21+ SAPJVM5 last version

10 on SPARC64bit SUN 1.4.2.13 to 1.4.2.17b06 or 1.4.2_21+ SAPJVM5 last versionSolaris

10 on x86_64 SUN 1.4.2.13 to 1.4.2.17b06 or 1.4.2_21+ SAPJVM5 last versionIA32 32bit SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SAPJVM5 last versionIA64 64bit SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SAPJVM5 last version

x86_64 64bit IBM 1.4.2 SR10 j9xa64142-20080130 SAPJVM5 last versionPower 64bit IBM 1.4.2 SR12 cxp64142-20080923 SAPJVM5 last version

Linux

SLES9 only onzSeries 64bit IBM 1.4.2 SR12 cxz64142-20080923 SAPJVM5 last version

I5/OS V5R3, V5R4 64bit IBM IT4J JVM SAPJVM5 last version

Z/OS 1.6, 1.7, 1.8, 1.964bit - -

TRU64 - -

For Diagnostics agent 711, the SAPJVM 5 is included with the SAP Kernel DVD. Therefore, you don’t need to installseparately the SAPJVM5 before installing the 711 agent.

The following SAP notes can be used to determine the JDK version but do not apply the JVM parameters proposedin those sap notes since they are only specific to the sapj2ee engine.

Note 716604 for the Sun JDK (Windows, Linux, Solaris) Note 716926 for the HP JDK (HP-UX) Note 716927 for the IBM JDK (AIX) Note 717376 for the IBM JDK (iSeries, OS/400) Note 746299 for the IBM JDK (zSeries) Note 810008 for the IBM JDK (Linux on POWER) Note 861215 for the IBM JDK (Linux on AMD64/EMT46)

1 Until further notice please don't use 1.4.2.17b12, 1.4.2_18 and 1.4.2_19.

2 Until further notice please don't use 1.4.2.20 (SAP note 1301397)


©SAP AG 2009

Most common JDK problems with agent

Timeout occurred

This error is logged in SMDSystem.log

Exception: Time out occurred when calling method 'configure' on object[com.sap.smd.api.IAgentContext_Stub@49f949f9] after 10000 ms.possible cause: com.sap.smd.server.util.concurrent.TimeoutException atcom.sap.smd.server.exec.asio.AsioInvocationHandler.invoke(AsioInvocationHandler.java:130)

This timeout issue is caused either by a wrong JDK Version or wrong JVM parameters. The timeoutoccurs often on old IBM SDK or when the J9 technology is not enabled in the java parameters of theDiagnostics agent. Check also the java stack thread settings (see chapter 8.1 Managing java stacksettings)

Agent applications are not updated

This error is still logged in SMDSystem.log

Exception: [Thread[Thread-5,5,main]] Error[com.sap.smd.om.impl.broker.ObjectProxyImpl@76933bcb (states:core.patch[proxy])]: error during load of states:core.patch[EXCEPTION] com.sap.smd.om.ObjectLoadException: failed to instantiateimplementation class atcom.sap.smd.core.plugin.service.ServiceEntityHandle.entityLoad(ServiceEntityHandle.java:159)......Caused by: java.lang.UnsatisfiedLinkError: inode0 at sun.nio.ch.FileChannelImpl.inode0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:868)

This problem occurs only with HP JDK; re-install a JDK version recommended for Diagnostics agent(currently 1.4.2.17 or 1.4.2.21+).


©SAP AG 2009

2.3 Managed SystemThe following list describes the SAP systems for which the SAPinst installs automatically a Diagnostics agent at theend of the SAP instance (central or dialog) installation procedure.

System SAPinst 7.0 SR3 SAPInst 7.01 Ehp1 SAPInst 7.11 Ehp1NW 2007 Abap stack - - -NW 2007 Java stack X X -

Netweaver Double stack X X -Solution Manager X X -

NW 711 Abap stack - - (X) Not scheduled yetNW 711 Java stack - - (X) Not scheduled yet

PI 7.11 - - XCE 7.11 - - XPI 7.10 - - -

NW 7.10 Java stack - - -

If the Diagnostics Agent has not been installed automatically, you have to install it as a standalone engine using thestandard SAPinst (Please refer the Agent Setup Guide for more information).To avoid libraries conflicts it is highly recommended to install a Diagnostics agent which has the same kernel releasethan the SAP systems already installed on the host.

Ex: On a NW04 or NW04s Managed System, you should install a Diagnostics agent 700 or 701,On a 711 or 710 Managed System, you should install a Diagnostics agent 711,…

Note for System NW2004 (640), the recommendation is to install a Diagnostics agent NW2004s (700)only if no previous Diagnostics agent 640 has been installed on system.

Ex: on a host with a sap system 640, if no agent is present -> install a new diagnostics agent 700(SR3) on a host with sap system 640, if one or more agent 640 are present -> install a new 640 agent.


©SAP AG 2009

2.4 SLD production systemTo register a Diagnostics agents to a productive SLD, the CIM Model version of the productive SLD must be 1.5.26 orhigher.

How to determine which version of the CIM Model is installed? Go in SLD with the url: http://sldhostname:port/sld Logon with a J2EE user having the role SAP_SLD_ADMINISTRATOR Click on link ‘About’. The CIM Model version is displayed in a window popup.

How to upgrade the CIM Model? Refer to SAP Note 669669 Updating the SAP Component Repository in the SLD

How to check if the Diagnostics agent is registered in SLD? Go in SLD with the url: http://sldhostname:port/sld Logon with a J2EE user having the role SAP_SLD_ADMINISTRATOR Go in the section ‘Administration’ Click on link ‘Content Maintenance’ Select the subset ‘All with Instances and class ‘Diagnostics Agent Instance’

If the class ‘Diagnostics Agent Instance’ does not appear in drop down list, No Diagnostics agent is registered In column Name of table, check if the Diagnostics agent is present.

Or Go in Agent Administration tool and click on link “agent candidates” Enter the information to connect to productive SLD The list of Diagnostics agents registered in this SLD will be displayed.

Which is the J2EE Group to assign to the SLD user used by the Diagnostics Agent? The SAP_SLD_CONFIGURATOR group or the security role DataSupplierLD should be assigned to the user

used by the Diagnostics Agent.

What is the SLD registration frequency of the Diagnostics Agents? Every 12 hours, the Diagnostics agent sends an SLD request to update the agent model information.

What is the frequency used by the Diagnostics Agent to check if an association is present in SLD? If the Diagnostics agent is not connected to the Managing System and registered in SLD, it will check every 20

minutes if an association has been created in SLD. If the Diagnostics agent is still connected to Managing System and registered in SLD, it is going to check every

hour if a new association has been created in SLD.

Why the Diagnostics agent is not visible in Agent Admin tool? You need to create an SLD association between the Diagnostics agent and the Managing system. To create

this association go in Agent Admin tool, click on link “agent candidate”. Enter the SLD connection data wherethe Diagnostics agent is registered then select the agent and click on the “attach” button.


©SAP AG 2009

3 Agent installation strategy

3.1 Standard environmentThe following check list must be validated before installing a new Diagnostics agent on a managed host:

Note: Don’t install a Diagnostics Agent on a Database or an SCS host without any Central Instance. Thisis not necessary in the current E2E RCA scenarios.

Check if a Diagnostic agent has already been installed on the managed system host. Verify if the managed system has been installed with a SAPInst version greater than NW04S SR3 Check on file system if the folder /usr/sap/DA or the folder /usr/sap/SMD exist. In both cases, an agent is already installed on the system.

If the Diagnostics agent is already installed on system, check if the agent is present in the SLD server wherethe managed system is registered. In this case attach the agent to the Solution Manager System using theagent admin tool (agent candidate view) or connect the agent with the script ‘smdsetup’ (see part 6.2.4).

If the Diagnostics agent is not installed, retrieve the virtual host or physical host name of the managed system.The rule is to install one diagnostics agent per virtual host. Go in the sap profile of the managed system (/usr/sap/SID/SYS/profile). Retrieve the hostname in the profile file name. (Ex: for the value W46_JC93_NCED60185755A, the

hostname is nced60185755a). The Diagnostics agent must be installed with this hostname.

When the installation is finished, on UNIX systems add the Diagnostics agent in OS startup/shutdown script(see chapter 4.5 Startup scripts)

For UNIX system, check these Operating System limits of OS agent user (see part 4.2).

3.2 HA/virtualization environment

In High Availability environment, the Diagnostics agents must be installed on each virtual host used by the SAPInstance (Except for the hosts running the database or the Central Services).

The Diagnostics Agents must be integrated in the cluster fail-over scripts used to control the managed systems. Forthis reason, the recommendation is to install the Diagnostics agent on a shared file system.During a failover scenario, the SAP instance of the managed system and the Diagnostic agent must be movedtogether.

When the SAP instance and the agent are moved to another system node, you should mount the agentinstance partition before the SAP instance partition. This is due to the fact that the classpath of the SAP javainstance depends on some jar files contained in the agent folder for the Wily IS instrumentation.


©SAP AG 2009

Microsoft Cluster (MSCS)

With MSCS you enable critical system components, Single Points of Failure (SPOFs), to be automatically switchedfrom one machine to the other in the event of hardware or software problems, or system maintenance affecting onemachine.Until now, SAP has only supported the installation of one clustered SAP system in one MSCS cluster with two MSCSnodes.

It is necessary to install a Diagnostics agent for each central instance and all dialog instances. The Diagnostics agentsmust be installed on a local disk.

On the failover node, you should activate the windows service used to start the Diagnostics Agent.

Figure 1: ABAP or Java System with SPOFs, where non-SPOFs are installed locally on the MSCS Cluster Nodes


©SAP AG 2009

Figure 2: ABAP+Java System where the non-SPOFs are installed on hosts outside of the MS Cluster

More information is available in SAP Note 112266 - SAP and MS Cluster Server: Frequent questions and tips.

Flexframe

A FlexFrame-based server landscape for SAP usually consists of the following components:

2 highly available control nodes (without SAP software) One or more network appliance filer(s) (IO subsystem, NFS server) Application-node with Linux or Solaris operating systems. Implemented high availability of SAP components (myAMC - FlexFrame autonomous agents). Virtualization of SAP services (at least 1 IP address for each service irrespective of the server IP address). This

can result in an instance being "changed" to another server.

In addition, a check would be carried out to see whether the instance in question is already running on anotherapplication node.

On this platform, install a Diagnostics agent for each service hosted on the application nodes identified by virtual IPaddresses.

Other virtualization solutions

Please refer to SAP Note 674851 - Virtualization on Windows to get a list of supported virtualization solutions on Windows.


©SAP AG 2009

4 Agent installer

4.1 SAPinst Documentation

The following SAP Notes contain the most recent version of the ‘Agent Setup guide’ documentation and describes theproblems that could occur during the installation:

SAP Note 1234387: Inst. 7.1 EhP1 - Diagnostics Agent installationSAP Note 1234384: Inst. NW/SOLMAN 7.0 EhP1 - Diagnostics Agent installationSAP Note 1144737 Inst. NW 7.0 SR3/SOLMAN 7.0 - Diagnostics Agent installation

Don’t use the SAPInst SPS13 for SMD AGENT 7.00 or lower (SMDAGENT13_0-20001420.SAR), thisinstaller is deprecated.

If you need to install more than 20 agents, it is recommended to use the Mass Deployement option of theDiagnostics agent.

SAP Note 1172764: Mass Deployment of SAP Diagnostics Agent 7.0 & 7.01

SAP Note 1294713: Mass Deployment of SAP Diagnostics Agent 7.11.

4.2 Troubleshooting with SAPinstThis section describes how to proceed when errors occur during the installation of the Diagnostics Agent with SAPinst.If an error occurs, SAPinst will stop the installation and display a dialog informing you about the error.

In that case, the procedure is the following:

a. To view the log file, choose View Logs.

b. If an error occurs during the dialog or processing phase, do either of the following:

Try to solve the problem

Abort the installation with Exit

Continue the installation by choosing Retry.

c. Check the log files of SAPinst in the installation directory sapinst_instdir SAPinst normally createsthis installation directory directly below the temporary directory. SAPinst finds the temporary directory bychecking the value of the environment variables TEMP, TMP, or TMPDIR.

If no value is set for these variables, SAPinst uses /tmp/sapinst_instdir as default installation directoryon UNIX platforms and %ProgramFiles%\sapinst_instdir on windows platforms. If you want to use analternative installation directory, set the environment variable TEMP, TMP, or TMPDIR to the required directory.


©SAP AG 2009

4.3 OS configuration checksAfter the installation of the Diagnostics agent under UNIX platform, check the following OS parameters correspondingto the Diagnostics agent OS user (ex: smdadm)

Check the Diagnostics agent OS user limit :o Logon with the agent OS user ‘su – smdadm’o Open ‘sh ‘shell with command ‘sh’o Execute the command ‘ulimit –a’o Compete the values with the following recommended values

time(seconds) unlimitedfile(blocks) unlimiteddata(kbytes) unlimitedstack(kbytes) 4194304memory(Kbytes) unlimitedcoredump(blocks) 4108901nofiles(descriptors) 8092

If the user’s limit parameters for the OS agent user are not correct, some OutOfMemory exceptions can occurin the extractor jobs of the Diagnostics agents.

Check File System access right:

If during an extractor job (datacollect), an access denied occurs in the agent log ‘SMDApplication.log’, check ifthe folder /usr/sap/<SID_AGT>/JXX/SMDAgent/temp is writable for the sapsys group. If is not the case,execute the command ‘chmod –R 775 /usr/sap/<SID_AGT>/JXX/SMDAgent/temp’.

Furthermore the file system of the Managed system must be readable by the Diagnostics agent.Logon with an agent OS user (su – smdadm), go in the managed system folder (ex:/usr/sap/SID/SYS/profile), and open one profile with the command ‘more SID_INSTANCE_HOST’. If the“more” command is not working, the access rights are not correct for the Diagnostics agents. In that case,check with your system Administrator to adapt the file system rights.

4.4 JVM parametersIn this section, you will find the recommended Diagnostics agent’s JVM parameters for each JDK vendor.

JDK JVM Parameters SAP Note

IBM SR10 or higher -DP4ClassLoad=P4Connection -Xj9 -Xgcpolicy:gencon -Xmn50m -Xmx256m-Xms256m 1093831

SUN JDK -DP4ClassLoad=P4Connection -Xmx128m -Xms128m n.aHP JDK -DP4ClassLoad=P4Connection -Xmx128m -Xms128m n.a

SAP JVM5 -DP4ClassLoad=P4Connection -Xmx128m -Xms128m n.a

Don’t use the java parameter verbose:gc in the Diagnostics agents JVM parameters, because thisparameter will steadily increase the size of the file work/std_SMDAgent.out


©SAP AG 2009

4.5 Startup scripts

Information about Diagnostics agent 7.0x

On Windows platform the SAP MMC allows to start or stop the Diagnostics agent. The NT service of theDiagnostics agent is configured to start automatically during Windows startup.

The green status in SAP MMC only indicates that the agent is running. It doesn’t report if the agent isconnected to the Managing system. If the status is yellow after the agent start, it is probably due to the factthat the agent process was stopped abnormally (see 7.3 Agent crash investigation)

On UNIX platform, the commands smdstart.sh, smdstop.sh and smdadmin.sh allow to operate the Diagnosticsagent. Since the SR3 release of SAPInst, those commands are included in the PATH variable of the OS agentuser. In that case, the commands can be called from every folder (not only in the agent script folder).

Command Actionsmdstart.sh <SID> <InstanceNumber> Starts the Diagnostics agent and the sapstartsrv process.smdstop.sh <SID> <InstanceNumber> Stops the Diagnostics agent and the sapstartsrv process

smdadmin.sh <SID> <instanceNumber> Launches jcmon tool to administrate the Diagnostics agentinstance

On Solaris platform, an error (/usr/sap/SMD/exe/smdstart.sh: usage: ulimit [ -HSacdfnstv ] [ limit ])occurs when the smdstart.sh is executed. Check the solution in sap note 1109366.

On HP-UNIX platform, an error (!! ERRROR !! current user is not the owner of this instance )occurs when the smdstart.sh is executed. Check the solution in sap note 1109366.

The following table gives the list of the return code of the smdstart.sh script.

Exit code of smdstart.sh Reason

0 Success to launch the agent process (This is not the confirmation that the agent isconnected to SMD).

2 Input Parameters are invalid.5 Agent is already running.4 User limit are not set correctly.3 Current OS user is invalided to start the agent.

For UNIX system, Diagnostics agents do not start automatically when the system reboots because the agent startupscripts are not included in the system startup scripts.The following command line can be used in the system startup scripts:

You will find below the command line for the shutdown sequence:

su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1 -s /bin/sh - smdadm

su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null 2>&1 -s /bin/sh - smdadm


©SAP AG 2009

The following procedure describes how to insert an agent in the boot sequence for a Linux SUSE 10 system:

Create a file diagnostics_agents in folder /etc/init.d Determine the init run level in file /etc/inittab Create a symbolic link named S80diagnostics_agents in the folder

/etc/init.d/rc<INIT_LEVEL>.d/ to the file diagnostics_agents (The agent has to be started afterthe initialization of the network services).

Create a symbolic link named S80diagnostics_agents in the folder /etc/init.d/rc6.d to the filediagnostics_agents (runlevel 6 corresponds to the System reboot).

When smdstart.sh and smdstop.sh are embedded in the system startup scripts, the exit code of the smdscripts cannot be retrieved. Further it is important to redirect the standard output and standar error with a pipelike 2>&1 to avoid that the smdstart.sh and smdstop.sh are blocking the parent script.

#!/bin/sh### BEGIN INIT INFO# Provides: diagnostics_agents# Required-Start: $network# Required-Stop: $network# Default-Start: 2 3 5# Description: start diagnostics agents### END INIT INFO

case "$1" in'start') su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1 -s /bin/sh - smdadm ;;'stop') su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null -s /bin/sh - smdadm ;;'status') su -c /usr/sap/SMD/J98/script/smdadmin.sh -s /bin/sh - smdadm

;;*) echo "Usage: $0 { start | stop | status }" ;;esacexit 0


©SAP AG 2009

Information about Diagnostics agent 7.11 or higher

With Diagnostics agent 711 the startup system is compliant to the SAP startup mechanism. It means that thecommands smdstart.sh and smdstop.sh have been removed.

Under Windows the SAP MMC allows to start or stop the Diagnostics agent. The NT service of Diagnosticsagent is configured to start automatically during the Windows startup.

The green status in SAP MMC only indicates that the agent is running. It doesn’t report if the agent isconnected to the Managing system. If the status is yellow after the agent start, it is probably due to the factthat the agent process was stopped abnormally (see 7.3 Agent crash investigation)

Under UNIX with an agent OS User, the commands startsap and stopsap allow to operate the Diagnosticsagent.

Command Actionstartsap SMDA<Instance_Number> [-vhost <virtual hostname>] Start the Diagnostics agent.stopsap SMDA<Instance_Number> [-vhost <virtual hostname>] Stop the Diagnostics agent.

The sapstartsrv process of Diagnostics agent is automatically started during the boot sequence of system if the sapinitprogram has been installed on system. In that case, you can start or stop the Diagnostics agent with a SAP MMC(remote).

About sapstartsrv process

SAP Note Description927637 - Web service authentication insapstartsrv as of Release 7.00

The Web service interface of sapstartsrv requires a user authenticationfrom the client (MMC, SAP MC, sapcontrol and so on) for protectedoperations such as starting and stopping the SAP instance.

Note 823941 - SAP Start Service on Unixplatforms

After an SAP upgrade, the SAP Start Service (sapstartsrv) on UNIXplatforms is not automatically started when the operating system isstarted.


©SAP AG 2009

5 Agent connectivity

5.1 Managing systemIn this section, you will find a description of the different connection modes used from the agent to the ManagingSystem. Two modes are available:

Connection through the SAPJ2EE message server (RECOMMENDED mode) Direct connection to SAPJ2EE p4 dispatcher.

Connection through the SAPJ2EE Message server

It is RECOMMENDED to connect the Diagnostics agent to Solution Manager with the SPA J2EE messageserver mode. To establish this connection, the information about the hostname of the J2EE Message server(SCS instance) and the HTTP port of message server must be collected from the SCS instance profile.

The agent connection mechanism can be described plainly:

A. During the installation of the Diagnostic agent, the SAPJ2EE Message server connection mode must beselected.

B. This information is stored in the agent configuration (runtime.properties) and the credentials are stored inthe agent secure store file (secstore.properties).

C. When the Diagnostics Agent starts, the P4 framework will retrieve all J2EE entries point available from theSCS message server (on HTTP port ex: 81xx).

D. Once the list of J2EE entries point has been collected, one P4 entry point is chosen from the list.E. The Diagnostics agent will then establish a P4 connection to the selected J2EE entry point with the host

name and the port mapped to the P4 entry point.


©SAP AG 2009

Troubleshooting Message server connectionBefore starting the agent installation or before executing the smdsetup script, you can check the content ofthe SCS message server with the following procedure:

1. Open the URL: http://msgserver:81xx/msgserver/text/logon (xx corresponding to instance numberof the SCS instance)

2. Check if a P4 entry point is available,3. On the managed system, open a shell with the <SID>adm user of the Managed system or the SMD

Agent.4. Execute the command “ping <host of P4 entry point>”: The host must be accessible,5. Execute the command “telnet <host of P4 entry point> <P4 port>”. If no error message

occurs (blank screen) the connection is OK.

Most common agent connection problems

All connection problems are logged in a file named SMDSystem.X.log (see chapter 6.2.5). Check this logto find the root exception of the issue: Once the exception is found, find the root cause of the exception bylocating the last “caused by” string.

No Entry point found

Exception: com.sap.engine.services.jndi.persistent.exceptions.NamingException:Exception while trying to get InitialContext. [Root exception iscom.sap.engine.interfaces.cross.DestinationException: there are no access points forservice: P4 registered on the message server]

No P4 entry point has been reported by the Message Server. This is probably due to the fact that it is notthe correct message server: it must be the Java and not the ABAP one. So check the content of theMessage server with the URL http://msgserver:81xx/msgserver/text/logon and verify if a P4 entry point isavailable (as shown below).

Unknown host

Exception: [sapms:// yupxxxxx.domain,com:8136] Checking server availability...Connecting to SMD server sapms://yupxxxxx.domain,com:8136 failed - error counter: 0- com.sap.engine.services.jndi.persistent.exceptions. NamingException:Exception while trying to get InitialContext. [Root exception iscom.sap.engine.interfaces.cross.DestinationException: cannot establish connection with any of the available instances #Nested exceptions are: #java.net.UnknownHostException: yupxxxxx: yupxxxxx ]

This is a network resolution problem due to the fact that the selected P4 hostname retrieved from the SCSMessage server list cannot be resolved from the Managed system host. Retrieve the list of the P4 entriespoint in the Message server as:

version 1.0J2EE933894500J2EE NCED6018XXXXA.dhcp.ncel.sap.corp 59300 LB=1J2EES NCED6018XXXXA.dhcp.ncel.sap.corp 59301 LB=1P4 NCED6018XXXXA.dhcp.ncel.sap.corp 59304 LB=1P4S NCED6018XXXXA.dhcp.ncel.sap.corp 59306 LB=1P4HTTP NCED6018XXXXA.dhcp.ncel.sap.corp 59305 LB=1


©SAP AG 2009

Execute the command “ping <hostname_of_P4_entry_point_5xx04>” in a Managed system shell.

If the host cannot be reached, then change the host definition in your DNS or in the /etc/hosts. You mayalternatively fix you engine installation to report the right host information in the message server (seechapter 5.3).

Direct Connection to P4 Dispatcher

Solution Manager in cluster environment

If the solution manager system is installed on a High Availability environment, this direct connectioncannot be used because the agent would always establish a connection to the same java instancecorresponding to the same p4 port.

To enable HA, you must select the SAP J2EE Message server connection mode. In that case, theconnection of the diagnostics agent will be load-balanced to the different available Java instances.


©SAP AG 2009

5.2 Managed systemAll connections to the Managed systems from the Diagnostics agent are executed locally on the same host. It is thereason why it is really imperative to install the Diagnostics agent on the same virtual host than the SAP Managedsystem.

The agent connects to the following different SAP System layers:

Java stack on port P4 (5xx04) Sapstartsrv http port of Managed System (5xx13) J2EE Message server (39xx and 81xx) RFC call to Managed Abap stack

Troubleshooting P4 connection problems with Managed system

Check on which host the P4 socket is boundo Run the OS command “netstat –an” to find the entry line containing the P4 port of the managed

system:

TCP 0.0.0.0:59304 0.0.0.0:0 LISTENING

Ortcp 0 0 *.50304 *.* LISTEN

In this example we see that the p4 port is bound for all hosts where a p4 request can be emitted (* or0.0.0.0), if the port is bound like “10.45.192.204:50304” only the p4 requests emitted from the host10.45.192.204 will be accepted.It explains why the Diagnostic agent must be installed on virtual host 10.45.192.204 with SAPInst. Thesituation can be fixed by changing the profile property SAPLOCALHOST (see 6.2.3 Change the virtualhost name).

o After verifying the port in the OS configuration, check in the J2ee dispatcher configuration that theproperty “bindHost” is set to 0.0.0.0. This should be visible with the J2EE configuration tool of themanaged system->Instance_xxx->dispatcher->services->p4. If the value is not configured to 0.0.0.0(which accepts all hosts), add the agent IP or install the agent on one of the listed IP.


©SAP AG 2009

Check if the agent can access the J2ee message server (SCS) of the Managed system with the host and thehttp port stored in the agent configuration (see the next section). For example :

o Open a shell with the agent OS user (smdadm) on the managed system.o Execute the command “msprot -l -mshost <host_j2ee_msg_srv> -msserv 39<SCS> or

36<SCS> -d J2EE –s”, the values of host and port must be retrieved from the Agent configuration(see the next section).

o The command must return the following content :

SAP Message Server Protocol Program, Version 1.3 (built: Jan 24 2008 01:27:19)

INFO CLUSTER-ID HOST GROUP-ID STATE J2EE-SERVICE-MASK

----------------------------------------------------------------------------------------------

---> LIST SERVICE:00000 NAME:ConfigurationManager

---> LIST SERVICE:00001 NAME:_CacheManager

---> LIST SERVICE:00003 NAME:{service_manager_deploy_distributor}

---> LIST SERVICE:00004 NAME:{service_manager_internal_connection}

---> LIST SERVICE:00005 NAME:shell

---> LIST SERVICE:00006 NAME:jmx_notification

---> LIST SERVICE:00007 NAME:p4

---> LIST SERVICE:00008 NAME:webservices

---> LIST SERVICE:00010 NAME:iiop

---> LIST SERVICE:00011 NAME:r3startup

---> LIST SERVICE:00012 NAME:telnet

---> LIST SERVICE:00013 NAME:log_configurator

---> LIST SERVICE:00014 NAME:jmx

---> LIST SERVICE:00016 NAME:http

---> LIST SERVICE:00022 NAME:com.sap.security.core.ume.service

---> LIST SERVICE:00023 NAME:security

---> LIST SERVICE:00024 NAME:tc~sec~certrevoc~service

---> LIST SERVICE:00025 NAME:servlet_jsp

---> LIST SERVICE:00026 NAME:webdynpro

LIST 0004105250 ncevhost.mydomain 0000041052 RUNNING XXXXXXXXOXXOXXXXXOXXXXXXXXXXXX

LIST 0004105200 ncevhost.mydomain 0000041052 RUNNING XXXXXXOXXXXXXXXXXOOOOOOOOOOOOO

In this content, the P4 service is listed as well as the j2ee nodes with hostnames (e.g.ncevhost.mydomain).Check if those hostnames can be resolved on your network by executing the command “pingncevhost.mydomain”

o Open the J2EE Visual admin of the Managed system and connect with the j2ee user used by the Diagnosticsagent (agent configuration see next section) to check if the user can be authenticated and if it has theAdministrator role.

o Check also the Managed system defaulttrace file to find some exception linked to P4 connection.


©SAP AG 2009

Connection configuration for managed systems

The following table lists all properties generated and updated by the Managed Setup Wizard allowing the Diagnosticsagent to connect to the Managed system in order to complete the E2E RCA operations.

Those properties are stored in the Diagnostics agent configuration under the scope named “server name”.o Managed systems connection information:

Connection Type toManaged system

Port Configuration stored in agelet Property

P4 port 5xx04 com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/P4/porthttp port 5xx00 com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/http/portJ2EE Message server InternPort

39xx com.sap.smd.agent.application.global.configuration [SID]/sapj2ee/msgserver/port

J2EE Message server Httpport

81xx com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/http/port

J2EE Message server host <Hostname> com.sap.smd.agent.application.global.configuration [SID]/sapj2ee/msgserver/host

o Credentials :

Credential Configuration stored in agelet Property user Property passwordJ2EE credentials com.sap.smd.agent.application.global.configuration [SID]/sapj2ee/admin/user [SID]/sapj2ee/admin/pwd

Credential Configuration stored in agelet Propertyclient

Property user Propertypassword

ABAP credentials com.sap.smd.agent.application.global.configuration [SID]/its/client [SID]/abap/com/user [SID]/its/client

Those values might be only changed in the agent Admin tool in case of debugging. The general recommendation is toupdate those values with the Managed system setup wizard.


©SAP AG 2009

5.3 Full Qualified Name resolution issueIn most of the cases, a connection issue from the Diagnostics Agent to the Managing system or the managed reveals anetwork configuration issue. You will find below a list of SAP notes which can help to solve some network resolutionproblems in different part of SAP products.

SAP Note ResumeNote 611361Hostnames of SAP servers

The use of hostnames that are too long or contain illegal characters mayresult in undetermined errors

Note 962955Use of virtual TCP/IP host names

You want to use one or more virtual host names for SAP software within anSAP server landscape to hide the physical network identities for the pieces ofsoftware from each other.

Note 773830FQHN determination in ICM

It is not clear which fully qualified host name (FQHN) the InternetCommunication Manager (ICM) is using.

Note 962955Use of virtual TCP/IP host names


Note 1282975Use of virtual TCP/IP host names inWindows


Note 129997Hostname and IP address lookup

Information about IP addresses lookup.

Note 1052122Host names in SLD

You wonder if a hostname that occurs in the System Landscape Directory(SLD) is correct. In the SLD, hostnames primarily occur in technical systemsand destinations

Note 979277Incorrect registration of computersystems in the SLD

In computer systems registered by the ABAP data supplier in the SLD,attributes may be missing in certain cases (for example, details of theoperating system).

5.4 SAP routerIn some network configuration, the Diagnostics agent has to connect to the Managing system though a SAP router.This feature will be fully supported in Solution Manager 7 Ehp2 (7.02).

In order to enable this feature in Ehp1, please contact the Diagnostics Agent support team to get information on how tosetup the agent and on which patch level you can perform the installation.


©SAP AG 2009

5.5 Self-MonitoringDuring a self-monitoring setup of Solution Manager, some special configurations in the different setup wizard areneeded due to some restrictions about the local agent connectivity. All information about the setup configuration of the“self-monitoring” is available in the following SAP Notes:

SAP Note Resume1273028 - Introscope 8 ReleaseNotes

BCI Agent 8.0 cannot be used for SMD self-monitoring. Please use Javaagent version 7.2 for self monitoring of the Solution Manager Java stack.

917080 - SMD Auto-Monitoring :single user for SMD Agent setup

DEPRECATED, DO NOT USE ANYMORE

1292811 - Wily setup wizard is failingin auto-monitoring system

An error occurs when performing the setup of a wily introscope agent onsolution manager itself (self-monitoring) although the user and password arecorrect.

Error in Managed system setup wizard in Self-monitoring

End of stream is reached unexpectedly during input from Socket

Exception:com.sap.smdagent.plugins.connectors.p4.exceptions.P4AuthorizationException: Access isdenied: Please make sure the <user> and <password> parameters are correct.; nestedexception is:com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception duringgetInitialContext operation.No server is running. [Root exception iscom.sap.engine.services.security.exceptions.BaseLoginException: Exception in creating newRemoteLoginContext instance.]atcom.sap.smd.agent.plugin.connectors.p4.P4ConnectionService.getP4Connection(P4ConnectionService.java:69)...at java.lang.Thread.run(Thread.java:838)Caused by: com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exceptionduring getInitialContext operation.No server is running. Root exception iscom.sap.engine.services.security.exceptions.BaseLoginException: Exception in creating newRemoteLoginContext instance.atcom.sap.engine.services.security.remote.login.RemoteLoginContextExt.<init>(RemoteLoginContextExt.java:34)...at java.lang.Thread.run(Thread.java:838)Caused by: java.io.EOFException: End of stream is reached unexpectedly during input fromSocket [addr=/56.zzz.xx.xx,port=50204,localport=22326]at com.sap.engine.services.rmi_p4.Connection.run(Connection.java:423)

The problem is linked to the value if the host name registered in the Solution Manager SCS message server.Check the hostname stored in SCS message server with the following urlhttp://scs_host:81<SCS_instance_number>/msgserver/text/logon.

The agent hostname reported in agent admin and SCS hostname of entry points must be equal.

Since 7.01 Ehp1 SPS19, add the property <SID>/selfmonitoring/enabled=true in the agelet global configurationof the Diagnostics agent XD XD which do the self monitoring to avoid some auto-monitoring connection issue.


©SAP AG 2009

6 Agent Maintenance

6.1 Upgrade SAP Kernel (saposcol update)

For specific agent issues, you might be asked to upgrade the SAP kernel binaries of the Diagnostics agent (Ex: This isrequested when upgrading saposcol). In that case, follow the procedure:

For 7.00 and 7.01

Determine the SID and instance number of the Diagnostics agent to upgrade. Determine the SAP Kernel version by opening a shell with <sid>adm for Unix or command prompt Windows in

folder /usr/sap/<SID_AGT>/exe Execute the command “sapcontrol -prot NI_HTTP -nr <Instance_Number_AGT> -function

GetVersionInfo”

GetVersionInfo

OK

Filename, VersionInfo, Time

C:\usr\sap\SMD\J98\..\exe\sapstartsrv.exe, 700, patch 144, changelist 957098, optU, NTintel,C:\usr\sap\SMD\J98\..\exe\msg_server.exe, 700, patch 144, changelist 957098, optU, NTintel,C:\usr\sap\SMD\J98\..\exe\enserver.exe, 700, patch 144, changelist 957098, optU, NTintel,C:\usr\sap\SMD\J98\..\exe\jcontrol.exe, 700, patch 144, changelist 957098, optU, NTintel,C:\usr\sap\SMD\J98\..\exe\jlaunch.exe, 700, patch 144, changelist 957098, optU, NTintel,

Stop all Diagnostics agents with same SID (because they share the binaries) Download the latest version of the SAP Kernel on service market place. Be careful if the agent kernel version is

700 you must upgrade to the latest available version of the 700 kernel. You shouldn’t upgrade a 700 kernel tothe latest version of the 701 kernel.

Extract the SAR File (ex: SAPEXE_179-20000978.SAR) in folder /usr/sap/<SID_AGT>/exe

For 711

Determine the SID and instance number of the Diagnostics agent to upgrade. Determine the SAP Kernel version, by opening a shell with <sid>adm for Unix or command prompt Windows in

folder /usr/sap/<SID_AGT>/SYS/exe/uc/<OS>. Execute the command “sapcontrol -prot NI_HTTP -nr <Instance_Number_AGT> -function

GetVersionInfo”

/usr/sap/DAA/SMDA96/exe/sapstartsrv, 711, patch 9, changelist 1010123, optu; sun_64,/usr/sap/DAA/SMDA96/exe/gwrd, 711, patch 9, changelist 1010123, optu , sun_64,/usr/sap/DAA/SMDA96/exe/icman, 711, patch 9, changelist 1010123, optu , sun_64,/usr/sap/DAA/SMDA96/exe/jcontrol, 711, patch 9, changelist 1010123, optu , sun_64,/usr/sap/DAA/SMDA96/exe/jlaunch, 711, patch 9, changelist 1010123, optu , sun_64,/usr/sap/DAA/SMDA96/exe/jstart, 711, patch 9, changelist 1010123, optu , sun_64,

Stop all Diagnostics agents with same SID. Download the latest version of the SAP Kernel 711 on service market place. (do not use SAP Kernel 710) Extract the SAR File (ex: SAPEXE_20-20003746.SAR) in the folder

/usr/sap/<SID_AGT>/SYS/exe/uc/<OS>

6.2 ConfigurationThis section describes all the procedures to update the Diagnostics agent configuration elements which cannot bechanged centrally from Solution Manager.


©SAP AG 2009

6.2.1 Change Path of the JDKIf the JDK path changes after the Diagnostics agent installation, you need to update the path of JDK in the agent profilefile.

For 7.00 and 7.01

Open the profile <SID_AGT>_J<INSTANCE_NUMBER>_<hostname> in folder/usr/sap/<SID_AGT>/sys/profile.

Update the parameter jstartup/vm/home by putting the absolute path to the JDK home. Remove the file /usr/sap/<SID_AGT>/JXX/profile/smd.properties.vmprop. Restart the Diagnostics agent.

For 7.11

As the Diagnostics agent 711 runs only on SAPJVM5, it is not recommended to change the path of the JDK.

6.2.2 Change JVM ParametersIf the JVM parameters need to be changed, you have to update the agent instance properties file.

For 7.00 and 7.01

Open the file smd.properties in folder /usr/sap/<SID_AGT>/J<Instance_Number>/profile. Update the parameter “smdagent.javaParameters” (see 4.4 JVM parameters). Remove the file /usr/sap/<SID_AGT>/JXX/profile/smd.properties.vmprop. Restart the Diagnostics agent.

For 7.11

Open the file smdagent.properties in the folder/usr/sap/<SID_AGT>/SMDA<Instance_Number>/SMDAgent.

Update the parameter “smdagent.javaParameters” (see 4.4 JVM parameters). Remove the file /usr/sap/<SID_AGT>/JXX/profile/smd.properties.vmprop. Restart the Diagnostics agent.

6.2.3 Change the virtual host nameIf the Diagnostics agent host name reported in the Agent Admin tool is not the expected name, you can change it in theagent instance profile.

For 7.00, 7.01 and 711

Open the profile <SID_AGT>_J<INSTANCE_NUMBER>_<hostname> in the folder/usr/sap/<SID_AGT>/sys/profile.

Update or add the parameter SAPLOCALHOST by setting the short hostname desired (not the FQN).


©SAP AG 2009

6.2.4 Usage of smdsetup scriptThe script ’smdsetup’ allows to update some configuration parameters for Diagnostics agent. This script is delivered inthe Diagnostics Agent since Solution Manager SPS16. However, this script is always automatically patched to thelatest version when the Diagnostics agent connects to the SMD server.So if the LMSERVICE has a version greater than SPS15, the smdsetup script of the Diagnostics agents will be updatedto the last available version.

To find out what is the version of the smdsetup script, you have to execute the command “smdsetup.sh” or“smdsetup.bat” without any parameters

o If the command output asks 5 parameters like “smdsetup.sh $JAVA_HOME smdserverhost 53004j2eadmin abc123 myhost”, it is a smdsetup with a version less than SPS13.

Example of smdsetup SP13 output:

SMD Agent Setup expects the following parameters: 1) Path to the Java Virtual Machine - JDK 1.4 is required 2) Host Name (or IP Address) of the SMD Server (managing SYSTEM) Note: Make sure to prefix the host name by sapms:// when using the message server port. 3) Message Server HTTP Port (ex: 8131) or P4 Port (ex: 50004) of the SMD Server 4) User (J2EE User member of 'Administrator' Role) 5) Password of the J2ee User 6) The so called 'SMSY Server Name' which is the unique key used

to declared the host in Solution Manager.

o If the output proposes some action as “managingconf”, the version of smdsetup version is greater thanSPS14.

Example of smdsetup SP14+ output:

>> Help Information----------------------

Script which allows to update some diagnostics agent configuration.smdsetup <action> <parameters>List of actions: * smdsetup secstorej2ee user:"<value>" pwd:"<value>" * smdsetup secstoresld user:"<value>" pwd:"<value>" * smdsetup sldconf hostname:"<fqn>" port:"<http port>" [optional user:"<value>" pwd:"<value>"] * smdsetup managingconf hostname:"sapms://<fqn>" port:"<MS httpPort>" [optional user:"<...>" pwd:"<...>"servername:"<...>"] * smdsetup managingconf hostname:"<fqn>" port:"<p4 port>" [optional user:"<value>" pwd:"<value>"servername:"<value>"]….

SMDSetup version <= SPS13

The smdsetup script in this version is located in /usr/sap/<SID_AGT>/J<Instance_Number>/SMDAgent.


©SAP AG 2009

In this version the script permits only to change the connection information and the credentials at the same time. Itmeans that all the input parameters are mandatory.

Classpath error

In case of a classpath error, you need to adapt the classpath of the smdsetup script SP12 or lower, becausein SPS13 the logging.jar has moved from the SMDAgent/lib/launcher directory to the SMDAgent/lib directory.

SMDSetup version >= SPS14

In 7.0 SR3 and 7.0 the smdsetup script in this version is located in the following folder:o /usr/sap/<SID_AGT>/J<Instance_Number>/script

In 7.11 the smdsetup script is located in the following folder:o /usr/sap/<SID_AGT>/SMDA<Instance_Number>/script

Action Description Availability

managingconf Change the managing system of the agent (host and port), restartagent after the changes. SPS14

sldconf Change the SLD server where the agent is registered (host andport), restart agent after the changes. SPS14

secstorej2ee Change only the J2EE user used to connect to the Managingsystem. (restart after the changes) SPS14

secstoresld Change only the J2EE user used to register the agent to the SLDsystem. (restart after the changes) SPS14

changeservername Change the server name of agent. (restart after the changes) SPS14

addsaprouter Full Implementation in Ehp2, so don’t use this command SPS18

saprouterpass Full Implementation in Ehp2, so don’t use this command SPS18

removesaprouter Full Implementation in Ehp2, so don’t use this command SPS18

supportlogs Execute some agent health-checks and generate a zip containingall agents logs (including the agent check report) SPS17 patch 3

MANAGINGCONF SYNTAX

To execute this command you need to stop the agent. The command updates the connection settings to the Managingsystem:

Connect the agent by P4 direct connection:smdsetup managingconf hostname:”myhost” port:”53004”

Connect the agent by J2EE Message server http port (not internal message server port 36xx):smdsetup managingconf hostname:”sapms://myhost.domain.corp” port:”8101”

Optionally, you can add the j2ee user information in the command line to update also the agent secure store:smdsetup managingconf hostname:”myhost.domain,corp” port:”53004” user:”SMD_ADMIN”pwd:”init1234”


©SAP AG 2009

SLDCONF SYNTAX

To execute this command you need to stop the agent. The command updates the connection settings to the SLDsystem of the agent registration:

Connect the agent to the SLD system by the http port of java stack.smdsetup sldconf hostname:”mySLDhost.domain.corp” port:”50000”

Optionally, you can add the sld user information in the command line to update the agent secure store:smdsetup sldconf hostname:”mySLDhost.domain.corp” port:”50000” user:”slddsuser” pwd:”xxxxx”

SECSTOREJEE SYNTAX

To execute this command you need to stop the agent. The command updates the credentials of the connection toManaging system:

Update the agent secstore file:smdsetup secstorej2ee user:”SMD_ADMIN” pwd:”init1234”

SECSTORESLD SYNTAX

To execute this command you need to stop the agent. The command updates the credentials of the connection to theSLD system:

Update the agent secstore filesmdsetup secstoresld user:”slddsuser” pwd:”xxxxxx”

CHANGESERVERNAME SYNTAX

To execute this command you need to stop the agent. The command updates the “servername” attribute of the agent:

Update the agent configuration filesmdsetup changeservername servername:"myservername"

SUPPORTLOGS SYNTAX

The Diagnostics agent must be started before executing this action. This command retrieves all agent log files andexecutes some basics tests to check the agent pre-requisites as:

jdk version check jvm parameters check hostname resolution check tcp port check p4 protocol check message server connectivity check

To execute the commandsmdsetup supportlogs

At the end of the execution of the command, a Zip file is generated in/usr/sap/<AGENT_SID>/<AGENT_INSTANCE_NUMBER>/SMDAgent/supportLogs_<TIMESTAMP>.zip.

The zip content is the following: SMD Agent log files SMD Agent profile SAPInst logs Work directory All properties files contained in the SMD agent folder.


©SAP AG 2009

6.2.5 Change Agent log levelThe Diagnostics agent logs are stored in /usr/sap/<SID_AGT>/<InstNbr>/SMDAgent/log folder and organized asfollow:

Filename Default settings Defaultlevel

Description

SMDSystem.log 10 files of 1MBytes INFO This file reports all information about agentframework like connection to managing system,SLD report, agelets loading and patch sequence.

SMDAgentApplication.log 10 files of 1Mbytes WARNING This file reports all information about the ageletsused by RCA applications, Setup Wizards andWily IS setup.

P4.log 10 files of 1MBytes NONE This file reports all information about agentconnection layer with managing system.

When changing the log level of the agent logging, you need to consider two cases:o The Diagnostic agent is connected to the Solution Manager System:

Go in Agent Administration tool, select the tab “Agent Log viewer” Select the agent click on display button Select the location In section “Log Level Control”, select the level and click on the “apply” button.

o The Diagnostics agent is not connected to the Solution Manager System:

Stop the agent Go in folder /usr/sap/<SID_AGT>/<InstNbr>/SMDAgent/configuration Open the file log-configuration.xml Adapt the level (attribute effective-severity) in following line (level values:

ALL,DEBUG,INFO,WARNING,ERROR,NONE):

For FilenameSMDSystem.log <log-controller effective-severity="INFO" maximum-severity="ALL" minimum-

severity="DEBUG" name="/SMDlogger/System">SMDAgentApplication.log <log-controller effective-severity="WARNING" maximum-severity="ALL" minimum-

severity="DEBUG" name="/SMDlogger/AgentApp">P4.log <log-controller effective-severity="NONE" maximum-severity="ALL" minimum-

severity="ALL" name="com.sap.engine.services.rmi_p4">

Save the file and restart the agent.


©SAP AG 2009

6.3 Managing J2EE userInformation about the agent j2ee user who connects to the Solution Manager System, is described in the “UserAdministration guide”, section 2.3.3 [SOLMAN.AGT.COM]: Diagnostics agent Communication User.

If the J2EE user used by the Diagnostic agent cannot be authenticated (wrong password, missing role assignment,user locked), the agent will shutdown automatically to avoid locking the corresponding account.

Most common agent connection problems

Authentication failed

all authentication problems are logged in file SMDSystem.log.

Exception: at com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:170)Caused by:com.sap.engine.services.security.exceptions.BaseLoginException:Authentication did not succeed. atcom.sap.engine.services.security.login.ModulesProcessAction.run (ModulesProcessAction.java:175) ... 13 more

Before restarting the agent It is important to execute some checks related to the J2EE user as follow:

1. Start the ABAP transaction SU01 on Solution Manager. Check if the J2EE user (SMD_ADMIN) is not locked Check that the Role assigned is compliant to SAP recommendations done in the “User Administration guide”.2. Logon to the Agent administration tool with the J2EE user (SMD_ADMIN) to check that the logon module ofthe Java stack works correctly.3. Call the smdsetup to update only the credentials to be sure the password is up-to-date.4. Restart the Diagnostic agent.

In dev_smdagent.log, if you find the exit code -11112, it is an authentication problem as described above.

[Thr 880] Wed Feb 04 09:28:47 2009[Thr 880] JLaunchIStartFunc: end of main method (rc = 0)[Thr 6000] JLaunchIExitJava: exit hook is called (rc = -11112)[Thr 6000] **************************************************************** ERROR => The Java VM terminated with a non-zero exit code.*** Please see SAP Note 943602 , section 'J2EE Engine exit codes'*** for additional information and trouble shooting.**********************************************************************

[Thr 6000] JLaunchCloseProgram: good bye (exitcode = -11112)


©SAP AG 2009

7 Tracking agent issues

7.1 Deprecated SAP notes

The following table contains a list of SAP notes which are now deprecated:

SAP Note Short description State

985140 SMD Agent/SAPinst: Virtual Host is not taken into account Deprecated906248 Problem with missign CA directory with NW04s SP5 SMD install Deprecated987092 Install an SMD Agent <= SP11 on IBM System i (i5/OS, OS400) Deprecated954861 SMD agent installation with SAP Inst failed (SSL problem) Deprecated985138 SMD Agent/SAPinst/IBM JDK: Java version wrongly detected Deprecated1037767 SMD Agent/SAPinst: DNS Domain name error on AIX Deprecated1055798 smdagent.sh start: java.lang.NoClassDefFoundError Deprecated1069588 SMD Agent/SAPinst: How to use an alternate System Name Deprecated

7.2 OSS message pre-requisitesBefore forwarding any OSS messages to the SAP support organization, it is important to check the following points:

The Diagnostics agent has been installed with SAPInst SR3, Ehp1 or 7.11. The OS limit for Unix platform are compliant with the agent OS user (<AgentSID>adm), JDK vendor and version is supported by Diagnostics agent, JVM parameters of Diagnostics agent are compliant, The Diagnostics agent connection to the Managing system is of type J2EE Message Server connection, Recommendations for the auto-monitoring scenario Managed setup wizard execution have been applied. The Java dispatcher of Solution Manager is correctly sized to avoid some agent disconnection (see part 8.1

Managing java stack settings.)

If those check points cannot solve the problem, follow the SAP Note 1076573 to create the OSS message. If theLMService version is greater or equal to NW04S SPS17 Patch 3, attach the support logs Zip file generated bythe smdsetup script (see part 6.2.4).


©SAP AG 2009

7.3 Agent crash investigationWhen the system process of the Diagnostics agent shutdowns abnormally, the procedure to determine the cause ofthis crash is the following:

Go in folder /usr/sap/<SID_AGT>/J<INSTANCE_NUMBER>/work Open the log file dev_jcontrol (This is the log file of the jcontrol process which is the controller of the

java agent process) to check if the jcontrol has started the jlaunch process as described below:

[Thr 3288] JControlICheckProcessList: process SMDAgent started (PID:3616) If the process jlaunch cannot be start, the error is probably located in SAP profile parameters

The message JStartupICheckFrameworkPackage: can't find framework packageC:\usr\sap\SMD\exe\jvmx.jar is NOT a critical error. The Diagnostics agent does not use specific SAPkernel functions and this message can be ignored.

If the jlaunch process has been started, open the file dev_SMDAgent which corresponds to the jlaunchprocess output :

Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Initial (0)][Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Waiting for start (1)]

[Thr 3860] Thu Dec 25 12:11:35 2008[Thr 3860] JLaunchISetState: change state from [Waiting for start (1)] to [Starting (2)][Thr 3860] JLaunchISetState: change state from [Starting (2)] to [Running (3)]

In this file, all information about the java of the Diagnostics agent is stored. You will information related toJVM loading, process state, crash of VM or bad JVM parameters. In case of process crash, you will find inthis file, the corresponding Exit Code or Error message.

JVM Loading problem

all JVM loading problems are logged in file dev_SMDAgent.

Error: [Thr 1084238144] *** ERROR => DlLoadLib: dlopen()= libjvm.so: cannot open sharedobject file: No such file or directory -> DLENOACCESS (0, Success) [dlux_mt.c 314][Thr 1084238144] *** ERROR => Can't load VM shared library (libjvm.so) (rc=-2)[jhvmxx_mt.c 1576][Thr 1084238144] *** ERROR => Cannot load DLL for Java VM [jlnchxxi_mt.717]

You will find potential JDK installation issues like : Wrong JDK version for Agent JDK is not compatible with SAP Kernel (32bits/64bits) Agent OS user have not rights to access to JDK Libraries

The SAP Note 940893 - Analysis of errors within the startup of J2EE Engine 6.40 can help to troubleshootJVM loading problems.

If the JVM is loaded correctly, now check the file std_SMDAgent.out and jvm_SMDAgent.out to find someerrors like OutOfMemory, StackOverflow or Nullpointer which could have stopped the agent process.


©SAP AG 2009

7.4 Thread dumpFacing a deadlock situation or a Diagnostic agent freeze, the only correct way to understand the cause of the problemis to perform a thread dump on the agent process.

Make a Thread dump under Windows

The Diagnostics agent is like any other SAP system, available in the SAP MMC.

In SAP MMC, open the node “AS Java Process Table” of the Diagnostics agent identified by the SID and the instancenumber. Only one process is displayed in the right part: It is the agent process.Select this process and right click on it to display the context menu, press the item “Dump stack Trace”.

Make a Thread dump under Unix

Under UNIX platform, you need to use the command smdadmin.sh.


©SAP AG 2009

Start in an UNIX shell with the OS user of the Diagnostics agent. Execute the command “smdadmin.sh XX” (XX is theinstance number of Diagnostics agent).For old agent installation, it is necessary to execute the command “../smdadmin.sh” from the folder/usr/sap/<SID_AGT>/J<InstanceNumber>/script.

Once the command is executed, the JCMON menu is displaying, select the entry “20: Local administration menu” thenexecute the action “10: Dump stacktrace” on process 0 (This is the process of the SMDAgent).

Where is the thread dump stored?

Once the thread dump is done, the output is written in file /usr/sap/<SID_AGT>/J<InstanceNumber>/work/std_SMDAgent.out for Sun and HP-UNIX JDK.

For IBM JDK, the thread dump output is written in java.core.<timestamp>.txt in folderusr/sap/<SID_AGT>/J<InstanceNumber>/SMDAgent.To find the absolute path of an IBM thread dump, open the filestd_SMDAgent.out and find the line beginning with “Java Dump written to”.

7.5 Activate P4 logsIn case of a persistent p4 connection problem with the managing system or the managed system (like timeout,connection reset, ..) , you should enable the P4 logs before sending the OSS message to the Development Supportorganization.

Connection problem to Managing System or Managed system

The procedure to enable the P4 logs on the Diagnostics agent is the following:

Stop the agent Clean the log folder under SMDAgent directory Set the log level (effective-severity) of SMDSystem location to ALL (see 6.2.5 Change Agent log level) Set the log level (effective-severity) of P4 location to ALL (see 6.2.5 Change Agent log level) Add the Java parameter –Ddebug=true in agent configuration (6.2.2 Change JVM Parameters)

Once the p4 log enabled on agent, go to Managing system or Managed system to execute the following instructions:

Open the Visual Administrator of Java stack Select the server node Under SID node Select the service “Log Configurator” Go in tab Locations Select the location com.sap.engine.services.rmi_p4 Increase the log level to DEBUG and save the modification.

Once all P4 locations are configured, restart the agent to reproduce the problem and attach the agent logs (use thecommand ‘smdsetup supportlogs’) and the current defaulttrace.log of the Managing or the Managed system to the OSSmessage.

Performance Impact

It is important to restore All log locations changed to their default level because the procedure above has asevere impact on the performance of the Managed system or the Managing system.


©SAP AG 2009

7.6 Telnet SMD server commandsSome telnet commands are available in the Solution Manager Java stack to check the installation and configuration ofthe SMD server.

To use those commands, start a telnet client to the SAP J2EE server of Solution Manager (default port 5xx18) and goon server node with the command ‘jump <nodeid>’. Add the group smdserver to enable the SMD commands by typinguse the command ‘add smdserver’.

Once the group smdserver is added, execute the command ’man –g smdserver’ to see all available SMD servercommands.

SMD server telnet command DescriptionSmdstatus Display the version and running mode of SMD server.Smdstatus –a Check connectivity of all agentslist_agents Display all diagnostics agents connected to SMD server with their status.find_agent <servername> or <fqn> Retrieve an agent from name, fully qualified domain name or IP.mm ON Enable the maintenance mode of SMD servermm OFF Disable the maintenance mode of SMD serverkill_agent <servername> or <fqn> Reboot the Diagnostics agent processstart_agent <servername> or <fqn> Start the agent applications of selected Diagnostics agentstop_agent <servername> or <fqn> Stop the agent applications of selected Diagnostics agenthost_mapping Display the mapping between server name and network names for all

agents.known_agents -c Displays the list of known agents. (Agents that have been registered at least

once to the SMD Server)known_agents -n Displays the list of known agents by FQN.as Provides ping statistics for all agents.es Provide statistics usage for the SMD server.

8 Scalability

8.1 Managing java stack settingsWe recommend adapting the J2EE configuration of Solution Manager Java stack if an important number of Diagnosticsagents will connect to Solution Manager.

The Java dispatcher has an important role in the agent connection’s stability. Therefore, you need to adapt themaximum number of threads in the pool and in the heap size configuration of the dispatcher to avoid some timeoutconnection issues.

Dispatcher Parameter Recommended ValueJava Max Heap Size (-Xmx) Number of agents installed * 1.28 (minimum 128MB)ThreadManager->MaxThreadCount ~ Number of agents installed * 1.75

Default value of MaxThreadCount

After an installation of Solution Manager, the default value of parameter MaxThreadCount in Java Dispatcherconfiguration is set to 70. These settings allow by default to connect a maximum of 40 agents.

diagagent troubleshooting

Documents