HOW-TO | CIF CONSULT | Redouane BELBAHRI

Informatica Informatica Create a crash dump usingCreate a crash dump using “Debug Diag” “Debug Diag”

Agenda

1 Terminology

Terminology

ISP – Informatica Services Platform– 9.x - The ISP is the administrative framework for Informatica services. It allows monitoring, start/stop,

and failover of services

Model Repository Service (MRS)– Model Reposiory Service (MRS) manages the model Repository which contains the metadata for Data

Services.

Data Integration Service (DIS)– The Data Integration Service (DIS) is a container for all data integration functionalities. Plug-ins for the

DIS provide different data integration functionalities.

Terminology

Data Transformation Manager (DTM)– DTM Interprets and executes mappings.

– It is re-engineered to be lightweight and embeddable. It allows repository-less execution: In previous versions of PowerCenter, DTM connects to the repository to fetch the plan. However, now the mapping can be run without reading from the repository, like mapping stored to disk.

– DTM enables concurrent execution semantics: multiple DTM instances can coexist in the same process.

– It allows different execution modes: single threaded and pipelined execution modes.

Profiling Service– The Profiling Service is a component integrated into Informatica Data Services. It includes the Profiling

Plug-in and Profiling Warehouse.

Terminology

Mapping Service Module– The Mapping Service Module is the component service in the Data Integration Service that manages

requests to preview target data and run mappings.

SQL Service Module– The SQL Service Module is the component service in the Data Integration Service that manages SQL

queries sent to an SQL data service from a third party client tool.

– When the Data Integration Service receives an SQL request from a third party client tool, the SQL Service Module starts a DTM instance to run the SQL query against the virtual tables in the SQL data service.

Terminology – Nodes and Domains

When you install and run the Informatica services, the installation is known as a node. The node becomes part of an Informatica domain. A domain is a grouping of one or more nodes.

Domain Basics– The most basic description of an Informatica domain is this: There is a node, and there is a domain

database. The node is where stuff runs. The database is where the node keeps stuff.The Node

– The node is a logical representation of the Informatica installation on a server. It is run by Apache Tomcat.

– Two Kinds of Nodes: Worker Nodes: A worker node runs a Service Manager process, and it can run application services. Gateway Nodes: A gateway node can run application services, and it runs its own Service Manager

Process. When you start the master gateway node, the management services log their startup progress to node.log and catalina.out. You can find both of these log files in INFA_HOME/tomcat/logs. Master Gateway Node: Any gateway node can also be a master gateway node.

Terminology – Nodes and Domains

Node Communication– The master gateway node is the one node in the domain that communicates with the domain

database, and it does so through JDBC. It periodically performs a heartbeat check against the domain database.

Following is the architectural diagram of Informatica 9.1

Blue components are from existing PowerCenter.

Green components are added in Informatica 9.x

Informatica 9.X Service-Oriented Architecture

Agenda

2 Log management

Catalina.out & ispLogs.log

Catalina.out contains– Core services initial startup events

– Tomcat’s ERROR + event until 9.1HF3

– ISP process’s stdout & stderr

ispLogs.log (introduced in 9.1HF3)– Tomcat’s ERROR + events in 9.1HF3

– Tomcat’s INFO + event until 9.1HF4+

– Not generated on windows

Logging level modified through isplogging.properties in ./tomcat/bin location

Node.log

Contains– Master Election & Re-election

– Some errors during inter-node communication

Error severity Level controlled in AdminConsole from Node’s properties– WARNING by default. Change it to INFO for information of master re-election

Exceptions.log

Logs exceptions faced in the ISP process– Any exception message seen in Node.log is typically logged with stack trace to exceptions.log

ServiceFramework.log (until 9.1.x)

Tomcat/bin– JSF logs to this file.

– All Mercury service processes, ISP process, Admin Console process write to the same

– Critical for domain/node issues as it contains most critical errors

Isp/bin– Infacmd commands write to this file (infacmd generate this file in whichever directory you execute

the command)

– Contains errors related to infacmd to node/service communication.

Domain Logs (exported from Adminconsole)

Contains some more log files by each node and domain– On missing heartbeat from node

– When a service process terminated unexpectedly

– When a service process wrote to stdout/stderr before crashing (stack trace)

AdminConsole’s log files

Tomcat/temp/_AdminConsole/logs/* Services/AdministratorConsole/Administrator.log

Application Service Logs

./tomcat/logs/MRS/<MRS_NAME>.log ./tomcat/logs/as/analyst.log ./tomcat/bin/disLogs/<DIS_NAME>.log

Agenda

3 Windows diagnostic tools

Debug Diagnostic Tool

The Debug Diagnostic Tool (DebugDiag) is designed to assist in troubleshooting issues such as hangs, slow performance, memory leaks or fragmentation, and crashes in any user-mode process. The tool includes additional debugging scripts focused on Internet Information Services (IIS) applications, web data access components, COM+ and related Microsoft technologies.

Supported Operating System– Windows 7

– Windows 2003-Windows XP • Windows 2008-Windows Vista • Windows 2008 R2- Windows7

Installation steps

The Installation is strait forwards.

Create a stack trace from a hanging session or server process

When session or a server process hangs on Windows, use one of the following methods to obtain a stack trace and send it to Informatica technical support:

1. Determine the process ID (PID) of the hanging process 2. Dump the process stack

Agenda

4 Determine the process ID (PID) of the hanging process

Introduction to Process Explorer

Process Explorer is a lot like Task Manager, which is the program you access via the following keys on your keyboard: "CTRL+ALT+DELETE". Task Manager is used to display process information such as the process name, memory usage and other application information. However, the information that Task Manager displays is rather limited compared to what Process Explorer can show you. With Process Explorer one can see the entire process tree for a particular applications, which consists of all other processes that were started by the original process, or parent, in the tree.

Process Explorer has many different uses such as examining what process are active and what processes are making connections to outside computers. In this guide I will demonstrate the kinds of information one can gleam from using Process Explorer as opposed to using task manager.

The tool can be downloaded for free from: http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

Determine the process ID (PID)

Open Process Explorer– The top window always shows a list of the currently active processes

– Highlight the process. The Services tab of the process properties dialog shows the list of services running within a process.

Names of the PowerCenter, Data Quality and Data Explorer processes on Windows When PowerCenter node is started up, you should see the below Tomcat Processes:

– infasvcs.exe

– Java.exe

– If you start Informatica service from command line, then you may not see infasvcs.exe in the list.

pmserver.exe– This is the process for Integration Service

pmrepagent.exe – This is the process for Repository Agent, will show when Repository Service has started up.

pmdtm.exe – This is the session process started by the Integration Service.

Names of the PowerCenter, Data Quality and Data Explorer processes on Windows Data Explorer

If you are using Data Explorer you will see the following processes:

–Idelistener.exe

–Ide_server.exe

Data QualityWhen Data Quality Server is running, you should see the following process from Task Manager:

–ImplRepo_Service

–Naming_Service

–ImR_Activator

Agenda

5 Dump the process stack

Using Debug Diag

Open Debug Diag and add a crash rule. Select Crash

Select the process ID

Select Target Type as A specific process Select the process ID (Where PID is the process id obtained in the first step) and

check This process instance only

Configure the following next parameters:

Create a Rule

Add an event for Child Process Exit and set the Action Type to Full Userdump 9.Save and close. Click Next > Finish

Achievements

Following is achieved after performing the steps above:– The Child process in this context is the process being debugged (pmserver), not its children.

– When the debugged process exits, debug diag collects the full user dump.

– Run crash analysis on the dump and collect the stack of the process during the time of its exit.

– Core dump analysis can also be performed on the full user dump if required (space consuming)

Crash/Hang Analyzers

If a crash occurs, open the Debug Diag and do as follows: – Click on Advanced Analysis

– Click on Add Data Files

– Browse to the location for the datafiles, and click Open Once you have selected the datafile for the dump, click Start Analysis and send the

report to Global Customer Support (GCS).

Crash/Hang Analyzers

You can also right click the dump created dump file Start Analysis and send the report to Global Customer Support (GCS).

Using dump file generated by Dr Watson

Dr. Watson ( drwtsn32.exe ) is a default component of the Windows operating system.

Crash/Hang Analyzers (from Dr Watson)

Once you have selected the datafile for th dump, click Start Analysis and send the report to Global Customer Support (GCS).

java.lang.OutOfMemoryError: Java heap space

When a Java program starts, the Java Virtual Machine (JVM) gets some memory from the operating system. The JVM uses this memory for all its needs and part of this memory is called Java heap memory. When a Java application needs memory, the Java application gets it from Java Heap Space and when the object ceases to exist (garbage collected), the memory returns to Java Heap Space.

When JVM starts, JVM heap space is the initial size of Heap specified by the -Xms parameter. As the application progresses, objects are created and JVM expands Heap space in Java to accommodate new objects. JVM also runs garbage collector to reclaim memory back from objects that cease to exist. JVM expands Heap in Java close to the Maximum Heap Size specified by -Xmx and if there is no memory left for creating new objects in Java heap, JVM results in "java.lang.outofmemoryerror" and the application ends.

The default Heap size in Java is 128 MB. You can increase the size of Java Heap space based on what the Java application needs. To increase the Java heap space, use the parameters Xms (denotes starting size of Heap) and Xmx (denotes maximum size of Heap in Java).

ANNEX

Many Thanks for you time today!