common cm issue

Oracle Corporate Support

Problem Repository

1. Prob# 1011248.6 WHAT ARE THE COMMON CONCURRENT MANAGER ISSUES?

2. Soln# 2059281.6 MOST COMMON CONCURRENT MANAGER ISSUES BULLETIN

1. Prob# 1011248.6 WHAT ARE THE COMMON CONCURRENT MANAGER ISSUES?

Problem ID : 1011248.6

Affected Platforms : Generic: not platform specific

Affected Products : Oracle Application Object Library

Affected Components : FOUNDATION V10.XX

Affected Oracle Vsn : Generic

Summary:

WHAT ARE THE COMMON CONCURRENT MANAGER ISSUES?

+=+

Problem Description:

====================

What are the common concurrent manager issue in applications release 10?

Key words: DIAGNOSIS EXAMPLE ORACLE USDSOP PERMISSIONS LOG OUT APPLCSF CHILD

PROCESS AFPEOT REP_PATH APPLPLS APPLSYS.ENV ORACLE_SID GWYUID FNDNAM TWO_TASK

AFPCAL CPMGR AFPCSQ AFPGCP SYS SYSTEM DUAL AFPGMG PMON LOCK

FND_CONCURRENT_REQUESTS STARTMGR AFPCMT AFPSMG AFPGSR V$PROCESS V$SESSION

V$PARAMETER DBMS_LOCK DBMS_PIPE STANDARD FND_V$PROCESS FND_CONCURRENT

AFPFPROS.PLS AFPFPROB.PLS AFCPDCPS.PLS AFCPDCPB.PLS AFCPUTLS.PLS AFCPUTLB.PLS

FNDCPMGR APPLDCP AFPPRT FDXLNG NLS_LANGUAGE

+==+

Diagnostics and References:

* {7022.6,Y,100} RELEASE 10 OF APPLICATIONS

2. Soln# 2059281.6 MOST COMMON CONCURRENT MANAGER ISSUES BULLETIN

Solution ID : 2059281.6

For Problem : 1011248.6

Affected Platforms : Generic: not platform specific

Affected Products : Oracle Application Object Library

Affected Components : FOUNDATION V10.XX

Affected Oracle Vsn : Generic

Summary:

MOST COMMON CONCURRENT MANAGER ISSUES BULLETIN

+=+

Solution Description:

4/21/2011 1011248.6

C:/…/1011248.6 1/10

=====================

Common Concurrent Manager Issues in Release 10

The concurrent manager is the heart of background processing within

Oracle Applications, and is by no means trivial to control or

diagnose. This bulletin attempts to pull together common problems

encountered by the concurrent manager. It is arranged first by error

number, and then follows with a list of more general solutions to

problems.

Error: APP-00902 usdsop cannot redirect standard output

1. Make certain that the OS login for the user trying to start

the concurrent manager is the owner of the applications code, or

has been given the appropriate permission to run startmgr and

batchmgr. This can be done by a) making the startmgr script

setuid for applmgr, b) changing the group permissions on the

scripts and placing users any users in that group who should be

able to bring up the concurrent manager.

2. Check to see that the permissions are correct on all of the

log directories.

a) If $APPLCSF is set so that log and out files are written

to a common directory, then make sure the user can write to

$APPLCSF/$APPLLOG.

b) Check the permissions on $FND_TOP/$APPLLOG. This should

be done whether or not $APPLCSF is set because the internal

manager log file will always be written to

$FND_TOP/$APPLLOG.

c) If $APPLCSF is not set, check the permissions on all of

the $<prod>_TOP/$APPLLOG directories. Note: Although the

log directories are not used when $APPLCSF is set, they must

still exist under each product.

3. Check that the value of $APPLLOG is less than 8 characters in

length.

Error: APP-00912 Child process XXXXX exited with status 0. Manager

not activated.

The internal concurrent manager will only start up those managers

which have a control code of A (for Activate) or R (for restart).

Login to the applications with the System Administrator

responsibility and follow the menu path: \Navigate Concurrent

Manager Administer. For each manager which needs to be started,

tab over to the Control column and select Activate.

You will still see the above message for any managers not

4/21/2011 1011248.6

C:/…/1011248.6 2/10

activated and, although listed in the logfile, it is not a

critical error. Exiting with status 0 is an indication that a

program exited cleanly. See chapter 8 of the System

Administration manual for more information.

Error: APP-00988 ORACLE error ecountered.

This is a generic error and precedes an Oracle error. See the

section on ORACLE errors for specific examples.

Error: APP-01080 afpeot: Cannot initialize concurrent request

information for request ID XXXX.

1. Check that APPLLOG, APPLOUT, and if set, APPLCSF, are pointing

to valid directories. Shutdown the concurrent manager, start

with a clean environment (it may be easiest to logout and back in

again, and rerun the APPLSYS.env file), and bring the concurrent

manager up again.

2. The environment variable REP_PATH should only include

$AU_TOP/$APPLPLS. In previous releases it pointed to other

directories, and by default when the APPLSYS.env file is created

it sets REP_PATH to also append previous values of the variable

to itself (REP_PATH=$AU_TOP/$APPLPLS:$REP_PATH). Therefore, if

the environment file is read several times before starting the

concurrent manager, REP_PATH grows to be longer than a single

directory path, even though it is the same directory being listed

multiple times.

Error: APP-01086 Can not set profile option value

Check that the correct $ORACLE_SID, or $TWO_TASK, is being

pointed to, and also check that $FNDNAM and $GWYUID are correct.

Error: APP-01114 AFPCAL received failure code while running CPMGR

This is a generic error message which is usually at the end of a

list of several applications errors. These include but are not

limited to:

1. APP-01227 XXXXXX encountered ORA-XXXX

a. AFPCMT encountered ora-3114: not connected to oracle

Check to make sure the database is actually up, and that

the correct ORACLE_SID is being used.

b. APP-01227 AFPCSQ encountered ORA-1005: null password

Check to see that the database is up and the values are

correct for ORACLE_SID, FNDNAM, and GWYUID. Check the

PMON method being used by the concurrent manager. Often

using the RDBMS method will avoid these errors.

4/21/2011 1011248.6

C:/…/1011248.6 3/10

c. APP-01227 AFPGCP encountered oracle error SQL-02112

select into returns too many rows

Check that there is only one row in sys.dual and

system.dual.

d. APP-01227 AFPGMG encountered oracle error ORA-00923:

FROM keyword not found where expected

i. Check the permission on the $APPLLOG directories.

Note that they need to be correct for all the $APPLLOG

directories, and not just under $FND_TOP.

ii. If the sql statement being executed involves

recursive SQL, this could be base bug 246213.

iv. Check the pmon method being used, and try using the

LOCK pmon method instead.

2. If there is no log file at all, but this error message

appears when trying to start the concurrent manager, check

the following:

a. The permissions are correct on all the $APPLLOG

directories, and they actually exist. This includes

each of the $<prod>_TOP directories, and not just

FND_TOP.

b. There aren't any terminated requests in the table

FND_CONCURRENT_REQUESTS. Execute the following SQL

statement as the applsys user:

select count(*)

from fnd_concurrent_requests

where status_code='T';

c. Although the startmgr script is supposed to clean out

any requests that have a status of 'Terminating', it

occasionally balks before doing so. The solution is to

either remove these requests from the table:

delete from fnd_concurrent_requests


or change the status to 'Terminated':

update fnd_concurrent_requests

set status_code='X', phase_code='C'


3. APP-01227 AFPCMT encountered an ORACLE error. ORA-01012:

not logged on

APP-01227 AFPSMG encountered an ORACLE error. ORA-01012:

not logged on

4/21/2011 1011248.6

C:/…/1011248.6 4/10

APP-01227 AFPGSR encountered an ORACLE error. ORA-03113:

end-of-file on communication channel

Most likely the database was shutdown or crashed while

the concurrent manager was still running. Shutdown the

concurrent managers, bring up the database, and then

bring the concurrent managers back up.

Error: APP-01125 Concurrent manager encountered an error while

running XXXX

Check that the executable XXXX actually exists, and is in the

correct location. Attempt to run the executable from the command

line and see if the problem persists. Specifically compare the

arguments passed on the command line and the arguments used by

the concurrent request (this should be in the request logfile).

Finally, try relinking the executable.

Error: APP-01144 Error while finding the status of the concurrent

manager.

1. You may have a corrupt or missing view, or be missing some

grants and/or synonyms. Check that the applsys user has access

to: V$PROCESS, V$SESSION, DBMS_LOCK, DBMS_PIPE, STANDARD. Other

Applications oracle ids also need access to these views and

packages.

If the view is in fact missing or invalid, rebuild it. From the

operating system, as the applmgr user type:

FNDLIBR FND FNDCPBWV applsys/fnd SYSADMIN

'System Administrator' SYSADMIN

or as the applsys user run the following SQL statement

Create View Fnd_V$Process as

( Select P.Addr, P.Pid, S.Process Spid,

S.Osuser Username, P.Serial#,

S.Terminal, S.Program, P.Background,

P.Latchwait, S.Lockwait

From V$process P, V$session S

Where P.Addr = S.Paddr );

2. As the applsys user run the following SQL statement:

select object_name from user_objects where status='INVALID';

If any of the FND_CONCURRENT% objects are invalid, run the

following PL/SQL scripts (from $FND_TOP/install/sql) as the

applsys user:

AFPFPROS.pls

AFPFPROB.pls

4/21/2011 1011248.6

C:/…/1011248.6 5/10

AFCPDCPS.pls

AFCPDCPB.pls

AFCPUTLS.pls

AFCPUTLB.pls

NOTE: Run the scripts in the above order.

Error: APP-01167 The internal concurrent manager encountered an

error.

This is another generic error which may be followed by Oracle

errors, or:

APP-01054 Distributed concurrent manager enabled, lock

PMON method requested

APP-01114 AFPCAL received failure code while running

FNDCPMGR

Make certain that $APPLDCP is set to off or, if using

distributed concurrent processing, change the PMON

method to LOCK by running afimpmon.sql from

$FND_TOP/sql.

Error: APP-01266 in AFPPRT

Check that there aren't any requests with a status code of

'Terminating' in the table fnd_concurrent_requests by executing

the following SQL statement as the applsys user:

select count(*)



If this returns anything other than 0, then run:

delete from fnd_concurrent_requests


OR




Oracle Errors

-------------

Error: ORA-12705 in FDXLNG

This occurs because the value of nls_language is NULL in the

table FND_CONCURRENT_REQUESTS. To fix this make certain that the

environment variable $APPLLNG is set and correct (usaeng in most

cases), and also execute the following SQL statement as the

applsys user:

4/21/2011 1011248.6

C:/…/1011248.6 6/10

Update Fnd_Concurrent_Requests

Set Nls_Language = Substr(Userenv('Language'),1,

Instr(Userenv('LANGUAGE'),'_')-1),

Nls_Territory = Substr(Userenv('Language'),

Instr(Userenv('LANGUAGE'),'_')+1,

Instr(Userenv('LANGUAGE'),'.')-

Instr(Userenv('LANGUAGE'),'_')-1)

Where Nls_Language is null

or Nls_Territory is NULL;

ORA-942 Table or view does not exist (when starting up the ICM)

1. Check that the APPLDCP is set to off, unless parallel

concurrent processing is being used. Specifically DBMS_LOCK

needs to be accessible by the applsys user when APPLDCP is ON.

2. Check that the view FND_V$PROCESS exists, and that the applsys

user has access to V$PROCESS, V$SESSION, DBMS_LOCK, DBMS_PIPE,

STANDARD, and, if using distributed concurrent processing,

V$THREAD.

Generic Issues

--------------

1. The concurrent manager continuously spawns new processes although

there is no obvious activity in terms of concurrent requests.

Change the PMON method to OS. This can be done by running

afimpmon.sql from $FND_TOP/sql as the APPLSYS user. Be sure to

shut down the concurrent manager before running this script.

WARNING: Using OS PMON method is not support in 10.4.

2. The concurrent manager starts from the command line, but not from

the Administer Concurrent Manager screen.

a. This is generally a problem with permissions on the

$FND_TOP/$APPLLOG directory. Check to make sure that its

writable by the user that started the concurrent manager, and

even more importantly make sure the directory is not a symbolic

link. When starting the manager from within Applications, it

incorrectly verifies that this directory exists, but a symbolic

link returns the wrong response to the query.

b. As the applsys user run the following SQL statement:

select object_name from user_objects where status='INVALID';

If any of the FND_CONCURRENT% objects are invalid, run the

following PL/SQL scripts (from $FND_TOP/sql) as the applsys user:

AFPFPROS.pls

AFPFPROB.pls

4/21/2011 1011248.6

C:/…/1011248.6 7/10

AFCPDCPS.pls

AFCPDCPB.pls

AFCPUTLS.pls

AFCPUTLB.pls

NOTE: Run the scripts in the above order.

3. When starting the concurrent manager FNDLIBR dumps core.

Try relinking FNDLIBR, if that does not check to see if there are

pending requests. If possible, make certain there aren't any

jobs in pending status, and then try to bring the manager up.

This is especially true if there are a high number of requests

with a status of pending because the concurrent manager reads in

all pending requests on startup.

To delete pending requests execute the following statement as

APPLSYS user:

4. The specialization rules that were just recently defined are not

taking effect.

Try bouncing the CM to make sure that all the current rules are

being implemented.

5. On VMS starting the manager results in an error indicating that the

user should verify they are logged in as the applmgr user, when in

fact the user already is.

Make sure that the queue for concurrent requests exists. By

default this queue is called FND$STDMGR.

6. When navigating to the concurrent requests screen (\Navigate

Concurrent Requests from the System Administrator Responsibility) it

justs sits there and hangs.

This is most likely due to base bug 246213. Make certain this

patch has been applied. It is actually a 7.1.3 RDBMS bug which

affects complicated views.

7. Although $APPLCSF is set, the internal concurrent manager log is

being written to $FND_TOP/$APPLLOG.

This has been logged as bug 259415. At this time there is no

patch for this, but it can be modified by changing the startmgr

(or startmgr.com) script to use $APPLCSF instead of $FND_TOP when

creating the logfile.

8. All the concurrent managers are running, and the concurrent manager

has been successfully bounced several time, but all the requests have

a status of 'Inactive' and 'No Manager'.

Check that there aren't any pending requests at all. If an

run-alone immediate concurrent program is submitted, but the

program is not registered in any of the concurrent libraries then

the status will be set to 'Inactive' and 'No Manager' because

4/21/2011 1011248.6

C:/…/1011248.6 8/10

none of the managers could run it. Any requests that follow this

request will then be unable to run until the original request

completes.

9. The concurrent manager forms within application indicate that the

concurrent manager is down, when in fact it is running and processing

requests without any problems.

This is a bug when using the LOCK pmon method. It is fixed in

10.4.1 and the workaround is to use RDBMS or OS pmon method.

10. Terminated concurrent processes that are "spawned" do not die,

although the View Concurrent Requests screen says that have been

Terminated. Also, after some time the Concurrent Requests screen will

changed the status to 'Completed Normal'.

This is bug 249158, and is known to occur on at least HP. A

patch is available.

11. How to recreate the view FND_V$PROCESS:

FNDLIBR FND FNDCPBWV applsys/fnd SYSADMIN 'System Administrator'

SYSADMIN

or as the applsys user execute the following SQL statement:

Create View Fnd_V$Process as

( Select P.Addr, P.Pid, S.Process Spid,

S.Osuser Username, P.Serial#,

S.Terminal, S.Program, P.Background,

P.Latchwait, S.Lockwait

From V$process P, V$session S

Where P.Addr = S.Paddr );

12. If there are multiple internal concurrent managers in the same

database, only one of those managers can use the LOCK pmon method.

13. When using parallel concurrent processing, the concurrent manager

names can not have embedded spaces in them. The workaround is to

replace the space with an underscore, or change the name to a single

word.

------------------------------------------------

When in doubt....

If you are encountering errors with the concurrent managers some steps

that can be safely executed include:

1. Bounce the concurrent managers. Bring them down, and then

making sure the environment is properly set up bring them back up

again. Remember that the concurrent managers' environment is set

at the time they start up, and any modifications that you make to

the environment after that will not be picked up until the

concurrent managers are bounced again.

2. Check for the existence of the $APPLLOG directory under each

PRODUCT_TOP directory, and that the permissions allow for writing

4/21/2011 1011248.6

C:/…/1011248.6 9/10

to those directories. If $APPLCSF is set, so that all the log

and out files get written to a common directory, the $APPLLOG

directories still must exist. Also, even with $APPLCSF set, the

internal concurrent manager log files will always be written to

$FND_TOP/$APPLLOG.

3. Make certain the login id that is starting the concurrent

manager has permission to do so. By default just applmgr can do

this.

4. Check that there aren't any requests which have a status of

'Terminating' in the table fnd_concurrent requests:

select count(*)



If there are any such rows then delete them from the table, or

modify them so they have status of 'Terminated':

delete from fnd_concurrent_requests where status_code='T';

OR




+==+

References:

4/21/2011 1011248.6

C:/…/1011248.6 10/10

common cm issue

Documents