Failes To start Manged Process

OPMNCTL Failed to Start Managed Process After Maximum Retry Limit on Solaris 10, OBIEE 11g 

If you've happened to glance at Oracle Technical Network, it appears that alot of practitioners are encountering an issue when upgrading their 11g platform to the newest 11.1.1.6.4 version if they're using a Solaris or Linux distribution.

After upgrading to 11.1.1.6.4 some users encounter the following error(s) when trying to start their BI services using opmnctl (Oracle Process Manager and Notification Server) with the 'opmnctl startall' command:

=====================================================================
Response: 1 of 5 processes started.

ias-instance id=instance1
++++++++++++++++++

ias-component/process-type/process-set:
coreapplication_obis1/OracleBIServerComponent/coreapplication_obis1/

Error
--> Process (index=1,uid=1445883997,pid=18522)
failed to start a managed process after the maximum retry limit

Log:
/u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBIServerComponent/coreapplication_obis1/console~coreapplication_obis1~1.log

ias-component/process-type/process-set:
coreapplication_obips1/OracleBIPresentationServicesComponent/coreapplication_obips1/

Error
--> Process (index=1,uid=1445883998,pid=0)
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/web/bin/sawserver
failed to start a managed process after the maximum retry limit
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/web/bin/sawserver

Log:
none

ias-component/process-type/process-set:
coreapplication_obisch1/OracleBISchedulerComponent/coreapplication_obisch1/

Error
--> Process (index=1,uid=1445884000,pid=18523)
failed to start a managed process after the maximum retry limit
Log:
/u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBISchedulerComponent/coreapplication_obisch1/console~coreapplication_obisch1~1.log

ias-component/process-type/process-set:
coreapplication_obiccs1/OracleBIClusterControllerComponent/BIClusterController/

Error
--> Process (index=1,uid=1445884001,pid=0)
Executable file does not have execute permission
/u01/app/oracle/obiee11g/Oracle_BI1/bifoundation/server/bin/nqsclustercontroller
failed to start a managed process after the maximum retry limit 

The important things to note are:
You may encounter all, some, or just 1 of your bi_services which fail to start. In the above example, the following services failed to start:
BI Server
BI Publisher
BI Cluster Controller
BI Scheduler
There are actually two issues:
Managed Processes failed to start after the maximum retry limit
Certain executable files do not have the execute permission enabled

Issue 1: Fix Maximum Retry Limit

This can be easily resolved by setting the ulimit at the user level to : ulimit -d unlimited , as outlined in Oracle's note ID: 1292887.1 . Anyone who has worked in OBIEE 10g should be familiar with this as it is required in 10g's installation process as well.

Issue 2: Enable read access on executable files

If you tried re-running opmnctl startall after resolving issue 1, you'll notice that the BI services still fail to run despite increasing the maximum retry limit. Upon examining the log files as outlined in the above error:

u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBIServerComponent/coreapplication_obis1/console~coreapplication_obis1~1.log
u01/app/oracle/obiee11g/instances/instance1/diagnostics/logs/OracleBISchedulerComponent/coreapplication_obisch1/console~coreapplication_obisch1~1.log

You'll notice that OPMNCTL still fails because it can't execute: nqsserver, nqscheduler and nqsclustercontroller. Below is the log from the above bullet point #1 :

/export/obiee/11g/Oracle_BI1/bifoundation/server/bin/biserver.sh: nqsserver: cannot execute
12/11/06 15:48:06 Start process

/export/obiee/11g/Oracle_BI1/bifoundation/server/bin/biserver.sh: nqsserver: cannot execute

During the 11.1.1.6.4 upgrade on your Solaris or Linux box, these files became read only!

How to resolve this read only issue:

Navigate to your $ORACLE_HOME/bifoundation/server/bin folder and make the the following files executable by using the chmod +x command: nqscheduler, nqsclustercontroller, nqsserver. These files should have read/write privileges as outlined in the image below:





Now when you run opmnctl startall, all services should execute successfully:




Important:
After the upgrade, it is critical that you follow Oracle's recommended start up & shutdown sequence for weblogic & OBIEE 11g

Start up:


$DOMAIN_HOME/bin/startWebLogic.sh (Weblogic)

$WL_HOME/server/bin/startNodeManager.sh (Node Manager)

$DOMAIN_HOME/bin/startManagedWebLogic.sh bi_server1 http://server.domain:7001 (Managed erver)


$ORACLE_INSTANCE/bin/opmnctl startall (OPMN) 

Shutdown:


$ORACLE_INSTANCE/bin/opmnctl stopall (OPMN)


$DOMAIN_HOME/bin/stopManagedWebLogic.sh bi_server1 t3://host.domain:7001 username password (Managed Server)


$DOMAIN_HOME/bin/stopWebLogic.sh username password t3://host.domain:7001 (Admin Server)

Thanks,


Satya Ranki Reddy