Untitled

Our system has a system shutdown in which OID 11.1.1.9
was running and not starting up using opmnctl after server is brought up. A few
things I went through to fix it up is posted below.

Check the status from the Oracle Instance home as below.

[apploid@oam01 bin]$ ./opmnctl status

Processes in Instance: asinst_1

———————————+——————–+———+———

ias-component                    | process-type       |    
pid | status

———————————+——————–+———+———

ohs1                             | OHS                |   19283 | Alive

ovd1                             | OVD                |   19282 | Alive

oid1                             | oidldapd           |    
N/A | Down

oid1                             | oidldapd           |    
N/A | Down

oid1                             | oidmon             |    5562 | Stop

EMAGENT                          | EMAGENT            |  
19281 | Alive

You can see oidmon is stopped and oidldap is down.
Normally opmnctl startall can bring up in almost all the cases, but I was
getting time outs and stuck at below in the oidmon****.log file.

[2015-09-04T13:23:59.800201+10:00] [OID]
[NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19273] [tid: 0]
OIDMON_STOP: Thread started

[2015-09-04T13:23:59.807343+10:00] [OID]
[NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19273] [tid: 0]
OIDMON_STOP: Connecting to database, connect string is oiddb

I did check the database connectivity from the OID
application server, checked the ODS schema status and almost everything
including the status in ODS_PROCESS_STATUS (I will post about this table soon).

Since we had a abrupt shutdown, I guessed we have some
uncleaned files in the directory below. Removed the files after taking the
backup after shutting down everything, then try starting up.

$ORACLE_INSTANCE/bin/opmnctl stopall

cd $ORACLE_INSTANCE/config/OPMN/opmn/states

[apploid@oam01 states]$ ls

p1515733570

[apploid@oam01 states]$ cd ..

[apploid@oam01 opmn]$ mv states states_bak

[apploid@oam01 opmn]$ mkdir states

$ORACLE_INSTANCE/bin/opmnctl startall

Check the log file

[apploid@oam01 opmn]$ tail -f
/u01/oid/Oracle/Middleware/asinst_1/diagnostics/logs/OID/oid1/oidmon-0000.log

[2015-09-04T13:33:43.438248+10:00] [OID] [NOTIFICATION:16]
[] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian: Starting
OIDLDAPD  cmdLine args = “instance=1
inst=1  “

[2015-09-04T13:33:43.446457+10:00] [OID]
[NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0]
Guardian: Starting OIDLDAP Server,PID=19818

[2015-09-04T13:33:43.449019+10:00] [OID]
[NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0]
Guardian: Updating Process Table…

[2015-09-04T13:33:43.452984+10:00] [OID] [NOTIFICATION:16]
[] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0] Guardian:
gsldda_DeleteEntry:gsldfgGetEntryID Error..1010

[2015-09-04T13:33:43.707887+10:00] [OID]
[NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0]
Guardian: CreateInstance: Successfuly created entry
cn=asinst_1_oid1_1_oam01.saroj.com.au,cn=osdldapd,cn=subregistrysubentry

[2015-09-04T13:33:53.403166+10:00] [OID]
[NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0]
Guardian: gslmmonOpmnProcStatus: Sending OIDLDAPD PID=,19956 status to OPMN

[2015-09-04T13:33:55.405243+10:00] [OID]
[NOTIFICATION:16] [] [OIDMON] [host: oam01.saroj.com.au] [pid: 19771] [tid: 0]
Guardian: gslmmonOpmnProcStatus:Sending OID Host and ports info to OPMN

         
HOST=oam01.saroj.com.au LDAP NONSSL PORT = 3060 LDAP SSL PORT = 3131

[apploid@oam01 bin]$ ./opmnctl status

Processes in Instance: asinst_1

———————————+——————–+———+———

ias-component                    | process-type       |    
pid | status

———————————+——————–+———+———

ohs1                             | OHS                |   19773 | Alive

ovd1                             | OVD                |   19772 | Alive

oid1                             | oidldapd           |  
19956 | Alive

oid1                             | oidldapd           |  
19818 | Alive

oid1                             | oidmon             |  
19771 | Alive

EMAGENT                          | EMAGENT            |  
19770 | Alive