Trying to register the database manually, it fails with:
$ srvctl add database -d <dbname> -o <ORACLE_HOME>
PRCR-1006 : Failed to add resource ora.<dbname>.db for uat
PRCT-1011 : Failed to run “osdbagrp”. Detailed error: null
Testing osdbagrp command with the following options, the result shows:
$ osdbagrp -d <<< works
$ osdbagrp -a <<< segmentation fault
$ osdbagrp -o <<< works
When checked in the OS log, /var/log/messages, it reports the following segmentation fault
Feb 11 11:59:30 node1 kernel: osdbagrp[21751]: segfault at 100000000 ip 000000367965eff0 sp 00007fffab30ddf0 error 4 in libc-2.5.so[3679600000+14f000]
Cause:
The binary osdbagrp in the RDBMS ORACLE_HOME/bin is corrupted for some reason.
Solution:
1. Regenerate the osdbagrp binary, as RDBMS software owner
Either relink the osdbagrp executable only (preferred):
$ cd $ORACLE_HOME/srvm/lib
$ make -f ins_srvm.mk iosdbagrp
or alternatively relink all executables, however this will require to shutdown all instances running from this ORACLE_HOME:
$ cd $ORACLE_HOME/bin
$ relink all
2. Copy the osdbagrp binary from other nodes of the cluster if it is working fine on the other nodes
cp /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/osdbagrp /u01/app/oracle/product/12.1.0.2/dbhome_3/bin/osdbagrp
Retry srvctl add database command.
