APPLIES TO:
Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Database Exadata Express Cloud Service - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
SYMPTOMS:
Clusterware fails to start up due to ASM instance fails to start with ORA-4031:
[root@racnode1 bin]# ./crsctl start cluster
CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-04031: unable to allocate 16416 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","dbktb: trace buffer")
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/11.2.0/log/mcxoradb/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.asm' on 'racnode1' failed
CRS-2679: Attempting to clean 'ora.asm' on 'racnode1'
CRS-2681: Clean of 'ora.asm' on 'racnode1' succeeded
CRS-4000: Command Start failed, or completed with errors.
CAUSE:
The problem is caused by db_cache_size is wrongly set to 1GB for ASM instance which in turn causing ASM fails to start with ORA-4031.
Per alert_+ASM1.log:
Sun Sep 16 19:24:42 2012
MEMORY_TARGET defaulting to 411041792.
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal)
...
Mon Sep 17 11:26:56 2012
ALTER SYSTEM SET db_cache_size='1G' SCOPE=SPFILE SID='*';
Mon Sep 17 11:27:33 2012
ALTER SYSTEM SET db_cache_size='1G' SCOPE=SPFILE;
Mon Sep 17 14:15:16 2012
ALTER SYSTEM SET db_cache_size='1G' SCOPE=SPFILE SID='+ASM1';
...
Mon Sep 17 14:18:39 2012
Shutting down instance (abort)
License high water mark = 11
Mon Sep 17 14:18:44 2012
MEMORY_TARGET defaulting to 1543503872.
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal)
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_3376.trc (incident=116161):
ORA-04031: unable to allocate 16416 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","dbktb: trace buffer")
Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_116161/+ASM1_ora_3376_i116161.trc
Use ADRCI or Support Workbench to package the incident.
ASM instance is not a database instance, it does not require db_cache_size to be set. By default it is 0. By setting this parameter, it reduces the shared_pool_size which can be used by ASM instance, hence lead to ORA-4031.
SOLUTION:
1. Stop clusterware processes on all nodes
as root user:
# crsctl stop crs -f
2. Start clusterware on node 1 in exclusive mode
as root user:
# crsctl start crs -excl
3. Create a pfile (for example: /tmp/asmpfile.ora) with the content from alert_+ASM<x>.log under the section "System parameters with non-default values:", here is a simple example, please ensure to alter it per your ASM requirement:
MEMORY_TARGET=1543503872
large_pool_size = 12M
instance_type = "asm"
remote_login_passwordfile= "EXCLUSIVE"
asm_power_limit = 1
diagnostic_dest = "/u01/app/oracle"
Save the file, then start ASM instance manually as grid user, recreate spfile on its original diskgroup, eg:
$ sqlplus / as sysasm
SQL> startup pfile='/tmp/asmpfile.ora';
SQL> create spfile='+DATADG' from pfile='/tmp/asmpfile.ora';
SQL> shutdown immediate
4. Stop clusterware on node 1 and restart clusterware on all nodes:
as root user:
# crsctl stop crs -f
# crsctl start crs
5. Wait for a while, verify the clusterware status via:
$ crsctl stat res -t
Recent Posts
