java.lang.OutOfMemoryError: unable to create new native thread
Then check for the below logfiles of 12c OEM agent,
$AGENT_HOME/sysman/log/emagent.nohup
emagent.nohup:
Agent is going down due to an OutOfMemoryError
—– Sat Sep 6 01:28:02 2014::31358::Checking status of EMAgent : 1685 —–
—– Sat Sep 6 01:28:02 2014::31358::EMAgent exited at Sat Sep 6 01:28:02 2014 with return value 57. —–
—– Sat Sep 6 01:28:02 2014::31358::EMAgent will be restarted because of an Out of Memory Exception. —–
—– Sat Sep 6 01:28:02 2014::31358::EMAgent is Thrashing. Exiting loop. —–
—– Sat Sep 6 01:28:02 2014::31358::Commiting Process death. —–
—– Sat Sep 6 01:28:02 2014::31358::writeAbnormalExitTimestampToAgntStmp: exitCause=OOM : restartRequired=0 —–
—– Sat Sep 6 01:28:02 2014::31358::Exited due to Thrash. —–
gcagent.log
2014-09-06 01:27:55,220 [57:GC.Executor.0 (oracle_database:abc.oracle.com7:problemTbsp_10i_Dct)] ERROR – Critical error:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.lang.UNIXProcess$1.run(UNIXProcess.java:141)
at java.security.AccessController.doPrivileged(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:103)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
at oracle.sysman.gcagent.common.SafeProcessBuilder.start(SafeProcessBuilder.java:120)
at oracle.sysman.gcagent.util.channel.OSChannel.runStreamingCommand(OSChannel.java:414)
at oracle.sysman.gcagent.util.channel.OSChannel.runCommand(OSChannel.java:449)
We are getting the above error because,
The kernel nproc limit is set too low and needs to be increased.
The more targets the agent is monitoring, the higher the nproc limit must be. We have more than 20 instance running in the server. So we have increased the nproc limit to unlimited as below and started the agent without any issues,
From:
[oracle@abc] ulimit -Su
1024
[oracle@abc] ulimit -Hu
16384
To:
[oracle@abc] ulimit -Su
unlimited
[oracle@abc] ulimit -Hu
unlimited
