ORACLE OMS启动失败之BEA-300040 I/O error while writing node manager status

        今天处理一个oracle优化管理工具oms启动异常的问题。oms问题处理之前，oms所在服务器上的一套asm和db所用

的审计目录文件系统inode耗尽导致asm和db异常宕机，经清理审计目录的日志后asm和db正常启动，但是oms启动异常。

一、OMS启动异常分析

OMS启动异常信息

[oracle@orcl bin]$ ./emctl start oms

Oracle Enterprise Manager Cloud Control 12c Release 4  

Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.

Starting Oracle Management Server...

Starting WebTier...

WebTier Successfully Started

Oracle Management Server Could Not Be Started

Check EM Server log file for details: /u01/app/oracle/oms/domains/GCDomain/servers/ORCL_OMS1/logs/ORCL_OMS1.out

Oracle Management Server is Down

[oracle@orcl bin]$ 

查看ORCL_OMS1.out日志

Mar 19, 2020 5:55:44 PM oracle.security.jps.internal.audit.AuditServiceImpl registerInternal

WARNING: Cannot register to audit service for component "JPS".

Mar 19, 2020 5:55:44 PM oracle.security.jps.internal.policystore.ldap.LdapPolicyStore initial

INFO: Container is not ready at the point null

<Mar 19, 2020 5:55:53 PM CST> <Error> <NodeManager> <BEA-300040> <I/O error while writing node manager status change message.

java.io.IOException: No space left on device

        at java.io.UnixFileSystem.createFileExclusively(Native Method)

        at java.io.File.checkAndCreate(File.java:1705)

        at java.io.File.createTempFile0(File.java:1726)

        at java.io.File.createTempFile(File.java:1803)

        at weblogic.nodemanager.util.ConcurrentUnixFile.write(ConcurrentUnixFile.java:38)

        Truncated. see log file for complete stacktrace

> 

<Mar 19, 2020 5:55:54 PM> <FINEST> <NodeManager> <Waiting for the process to die: 6906>

<Mar 19, 2020 5:55:54 PM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

<Mar 19, 2020 5:55:54 PM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

根据.out日志提示，跟之前asm和db启动异常报错一致，也是之前oracle数据库审计日志过多导致文件系统inode耗尽

其他进程无法写入导致。

二、问题处理

对于oms来说，单纯的重启oms无法解决问题，因为nodemanager进程未正常停止

使用ps -ef|grep java，把oms相关的java进程杀掉kill -9 java_process_number

再次尝试启动oms进程，成功启动