Dbabc.Net

春节上班第一天，收到如下mail信息，先说下环境：
环境：Agent Version : 10.2.0.5.0
OMS Version: 10.2.0.5.0
Repository DB Version:10.2.0.4OS Version：RHEL4U8
邮件标题：

EM Alert: Unreachable Start:DBABC.NET – Agent has stopped monitoring. The following errors are reported : COLL_DISABLED|DISK_FULL.

邮件正文：

Target Name=DBABC.NET_GC
Target type=Host
Host=ＧＣ.DBABC.NET
Occurred At=Feb 10, 2011 12:38:56 PM (UTC+08:00)
Message=Agent has stopped monitoring. The following errors are reported : COLL_DISABLED|DISK_FULL.
Severity=Unreachable Start
Acknowledged=No
Notification Rule Name=Host Availability and Critical States
Notification Rule Owner=SYSMAN

刚看到mail信息,就想到磁盘空间满了,立马登录到服务器上查看。

[root@dbabc.net ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hdb8             2.0G  299M  1.6G  16% /
/dev/hdb1             494M   18M  451M   4% /boot
/dev/md0              587G  299G  259G  54% /md0
none                  989M     0  989M   0% /dev/shm
/dev/hdb5             9.9G   56M  9.3G   1% /home
/dev/hdb7             2.0G   36M  1.9G   2% /tmp
/dev/hdb9             109G  102G  1.6G  99% /u01
/dev/hdb3             9.9G  7.2G  2.2G  77% /usr
/dev/hdb2             9.9G  340M  9.1G   4% /var

发现没有100%的磁盘出现。发现/u01下的磁盘99%了。心想应该没问题啊，还剩余1.6G呢，怎么会满呢，再查看df -i

[root@dbabc.net ~]# df -h
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/hdb8               256K    9.6K    247K    4% /
/dev/hdb1               128K      41    128K    1% /boot
/dev/md0                 75M     14K     75M    1% /md0
none                    248K       1    248K    1% /dev/shm
/dev/hdb5               1.3M     171    1.3M    1% /home
/dev/hdb7               256K      32    256K    1% /tmp
/dev/hdb9                14M     80K     14M    1% /u01
/dev/hdb3               1.3M    314K    967K   25% /usr
/dev/hdb2               1.3M    9.4K    1.3M    1% /var

也是相当的没问题啊。查看gc的日志信息发现

vi $AGENT_HOME/sysman/log/emagent.trc
……
2011-02-10 22:39:06,366 Thread-3811433376 ERROR upload: Exceeded max. amount of upload data: 1 files, 0.000000 MB Data. 98.03% of disk used. Disabling collections.
2011-02-10 22:39:06,414 Thread-3811433376 WARN  collector: Disable collector
2011-02-10 22:39:06,483 Thread-3811433376 ERROR upload: Disabling Logging and Tracing: 98.03% of disk used
……

难道超过98%就Disable，查找资料发现$AGENT_HOME/sysman/config/emd.properties中有限制

#
# The maximum amount (%) of disk space that can be used on the EMD filesystem
# before the following is disabled:
#    – Collection of data (upload manager)
#    – Logging and tracing
#@description=
#@valueType=Int
#@LOV=
#@default=98
#
UploadMaxDiskUsedPct=98

#
# The amount (%) of disk space that can be used on the EMD filesystem
# before the following is re-enabled after being disabled previously:
#    – Collection of data (upload manager)
#    – Logging and tracing
#@description=
#@valueType=Int
#@LOV=
#@default=95
#
UploadMaxDiskUsedPctFloor=95

原来默认的设置的限制是98%,怪不得呢，本来想将其修改为100%一想算了吧，还是将磁盘空间清理一下吧。注意要想再次启用的话至少要保证有95%的磁盘空间。
查看了一下GC中的相关信息，发现警告/u01在超过80%的时候就开始警告了，由于当时的策略设置的本机不发mail预警，所以这警告信息没有及时的发现，所以制定一个周全的预警机制是很有必要的，以此为记。

附：修改UploadMaxDiskUsedPct值的方法

1. Stop the agent.
2. Edit $AGENT_HOME/sysman/config/emd.properties file and set
UploadMaxDiskUsedPctFloor=100
UploadMaxDiskUsedPct=100
3. Start the agent and upload.

这样的话只要磁盘不达到100%就可以一直使用，不过这是临界值哦，达到100%的时候其他的东东也会报错哦。

附：
Bug 9236295: ALERT: AGENT HAS STOPPED MONITORING – COLL_DISABLED|DISK_FULL LIKE BUG;6780904

显示 Bug 属性 Bug 属性
类型 B – Defect 已在产品版本中修复 10.2.0.5.0
严重性 2 – Severe Loss of Service 产品版本 10.2.0.5.0
状态 92 – Closed, Not a Bug 平台 912 – Microsoft Windows (32-bit)
创建时间 22-Dec-2009 平台版本 2003 R2
更新时间 17-Feb-2010 基本 Bug -
数据库版本 N/A
影响平台 Generic
产品源 Oracle

显示相关产品 相关产品
产品线 Enterprise Management 系列 Enterprise Manager Consoles, Packs, and Plugins
区域 Enterprise Manager Grid Control 产品 1370 – Enterprise Manager Grid Control

Hdr: 9236295 N/A AGENT 10.2.0.5.0 PING PRODID-1370 PORTID-912
Abstract: ALERT: AGENT HAS STOPPED MONITORING – COLL_DISABLED|DISK_FULL LIKE BUG;6780904

*** 12/22/09 11:13 am ***
TAR
—

Problem Description
——————-
Rec’d an email from the OMS server stating that the agent was not reachable.
Below is the text of the email:

Target Name=.:3872
Target type=Agent
Host=.
Occurred At=Dec 14, 2009 6:04:51 PM MST
Message=Agent has stopped monitoring. The following errors are reported :
COLL_DISABLED|DISK_FULL.
Severity=Unreachable Start
Acknowledged=No
Notification Rule Name=Agents Unreachable – Prod
Notification Rule Owner=COREHOURSMONITOR

Also rec’d the following in the emagent.trc file:

2009-12-14 22:11:29,244 Thread-2988 WARN recvlets.aq: [oracle_database
2009-12-14 22:12:08,635 Thread-1240 ERROR upload: Exceeded max. amount of
upload data: 0 files, 0.049802 MB Data. 98.85% of disk used. Disabling
collections.

Environment Information
———————–
Windows 2003 w/ SP2
Agent v10.2.0.5

Test Case Step-by-Step Instructions
———————————–
n/a

Test Case Location
——————
n/a

Diagnostic Analysis
——————-
According to Bug:6780904, this was fixed in Agent 10.2.0.5; However, customer
is running 10.2.0.5. Attempted the following solution from unpublished
Note:734643.1 without success:

1. Stop the Agent
2. Backup then edit /sysman/config/emd.properties and update it
with the below entries.
UploadMaxBytesXML=100 UploadMaxNumberXML=10000 and
UploadMaxDiskUsedPct=100
3. Restart the agent.

Performance
———–

NLS Information
—————

Patches
——-

Log Files Location
——————

Reproducibility
—————
customer

URL
—
n/a

Did you test with the latest version?
————————————-
yes

Available Workarounds
———————
none that worked

Related Bugs
————
Bug:6780904

Severity 1 Information
———————-

Additional Information
———————-

*** 12/22/09 11:19 am ***
*** 12/22/09 11:20 am *** (CHG: Sta->16)
*** 01/05/10 04:25 am *** (CHG: Asg->NEW OWNER OWNER)
*** 01/05/10 05:42 am *** (CHG: Sta->10)
*** 01/05/10 05:42 am ***
*** 02/17/10 04:39 am *** (CHG: Sta->32)
*** 02/17/10 04:39 am ***
*** 02/17/10 08:31 am ***
*** 02/17/10 08:31 am *** (CHG: Fixed->10.2.0.5.0)
*** 02/17/10 08:31 am *** (CHG: Sta->90)
*** 02/17/10 08:59 am *** (CHG: Sta->92)