OnaCorpuscle ~ Human-Tech Duality

EM12c ConfigStateMgr:798 errors (storage_reporting_data) Troubleshooting

EM12c ConfigStateMgr:798 errors

(storage_reporting_data) Troubleshooting
This post http://www.snapdba.com/2013/02/em12c-configstatemgr-798-errors-storage_reporting_data/
associates these error messages to bug 13547855 which in most cases are related to Oracle Cloud
Control agents deployment into an environment. Applying the procedure described in the Oracle Support
document referenced can be set.
However, not always like this, despite following the instructions to fix it, the console continues repeatedly
reporting these error messages. If you review the log file where the bug reports see:
# cat /u01/app/12cagent/agent_inst/sysman/log/gcagent_errors.log | grep "2013-09-09"
2013-09-09 13:53:19,563 [201607:GC.Executor.75083 (host:srva0d1:host_storage)
(host:srva0d1:host_storage:storage_reporting_keys)] ERROR - null
2013-09-09 13:53:19,564 [201607:GC.Executor.75083 (host:srva0d1:host_storage)
(host:srva0d1:host_storage:storage_reporting_keys)] ERROR host:srva0d1:host_storage:storage_reporting_keys
2013-09-09 13:53:19,564 [201607:GC.Executor.75083 (host:srva0d1:host_storage)] ERROR Critical error:
2013-09-09 13:53:19,701 [201607:GC.Executor.75083 (host:srva0d1:host_storage)
(host:srva0d1:host_storage:storage_reporting_alias)] ERROR - null
2013-09-09 13:53:19,702 [201607:GC.Executor.75083 (host:srva0d1:host_storage)
(host:srva0d1:host_storage:storage_reporting_alias)] ERROR host:srva0d1:host_storage:storage_reporting_alias
2013-09-09 13:53:19,702 [201607:GC.Executor.75083 (host:srva0d1:host_storage)] ERROR Critical error:
2013-09-09 13:53:20,346 [201731:oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl Incident Dump Executor (created: Mon Sep 09 13:53:19 CEST 2013)] ERROR - Result set
exceeded max flood control level
2013-09-09 13:53:20,567 [201731:oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl Incident Dump Executor (created: Mon Sep 09 13:53:19 CEST 2013)] ERROR - Result set
exceeded max flood control level
We see the query executed by the function of the agent program returns "Result set exceeded max flood
control level" (in next blog post I will explain how to fix it by modifying the code of the perl program
responsible for the incident) but there may be other errors. So, we follow the procedure and check the
contents of the log file with the errors of the agent related to data storage::
# cat $AGENT_HOME/agent_inst/sysman/log/gcagent_errors.log | grep storage_reporting_data
2013-09-04 13:53:19,368 [101557:GC.Executor.39108 (host:srva0d1:host_storage)
(host:srva0d1:host_storage:storage_reporting_data)] ERROR - null
2013-09-04 13:53:19,373 [101557:GC.Executor.39108 (host:srva0d1:host_storage)
(host:srva0d1:host_storage:storage_reporting_data)] ERROR host:srva0d1:host_storage:storage_reporting_data
2013-09-05 13:53:19,554 [121731:GC.Executor.46652 (host:srva0d1:host_storage)
(host:srva0d1:host_storage:storage_reporting_data)] ERROR - null
2013-09-05 13:53:19,559 [121731:GC.Executor.46652 (host:srva0d1:host_storage)
(host:srva0d1:host_storage:storage_reporting_data)] ERROR host:srva0d1:host_storage:storage_reporting_data
2013-09-07 13:53:19,332 [161745:GC.Executor.60863 (host:srva0d1:host_storage)
(host:srva0d1:host_storage:storage_reporting_data)] ERROR - null
2013-09-07 13:53:19,337 [161745:GC.Executor.60863 (host:srva0d1:host_storage)
(host:srva0d1:host_storage:storage_reporting_data)] ERROR host:srva0d1:host_storage:storage_reporting_data
If we analyze the nature of the message can deduce that the agent Oracle Cloud Control is not getting
data from a target defined in its configuration storage:

Author: Xavier Picamal

01/02/14

1/2

OnaCorpuscle ~ Human-Tech Duality

EM12c ConfigStateMgr:798 errors (storage_reporting_data) Troubleshooting

2013-09-09 13:53:19,408 [201607:GC.Executor.75083 (host:srva0d1:host_storage)

(host:srva0d1:host_storage:storage_reporting_data)] ERROR - null
2013-09-09 13:53:19,412 [201607:GC.Executor.75083 (host:srva0d1:host_storage)
(host:srva0d1:host_storage:storage_reporting_data)] ERROR host:srva0d1:host_storage:storage_reporting_data
In our scenario, we recently had a change in the disks configuration of the server responsible for the
incident, if we review the content of the file is shown below:
# cat $AGENT_HOME/agent_inst/sysman/emd/state/configstate/host/$(hostname s)/
storage_reporting_issues.xml
<DC CN="" TS="2013-09-05T13:53:20.008+02:00"
TI="TARGET_GUID=D5215C2D28F0768DE043428814AC10B0" TN="srva0d1" TT="host">
<RS TS="2013-09-05T13:53:20.005+02:00" MN="storage_reporting_issues">
<RH>
<CI NM="type">ERROR</CI>
<CI NM="message_counter">10</CI>
<CI NM="message_id">ERROR_INST_MAPPING</CI>
<CI NM="message_params">Filesystem /dev/mapper/vg00-LV04_var</CI>
<CI NM="action_id">ACTION_INST_RESOLV_ISSUE</CI>
<CI NM="action_params"/>
</RH>
<RC>ERROR,1,ERROR_INST_NO_GID,DISK /dev/sdd,ACTION_INST_RESOLV_ISSUE,</RC>
We confirm that can't read or access to the volume, /dev/sdd. We checked from the file system as root:
# fdisk /dev/sdd
Unable to read /dev/sdd
# /opt/mpp/lsvdev | grep sdd
# pvscan
/dev/sdd: read
/dev/sdd: read
/dev/sdd: read
/dev/sdd: read
PV /dev/sdb
PV /dev/sda2
Total: 2 [1.95

failed after 0 of 4096 at

failed after 0 of 4096 at
failed after 0 of 4096 at
failed after 0 of 4096 at
VG vg01
lvm2 [1.82 TB /
VG vg00
lvm2 [135.84 GB
TB] / in use: 2 [1.95 TB]

0: Input/output error
107374116864: Input/output error
107374174208: Input/output error
4096: Input/output error
837.71 GB free]
/ 30.84 GB free]
/ in no VG: 0 [0
]

And indeed, we have no access to volume /sdd/. We have to fix this problem in the configuration of the
operating system or replace the disk if in poor condition.
login as: root
root@ srva0d1 password:
Last login: Thu Sep 12 07:50:14 2013 from 172.24.24.164
[root@ srva0d1 ~]# pvscan
PV /dev/sdb
VG vg01
lvm2 [1.82 TB / 837.71 GB free]
PV /dev/sda2
VG vg00
lvm2 [135.84 GB / 30.84 GB free]
Total: 2 [1.95 TB] / in use: 2 [1.95 TB] / in no VG: 0 [0

Once done, following the procedure described in the Oracle Support document, messages disappears from
em12c console.

Author: Xavier Picamal

01/02/14

2/2