CLARiON Panic

>
>
>
>
>
Both the DMs panicked and hence no failover

All FS Unmounted
LCC issues on the backend
No dumps got generated, neither in Dump slots
Support Materials and SP Collects are attached to the SR
Similar issues occurred on SR 70918566 and SR 70891766 (This is the third time s
ame issues on Clariion has occurred), CE action has been performed on the backen
d.
Currently SP Collects are being checked again internally.
*** slot_0 primary control station ***

[nasadmin@ws1-whs-nascont1 dump]$ nas_version
5.6.47-11
[nasadmin@ws1-whs-nascont1 dump]$ server_version ALL
server_2 : Product: EMC Celerra File Server Version: T5.6.47.11
server_3 : Product: EMC Celerra File Server Version: T5.6.47.11
[nasadmin@ws1-whs-nascont1 tools]$ nas_server -l
id
type acl slot groupID state name
1
1
0
2
0
server_2
2
4
0
3
0
server_3
[nasadmin@ws1-whs-nascont1 tools]$ /nas/sbin/getreason
10 - slot_0 primary control station
5 - slot_2 contacted
5 - slot_3 contacted
[nasadmin@ws1-whs-nascont1 tools]$ server_uptime ALL
server_2 : up 0 days 3 hours 39 min 28 secs
server_3 : up 0 days 3 hours 38 min 0 secs
[nasadmin@ws1-whs-nascont1 log]$ nas_logviewer /nas/log/sys_log |grep -i panic
Apr 26 23:21:54 2015:CS_PLATFORM:NaviEventMonitor:CRITICAL:4::::1430104914:CLARi
iON event number 0x907 Host SPA Storage Array APM00093001417 SPA Device SP A
SoftwareRev 6.29.0 (6.1) BaseRev 04.29.000.5.006 Description Microcode Panic.
iON event number 0x907 Host SPB Storage Array APM00093001417 SPB Device SP B
Apr 28 13:18:55 2015:CS_PLATFORM:BoxMonitor:CRITICAL:303::::1430241535:Slot 3 ha
s panicked.
s panicked.
s panicked - post panic logging complete: UNKNOWN PANIC DATA. Dump information c
an be found in /nas/log/post_panic_info_2015_04_28_13:25:21_EDT_slot_2.log.
iON event number 0x907 Host SPA Storage Array APM00093001417 SPA Device SP A
Apr 28 13:34:45 2015:CS_PLATFORM:LogCollect:INFO:311::::1430242485:automaticcoll

ection[20028]: Slot 2 does not have a panic dump for retrieval.
[nasadmin@ws1-whs-nascont1 dump]$ ls
total 141M
drwx------ 2 root root 16K Jul 28
drwxr-xr-x 2 root root 4.0K Jul 28
-rw-r--r-- 1 root bin 141M Nov 1
-lh
2009 lost+found
2009 mgmtd
2010 trcpdump.trc.gz
[nasadmin@ws1-whs-nascont1 dump]$ /nas/sbin/dump_slot -v -F -x

Partition 0 starts at 0x1, size 273042 (0x42a92) blocks.
DART Work Partition Layout found @ LBA 0x43000 (134MB boundary)
Checking LBA 0xf3800 for Full Dump...
Checking LBA 0x3f3805 for Full Dump...
Checking LBA 0x6f380a for Full Dump...
[nasadmin@ws1-whs-nascont1 dump]$ /nas/sbin/dump_slot -v -x
Partition 0 starts at 0x1, size 273042 (0x42a92) blocks.
DART Work Partition Layout found @ LBA 0x43000 (134MB boundary)
[nasadmin@ws1-whs-nascont1 dump]$ nas_server -i server_2
id
= 1
name
= server_2
acl
= 0
type
= nas
slot
= 2
member_of =
standby = server_3, policy=auto
status
:
defined = enabled
actual = online, active
[nasadmin@ws1-whs-nascont1 dump]$ server_log server_2 -s -a |grep -i logsys
2015-03-27 10:00:13: ADMIN: 6: Command succeeded: logsys add output disk=root_l
og_2 bufsz=256
2015-03-27 10:00:13: ADMIN: 6: Command succeeded: logsys add output event bufsz
=2048
2015-03-27 10:02:26: ADMIN: 6: Command succeeded: logsys delete output console
og_2 bufsz=256
=2048
og_2 bufsz=256
=2048
og_2 bufsz=256
=2048
2015-04-23 16:27:17:
og_2 bufsz=256
2015-04-23 16:27:17:
=2048
2015-04-23 16:28:43:
2015-04-24 20:15:00:
og_2 bufsz=256
2015-04-24 20:15:00:
=2048
2015-04-24 20:16:04:
2015-04-24 20:21:05:
og_2 bufsz=256
2015-04-24 20:21:05:
=2048
2015-04-24 20:21:38:
2015-04-24 20:24:04:
og_2 bufsz=256
2015-04-24 20:24:04:
=2048
2015-04-24 20:25:07:
2015-04-25 15:25:43:
og_2 bufsz=256
2015-04-25 15:25:43:
=2048
2015-04-25 15:26:34:
2015-04-27 10:25:25:
og_2 bufsz=256
2015-04-27 10:25:25:
=2048
2015-04-27 10:26:05:
2015-04-28 13:33:13:
og_2 bufsz=256
2015-04-28 13:33:13:
=2048
2015-04-28 13:34:15:
ADMIN: 6: Command succeeded: logsys add output disk=root_l

ADMIN: 6: Command succeeded: logsys add output event bufsz
ADMIN: 6: Command succeeded: logsys delete output console
[nasadmin@ws1-whs-nascont1 dump]$ server_log server_2 -s -a |grep -i 2015-04-28

2015-04-28 01:32:36: LGDB: 6: last message repeated 1 times
2015-04-28 02:57:33: LGDB: 6:[vdm_1] Restricted Groups enforced for ws1-whs-nash
ost1
2015-04-28 03:51:44: MGFS: 3: Add connection failed, status 8 - DuplicateEntry o
n file system id: 27.
2015-04-28 03:51:44: ADMIN: 3: Command failed: connect fsid=27 vendorId=1 type=
HTTP rpolicy=2 wpolicy=1 http=http://fmahostprimary.ad.wellstar.org/fmroot httpP
ort=8000 timeout=60 cgi=n account=rffm passwd=****************
ost1
2015-04-28 06:12:13: SMB: 6:[vdm_1] DCInvalidate WS1-WHS-DC01 from updateW2kInfo
2015-04-28 06:34:51: SMB: 6:[vdm_1] >DC=WS1-WHS-DC01(10.70.0.36) R=8 T=1 ms S=0,
0x1/-1
ost1
ost1
og_2 bufsz=256
=2048
2015-04-28 13:33:13: BOOT: 6: processor 1 booted and running.
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: ap boot
2015-04-28 13:33:16: LIB: 6: CELERRAID is set to APM000930014170000 and is 18 ch
aracters long.
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: setid celerraid=APM0009300141
70000
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: uniqueid validate=50060160:41
e10458
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: time timezone=America/New_Yor
k+5America/New_York+4,J073/02:00:00,J311/02:00:00
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: param kernel autoreboot=600
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: param config cs_ip=128.221.25
2.100:128.221.252.101:128.221.253.100:128.221.253.101
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: param config cs_external_ip=1
0.70.7.23
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: param cifs djAddAdminToLg=1
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: param cifs djEnforceDhn=0
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: param cifs djUseKpasswd=0
2015-04-28 13:33:16: KERNEL: 6: 666c8 bufs of size 2000 allocated starting at d4
c38000
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: bufcache
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: device isa isa-0
2015-04-28 13:33:16: KERNEL: 6: System Version-T5.6.47.11
2015-04-28 13:33:16: KERNEL: 6: System BIOS Rev Sledgehammer
2015-04-28 13:33:16: STORAGE: 3: find_bios32-> BIOS32 Service Directory found at
000fd460
2015-04-28 13:33:16: KERNEL: 3: pack: 0x00000210, signature: 0x20494350
2015-04-28 13:33:16: KERNEL: 6: PCI BIOS Rev 02.10
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: device pci pci-0
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: fcp serial APM00092901520
2015-04-28 13:33:16: ADMIN: 6: Command succeeded: dskdumpconfig full slot=2
2015-04-28 13:33:16: DRIVERS: 6: Cached memory - 0x730db bytes @ 0x4ec2004
2015-04-28 13:33:16: DRIVERS: 6: Noncached memory - 0x20b001 bytes @ 0x4f42004
(0x4f42004 phys)
2015-04-28 13:33:16: DRIVERS: 6: QX4
HBA 0 @ 0, irq 18, bus 9, func 0, de
v 0
2015-04-28 13:33:16: DRIVERS: 6: FCDMTL 0 [9.0.0] Find Adapter - OSW DB PTR 0x4e
be100
2015-04-28 13:33:16: DRIVERS: 4: 7: Fiber Port 0 SFP inserted: connector type op
tical
2015-04-28 13:33:16: DRIVERS: 6: optical2015-04-28 13:33:16: DRIVERS: 6: FCDMTL
0 [9.0.0] DVM Initialized
2015-04-28 13:33:16: DRIVERS: 6: FCDMTL 0 [9.0.0] IMQ base ptr = 04f80000; IMQ l
ength = 20000
2015-04-28 13:33:16: DRIVERS: 6: FCDMTL 0 [9.0.0] Signal Address = dfa000e0
2015-04-28 13:33:16: DRIVERS: 6: FCDMTL 0 [9.0.0] Driver Initialization
2015-04-28 13:33:16: DRIVERS: 6: FCDMTL 0 [9.0.0] TPM Notify: st=0xa000000, flg=
0x4, cmd=0x1
2015-04-28 13:33:16: DRIVERS: 6: FCDMTL 0 [9.0.0] Set PCI EXP DEV CTRL MRRS to 4
096 Bytes
2015-04-28 13:33:17: DRIVERS: 6: FCDMTL 0 [9.0.0] TPM Hndle API Event: cntx=0x4e
c1270, evnt=0x4002, info=0x0
2015-04-28 13:33:17: DRIVERS: 6: FCDMTL 0 [9.0.0] TPM Lnk Up: state=0xa000000, f
lg=0x84
2015-04-28 13:33:17: DRIVERS: 6: FCDMTL 0 [9.0.0] CALLBACK: LINK UP - connection
type 2
2015-04-28 13:33:17: DRIVERS: 6: FCDMTL 0 [9.0.0] DVM:PLOGI 5 id=0000ef
2015-04-28 13:33:17: DRIVERS: 6: FCDMTL 0 [9.0.0] DVM:PRLI 6 id=0000ef

2015-04-28 13:33:17: DRIVERS: 6: FCDMTL 0 [9.0.0] Found loop ID 0., via address
ID: context 0., pend/login 00000000
2015-04-28 13:33:21: FCP: 6: HBA 0: scan_for_live_port:01 fcLinkUp:512 lupTime:7
6 devTime:117
2015-04-28 13:33:21: FCP: 6: ONLINE
HBA 0: ALPA 000001
WWN: 5006016041e1
0458
2015-04-28 13:33:21: FCP: 6: scsi-0:
HBA 0: ALPA 0000ef SP-a00: 500601604460
09b5
2015-04-28 13:33:21: DRIVERS: 6: Cached memory - 0x730db bytes @ 0x535a004
2015-04-28 13:33:21: DRIVERS: 6: Noncached memory - 0x20b001 bytes @ 0x53da004
(0x53da004 phys)
2015-04-28 13:33:21: DRIVERS: 6: QX4
HBA 1 @ 0, irq 19, bus 9, func 1, de
v 0
2015-04-28 13:33:21: DRIVERS: 6: FCDMTL 1 [9.0.1] Find Adapter - OSW DB PTR 0x53
56100
tical
2015-04-28 13:33:21: DRIVERS: 6: FCDMTL 1 [9.0.1] IMQ base ptr = 05400000; IMQ l
ength = 20000
0x4, cmd=0x1
096 Bytes
2015-04-28 13:33:22: DRIVERS: 6: FCDMTL 1 [9.0.1] TPM Hndle API Event: cntx=0x53
59270, evnt=0x4002, info=0x0
2015-04-28 13:33:22: DRIVERS: 6: FCDMTL 1 [9.0.1] TPM Lnk Up: state=0xa000000, f
lg=0x84
2015-04-28 13:33:22: DRIVERS: 6: FCDMTL 1 [9.0.1] CALLBACK: LINK UP - connection
type 2
2015-04-28 13:33:22: DRIVERS: 6: FCDMTL 1 [9.0.1] DVM:PLOGI 5 id=0000ef
2015-04-28 13:33:22: DRIVERS: 6: FCDMTL 1 [9.0.1] DVM:PRLI 6 id=0000ef
2015-04-28 13:33:22: DRIVERS: 6: FCDMTL 1 [9.0.1] Found loop ID 0., via address
ID: context 0., pend/login 00000000
7 devTime:117
2015-04-28 13:33:26: FCP: 6: ONLINE
HBA 1: ALPA 000001
WWN: 5006016141e1
0458
2015-04-28 13:33:26: FCP: 6: scsi-16: HBA 1: ALPA 0000ef SP-b00: 500601684460
09b5
2015-04-28 13:33:26: DRIVERS: 6: Cached memory - 0x730db bytes @ 0x57ec004
2015-04-28 13:33:26: DRIVERS: 6: Noncached memory - 0x20b001 bytes @ 0x586c004
(0x586c004 phys)
2015-04-28 13:33:26: DRIVERS: 6: QX4
HBA 2 @ 0, irq 1a, bus 9, func 2, de
v 0
2015-04-28 13:33:26: DRIVERS: 6: FCDMTL 2 [9.0.2] Find Adapter - OSW DB PTR 0x57
e8100
tical
2015-04-28 13:33:26: DRIVERS: 6: FCDMTL 2 [9.0.2] IMQ base ptr = 058a0000; IMQ l
ength = 20000
0x4, cmd=0x1
096 Bytes
2015-04-28 13:33:30: DRIVERS: 6: FCDMTL 2 [9.0.2] TL status = 3c000006; inbound_
resource_status = 0
2015-04-28 13:33:30: DRIVERS: 6: FCDMTL 2 [9.0.2] FM config = 1010104; FM contro
l = 6; FM status = 330000e0
2015-04-28 13:33:30: DRIVERS: 6: FCDMTL 2 [9.0.2] PCI reset control = 0
2015-04-28 13:33:30: DRIVERS: 6: FCDMTL 2 [9.0.2] PCI int pend = 0; PCI int enab
= 7; PCI int status = 0
2015-04-28 13:33:30: DRIVERS: 6: FCDMTL 2 [9.0.2] PCI status reg 10, PCI X statu
s reg 0, DX2 Function status reg 140000
2015-04-28 13:33:30: DRIVERS: 6: FCDMTL 2 [9.0.2] TPM Hndle API Event: cntx=0x57
eb270, evnt=0x400e, info=0x0
2015-04-28 13:33:30: DRIVERS: 6: FCDMTL 2 [9.0.2] TPM Lup Failure: st=0xa000000,
flg=0x84, info=0x400e
2015-04-28 13:33:30: DRIVERS: 6: FCDMTL 2 [9.0.2] DVM DDB Shtdwn, ID: fffffe, Pr
i/Sec Err: 0, State: 0
2015-04-28 13:33:30: DRIVERS: 6: FCDMTL 2 [9.0.2] DVM DDB Shtdwn, ID: fffffd, Pr
2015-04-28 13:33:30: DRIVERS: 6: FCDMTL 2 [9.0.2] DVM DDB Shtdwn, ID: fffffc, Pr
2015-04-28 13:33:30: DRIVERS: 6: FCDMTL 2 [9.0.2] CALLBACK: type 44004, word0 05
7e8100, word1 10000001
2015-04-28 13:33:32: DRIVERS: 6: FCDMTL 0 [9.0.0] DVM Disc Comp- Dev List Size:
4
0x208, cmd=0x0
devTime:120
2015-04-28 13:33:36: FCP: 6: OFFLINE
HBA 2: ALPA 000001
WWN: 5006016241e1
0458
2015-04-28 13:33:36: FCP: 6: scsi-32: HBA 2: CHAINS 32 - 47 OFFLINE
2015-04-28 13:33:36: DRIVERS: 6: Cached memory - 0x730db bytes @ 0x5c7e004
2015-04-28 13:33:36: DRIVERS: 6: Noncached memory - 0x20b001 bytes @ 0x5cfe004
(0x5cfe004 phys)
2015-04-28 13:33:36: DRIVERS: 6: QX4
HBA 3 @ 0, irq 1b, bus 9, func 3, de
v 0
2015-04-28 13:33:36: DRIVERS: 6: FCDMTL 3 [9.0.3] Find Adapter - OSW DB PTR 0x5c
7a100
2015-04-28 13:33:36: DRIVERS: 4: 11: Fibre Port 3 SFP removed: status 0x0 state
0x0 type 0x0
2015-04-28 13:33:36: DRIVERS: 6: FCDMTL 3 [9.0.3] SFP removed : status 0x0 state
0x0 type 0x0
2015-04-28 13:33:36: DRIVERS: 6: FCDMTL 3 [9.0.3] SFP port disabled : state 6 ty
pe 0
2015-04-28 13:33:36: DRIVERS: 6: FCDMTL 3 [9.0.3] DVM Initialized
2015-04-28 13:33:36: DRIVERS: 6: FCDMTL 3 [9.0.3] IMQ base ptr = 05d20000; IMQ l
ength = 20000
0x4, cmd=0x1
096 Bytes
2015-04-28 13:33:37: DRIVERS: 6: FCDMTL 1 [9.0.1] DVM Disc Comp- Dev List Size:
4
0x208, cmd=0x0
2015-04-28 13:33:40: DRIVERS: 6: FCDMTL 3 [9.0.3] TL status = 3c000007; inbound_
resource_status = 0
2015-04-28 13:33:40: DRIVERS: 6: FCDMTL 3 [9.0.3] FM config = 1010104; FM contro
l = 0; FM status = 370000e0
2015-04-28 13:33:40: DRIVERS: 6: FCDMTL 3 [9.0.3] PCI reset control = 0
2015-04-28 13:33:40: DRIVERS: 6: FCDMTL 3 [9.0.3] PCI int pend = 0; PCI int enab
= 7; PCI int status = 0
2015-04-28 13:33:40: DRIVERS: 6: FCDMTL 3 [9.0.3] PCI status reg 10, PCI X statu
s reg 0, DX2 Function status reg 140000
2015-04-28 13:33:40: DRIVERS: 6: FCDMTL 3 [9.0.3] TPM Hndle API Event: cntx=0x5c
7d270, evnt=0x400e, info=0x0
2015-04-28 13:33:40: DRIVERS: 6: FCDMTL 3 [9.0.3] TPM Lup Failure: st=0xa000000,
flg=0x84, info=0x400e
2015-04-28 13:33:40: DRIVERS: 6: FCDMTL 3 [9.0.3] DVM DDB Shtdwn, ID: fffffe, Pr
2015-04-28 13:33:40: DRIVERS: 6: FCDMTL 3 [9.0.3] DVM DDB Shtdwn, ID: fffffd, Pr
2015-04-28 13:33:40: DRIVERS: 6: FCDMTL 3 [9.0.3] DVM DDB Shtdwn, ID: fffffc, Pr
2015-04-28 13:33:40: DRIVERS: 6: FCDMTL 3 [9.0.3] CALLBACK: type 44004, word0 05
c7a100, word1 10000001
devTime:120
2015-04-28 13:33:45: FCP: 6: OFFLINE
HBA 3: ALPA 000001
WWN: 5006016341e1
0458
2015-04-28 13:33:45: FCP: 6: scsi-48: HBA 3: CHAINS 48 - 63 OFFLINE
2015-04-28 13:33:45: DRIVERS: 6: BCM_AllocateMemory: mge0 - Using Low_malloc for
shared memory block allocation.
2015-04-28 13:33:45: DRIVERS: 6: mge0: irq=16
2015-04-28 13:33:45: DRIVERS: 6: mge0: Current Mac Address -> 0:60:16:35:8:85
2015-04-28 13:33:45: DRIVERS: 6: mge0: Link status - 100 Mbps full duplex
2015-04-28 13:33:45: DRIVERS: 4: 3: mge0: link up
2015-04-28 13:33:45: DRIVERS: 6: BCM_AllocateMemory: mge1 - Using Low_malloc for
2015-04-28 13:33:45: DRIVERS: 6: mge1: irq=18
2015-04-28 13:33:45: DRIVERS: 6: mge1: Current Mac Address -> 0:60:16:35:8:86
2015-04-28 13:33:45: DRIVERS: 6: mge1: Link status - 100 Mbps full duplex
2015-04-28 13:33:45: DRIVERS: 4: 3: mge1: link up
2015-04-28 13:33:45: DRIVERS: 6: BCM_AllocateMemory: cge0 - Using Low_malloc for
jumbo ring allocation.
2015-04-28 13:33:46: DRIVERS: 6: cge0: irq=22
2015-04-28 13:33:46: DRIVERS: 6: cge0: Current Mac Address -> 0:60:16:26:cb:c4
2015-04-28 13:33:46: DRIVERS: 6: cge1: irq=23
2015-04-28 13:33:46: DRIVERS: 6: cge2: irq=20
2015-04-28 13:33:46: DRIVERS: 6: cge3: irq=21

2015-04-28 13:33:46: CAM: 6: Scan All Buses
2015-04-28 13:33:46: ADMIN: 6: Command succeeded: pciautoconfig
2015-04-28 13:33:46: STORAGE: 6: Allow overlap on: Basic-Volume 0
2015-04-28 13:33:46: ADMIN: 6: Command succeeded: volume disk logdsk c0t0l0
2015-04-28 13:33:46: STORAGE: 6: Allow overlap on: logdsk
2015-04-28 13:33:46: ADMIN: 6: Command succeeded: volume disk logdsk c16t0l0
2015-04-28 13:33:46: ADMIN: 6: Command succeeded: volume slice root_log_2 81715
2 8191 logdsk
2015-04-28 13:33:46: ADMIN: 6: Command succeeded: el3config tier2-aui
2015-04-28 13:33:46: DRIVERS: 6: cge0: Shutting down Tigon3
2015-04-28 13:33:46: FCP: 3: 4: HBA 2: Link Down.
2015-04-28 13:33:47: ADMIN: 6: Command succeeded: sysconfig net name=cge0 optio
n="speed=auto duplex=auto txflowctl=disable rxflowctl=disable "
2015-04-28 13:33:48: FCP: 3: 4: HBA 3: Link Down.
2015-04-28 13:33:48: DRIVERS: 3: trk0: using mac 00:60:16:26:cb:c4
2015-04-28 13:33:49: DRIVERS: 6: cge1: New mac address 00:60:16:26:ffffffcb:ffff
ffc4 is now added.
2015-04-28 13:33:49: ADMIN: 6: Command succeeded: sysconfig virtual name=trk op
tion="create=trk0 device=cge0 device=cge1"
2015-04-28 13:33:49: DRIVERS: 3: trk1: using mac 00:60:16:26:cb:c2
ffc2 is now added.
2015-04-28 13:33:49: ADMIN: 6: Command succeeded: sysconfig virtual name=trk op
tion="create=trk1 device=cge2 device=cge3"
2015-04-28 13:33:49: DRIVERS: 6: cge0: Link status - 1000 Mbps full duplex
2015-04-28 13:33:49: DRIVERS: 4: 5: cge0: link up
2015-04-28 13:33:49: DRIVERS: 4: 3: trk0: link up
2015-04-28 13:33:49: DRIVERS: 6: cge2: Mac address 00:60:16:26:ffffffcb:ffffffc2
is already set.
2015-04-28 13:33:49: DRIVERS: 4: 4: cge0: link down
2015-04-28 13:33:49: DRIVERS: 4: 2: trk0: link down
ffc2 is now added.
ffc2 is now added.
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: sysconfig virtual name=fsn op
tion="create=fsn0 device=trk1 device=trk0"
2015-04-28 13:33:50: DRIVERS: 4: 3: fsn0: link up, trk1: active <- trk1: link up
2015-04-28 13:33:50: IP: 3: Created IPv4 table useHash 0
2015-04-28 13:33:50: IP: 3: Initializing the IPv6 route table
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: ifconfig el30 protocol=IP dev
ice=mge0 local=128.221.252.2 netmask=255.255.255.0 broadcast=128.221.252.255 mtu
=1500 vlan=0 mac=0:60:16:35:8:85
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: ifconfig el31 protocol=IP dev

ice=mge1 local=128.221.253.2 netmask=255.255.255.0 broadcast=128.221.253.255 mtu
=1500 vlan=0 mac=0:60:16:35:8:86
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: ifconfig 10-70-7-18 protocol=
IP device=fsn0 local=10.70.7.18 netmask=255.255.255.0 broadcast=10.70.7.255 mtu=
1500 vlan=0 mac=0:60:16:26:cb:c2
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: route add default 10.70.7.1 0
.0.0.0
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: hostname server_2 selfid=2
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: file initialize nodes=256000
dnlc=384000
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: volume disk 1 num_paths=2 c0t
0l0 c16t0l0 size=11263 disk_id=1 cel_id=APM000930014170000
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: volume slice 11 309248 32768
1
472 2
74
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: volume hyper 12 2 11 82
2015-04-28 13:33:50: STORAGE: 3: Basic Volume 98 not created, incorrect disk_id
14.
2015-04-28 13:33:50: ADMIN: 3: Command failed: volume disk 98 num_paths=2 c16t1
l5 c0t1l5 size=751133 disk_id=14 cel_id=APM000930014170000
2015-04-28 13:33:50: STORAGE: 3: Cannot create Hyper Volume 99, Volume 98 not fo
und.
2015-04-28 13:33:50: ADMIN: 3: Command failed: volume hyper 99 1 98
2015-04-28 13:33:50: STORAGE: 3: Cannot create Slice Volume 101, Volume 99 not f
ound.
2015-04-28 13:33:50: ADMIN: 3: Command failed: volume slice 101 0 262144 99
2015-04-28 13:33:50: STORAGE: 3: Cannot create Hyper Volume 102, Volume 101 not
found.
13.
2015-04-28 13:33:50: ADMIN: 3: Command failed: volume disk 97 num_paths=2 c16t1
l3 c0t1l3 size=751133 disk_id=13 cel_id=APM000930014170000
2015-04-28 13:33:50: STORAGE: 3: Cannot create Hyper Volume 103, Volume 97 not f
ound.
2015-04-28 13:33:50: STORAGE: 3: Cannot create Slice Volume 105, Volume 103 not
found.
3
12.
2015-04-28 13:33:50: ADMIN: 3: Command failed: volume disk 96 num_paths=2 c0t1l
4 c16t1l4 size=751133 disk_id=12 cel_id=APM000930014170000
ound.
found.
found.
2015-04-28 13:33:50: ADMIN: 3: Command failed: volume hyper 107 2 105 106
11.
2015-04-28 13:33:50: ADMIN: 3: Command failed: volume disk 95 num_paths=2 c0t1l
2 c16t1l2 size=751133 disk_id=11 cel_id=APM000930014170000
ound.
found.
8
ound.
2015-04-28 13:33:50: ADMIN: 3: Command failed: volume slice 110 262144 51277209
6 99
found.
found.
48288 104
ound.
86144 99
found.
1
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: volume hyper 40 1 39
2
472 2
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: volume slice 81 0 16384 74
2015-04-28 13:33:50: ADMIN: 6: Command succeeded: startPH volume=81
2015-04-28 13:33:50: STORAGE: 3: MEPanic::Recover() no Basic Volume PH buffer fo
und:err:0
2015-04-28 13:33:50: VRPL: 6: Bad Sector PH registerSelf panicHandler
2015-04-28 13:33:50: UFS: 3: log recovery: failed to locate volume '102'
2015-04-28 13:33:50: UFS: 6: Recovering filesystems from log 58
2015-04-28 13:33:51: UFS: 3: Super blocks of all file system in recovery have be
en read ahead of time.
2015-04-28 13:33:51: UFS: 6: Replay ufs log version 3
2015-04-28 13:33:51: UFS: 3: Super blocks file system not read next time.
2015-04-28 13:33:51: UFS: 6: successful recovery took 133656 Usec
2015-04-28 13:33:51: UFS: 6: processed records 392 - 399, 16 reads, 2 writes
2015-04-28 13:33:51: ADMIN: 6: Command succeeded: file recover ufs 58=131072 12
=2 102=25 107=27 111=28 114=29 40=16

2015-04-28 13:33:52: DRIVERS: 6: fsn0: link up, trk1: active <- trk0: link up
2015-04-28 13:34:10: ADMIN: 6: Command succeeded: log sectors=131072 58
2015-04-28 13:34:10: ADMIN: 6: Command succeeded: mount fastmount
2015-04-28 13:34:10: ADMIN: 6: Command succeeded: mount begin
2015-04-28 13:34:10: UFS: 6: Volume name:12
2015-04-28 13:34:10: UFS: 6: I18N is enabled for the system
2015-04-28 13:34:10: UFS: 6: starting gid map file processing for fs:2.
2015-04-28 13:34:10: UFS: 6: gid map file processing is completed for fsid:2 in
6 msec
2015-04-28 13:34:10: UFS: 6: initFS: initialize acl cache on fs 2
2015-04-28 13:34:10: UFS: 6: Filesystem 2 acl file size 49152
2015-04-28 13:34:10: UFS: 6: Filesystem 2 acl firstfree 6
2015-04-28 13:34:10: UFS: 6: end acl slot: 6
2015-04-28 13:34:10: UFS: 6: file system 2 acl initialization done: 5
2015-04-28 13:34:10: MGFS: 6: CSX initialilize succeeded
2015-04-28 13:34:10: DEDUPE: 6: PE: The Policy Engine has been started with 1 ta
sk thread(s) and 1 recall thread(s)
2015-04-28 13:34:11: DBMS: 6: Checkpoint thread starting
2015-04-28 13:34:11: DBMS: 6: Deadlock detection thread starting
2015-04-28 13:34:11: ADMIN: 6: Command succeeded: file mount uxfs rw / 12=2 rw
2015-04-28 13:34:11: ADMIN: 6: Command succeeded: file mount uxfs rw /root_vdm_
2/.etc 102=25 rw
2/fs01 107=27 rw
2015-04-28 13:34:11: DPSVC: 3: DpService::FindSavedMountOption() DpFilesystemDB
read failed, status: DpRequest_File_StaleHandle
2/fs02 111=28 rw
2/fs03 114=29 rw
2015-04-28 13:34:11: UFS: 3: file mount: failed to locate volume '111'
2015-04-28 13:34:11: UFS: 4: 48: The file system 28 mount failed during failover
because the file system was corrupt.
2015-04-28 13:34:11: UFS: 6: Volume name:40
2015-04-28 13:34:11: UFS: 6: starting gid map file processing for fs:16.
2015-04-28 13:34:11: UFS: 6: gid map file processing is completed for fsid:16 in
17 msec
2015-04-28 13:34:12: XLT: 6: line 25: (NULL)::10.70.7.16::: FMA requires no tra
nslation (UTF-8)
2015-04-28 13:34:12: XLT: 6: line 26: (NULL)::::8859-1.txt: Any thing that didn
't match above will be assumed to be latin-1
2015-04-28 13:34:12: ADMIN: 6: Command succeeded: file mount uxfs ro /.etc_comm
on 40=16 ro
2015-04-28 13:34:12: UFS: 4: One or more file systems failed to mount. Please se
e the sys_log for details.
2015-04-28 13:34:12: ADMIN: 6: Command succeeded: mount end
2015-04-28 13:34:12: ADMIN: 6: Command succeeded: export "/" anon=0 access=128.
221.252.100:128.221.253.100:128.221.252.101:128.221.253.101
2015-04-28 13:34:12: IP: 6: link ip loop
2015-04-28 13:34:12: IP: 6: route add host 127.0.0.1 127.0.0.1 loop
2015-04-28 13:34:12: IP: 6: link ip6 loop
2015-04-28 13:34:12: IP: 6: route add host ::1 ::1 loop
2015-04-28 13:34:12: ADMIN: 6: Command succeeded: transport start
2015-04-28 13:34:12: IP: 6: Transport timer thread starting ...
2015-04-28 13:34:12: ADMIN: 6: Command succeeded: udp checksum on
2015-04-28 13:34:12: TCP: 6: Initial ephemeral port = 61095
2015-04-28 13:34:12: ADMIN: 6: Command succeeded: tcp
2015-04-28 13:34:12: ADMIN: 6: Command succeeded: rpc
2015-04-28 13:34:12: RCPD: 6: Rcpd service started.
2015-04-28 13:34:12: ADMIN: 6: Command succeeded: rcp
2015-04-28 13:34:12: RCPD: 6: Starting RCP server on 8888
2015-04-28 13:34:12: STORAGE: 6: Allow overlap on: 1
2015-04-28 13:34:12: ADMIN: 6: Command succeeded: volume disk NBS1 c0t0l0
2015-04-28 13:34:12: STORAGE: 6: Allow overlap on: NBS1
2015-04-28 13:34:12: ADMIN: 6: Command succeeded: volume hyper SNBS1 1 NBS1
2015-04-28 13:34:12: STORAGE: 6: Addressing absolute location of first 8K on Dis
k: NBS1
2015-04-28 13:34:12: NBS: 6: reconstructing NBS database
2015-04-28 13:34:12: STORAGE: 3: Cannot create Hyper Volume SNBS5, Volume NBS5 n
ot found.
2015-04-28 13:34:12: ADMIN: 3: Command failed: volume hyper SNBS5 1 NBS5
2015-04-28 13:34:12: NBS: 3: BlkSrvConfig::AddID: new vol:volume hyper SNBS5 1 N
BS5 can not be created
2015-04-28 13:34:12: NBS: 3: nbsid: AddIDLocked failed extname=5 path=NBS5
2015-04-28 13:34:12: STORAGE: 3: Cannot create Hyper Volume SNBS6, Volume NBS6 n
ot found.
2015-04-28 13:34:12: ADMIN: 3: Command failed: volume hyper SNBS6 1 NBS6
2015-04-28 13:34:12: NBS: 3: BlkSrvConfig::AddID: new vol:volume hyper SNBS6 1 N
BS6 can not be created
2015-04-28 13:34:12: NBS: 3: nbsid: AddIDLocked failed extname=6 path=NBS6
2015-04-28 13:34:12: NBS: 4: invalid entry(6) in NBS database
2015-04-28 13:34:12: NBS: 4: invalid entry(5) in NBS database
2015-04-28 13:34:12: NBS: 6: NBS database reconstructed
2015-04-28 13:34:12: FTPD: 6: Starting ftpd service
2015-04-28 13:34:12: FTPD: 6: ftpd server started
2015-04-28 13:34:12: ADMIN: 6: Command succeeded: nbs add nbsid=1 vol=NBS1 rw=1
28.221.252.100:128.221.252.101:128.221.253.100:128.221.253.101 exclusive raw sha
re
k: NBS5
re

k: NBS6
re
2015-04-28 13:34:13: NBS: 6: reconstructing NBS database
2015-04-28 13:34:13: NBS: 6: NBS database reconstructed
2015-04-28 13:34:13: NBS: 6: BlkSrv: listening on port 5033
2015-04-28 13:34:13: ADMIN: 6: Command succeeded: nbs start
2015-04-28 13:34:13: ADMIN: 6: Command succeeded: communityname public
2015-04-28 13:34:13: ADMIN: 6: Command succeeded: syscontact nasadmin
2015-04-28 13:34:13: ADMIN: 6: Command succeeded: syslocation here
2015-04-28 13:34:13: ADMIN: 6: Command succeeded: timesync protocol=ntp action=
start host=10.70.0.36
2015-04-28 13:34:13: ADMIN: 6: Command succeeded: dns server=10.70.0.38 server=
10.70.0.37 udp domain=ad.wellstar.org
2015-04-28 13:34:13: ADMIN: 6: Command succeeded: routed
2015-04-28 13:34:13: TIMESYNC: 3: Connect to timeserver
2015-04-28 13:34:14: KERNEL: 6: Time set
2015-04-28 13:34:14: ADMIN: 6: Command succeeded: nfs start openfiles=240000 nf
sd=256
2015-04-28 13:34:14: ADMIN: 6: Command succeeded: statd
2015-04-28 13:34:14: ADMIN: 6: Command succeeded: lockd
2015-04-28 13:34:14: NDMP: 4: Thread config param PAX.readWriteBlockSizeInKB is
set to 64
2015-04-28 13:34:14: NDMP: 4: Thread config maximum number of buffers per restor
e job is 512
2015-04-28 13:34:14: NDMP: 4: Thread config number of buffers per restore job is
64
2015-04-28 13:34:14: NDMP: 4: Thread config maximum param PAX.nThread value is 1
28
2015-04-28 13:34:14: NDMP: 4: Thread config maximum param PAX.nRestore value is
16
2015-04-28 13:34:14: NDMP: 4: Thread config maximum param PAX.nPrefetch value is
8
2015-04-28 13:34:14: NDMP: 4: Thread config maximum param PAX.paxWriteBuff value
is 256
2015-04-28 13:34:14: ADMIN: 6: Command succeeded: pax
2015-04-28 13:34:14: ADMIN: 6: Command succeeded: rquotad action=start
2015-04-28 13:34:14: NDMP: 6: MaxBufferSize = 131072
2015-04-28 13:34:14: NDMP: 6: Listen thread started
2015-04-28 13:34:14: ADMIN: 6: Command succeeded: ndmp port=10000
2015-04-28 13:34:14: ADMIN: 6: Command succeeded: xattrp
2015-04-28 13:34:14: FTPD: 3: The FTP daemon is already started.
2015-04-28 13:34:14: ADMIN: 3: Command failed: ftpd
2015-04-28 13:34:14: ADMIN: 6: Command succeeded: sysman
2015-04-28 13:34:14: FSTOOLS: 6: System management component initializing...
2015-04-28 13:34:14: ADMIN: 6: Command succeeded: kerbinit
2015-04-28 13:34:14: ADMIN: 6: Command succeeded: cifs add wins=10.70.0.38
2015-04-28 13:34:14: SMB: 6: Threads number not specified and memory=4095 Mbytes
, use 256 threads
2015-04-28 13:34:14: FSTOOLS: 6: E003 - evd (V1.10) initialization complete
2015-04-28 13:34:14: FSTOOLS: 6: M004 - snmp_pe (V1.10) initialization complete
2015-04-28 13:34:14: NDMP: 3: Thread pax00 BlockBuffer count in PAX Pool: 268
2015-04-28 13:34:14: USRMAP: 6: Starting usermapper service
2015-04-28 13:34:14: USRMAP: 6: 2: Usermapper service has been enabled.
2015-04-28 13:34:15: SMB: 6: Registry initialization OK
2015-04-28 13:34:15: SMB: 6: Create SMB threads type 0
2015-04-28 13:34:15: SMB: 6: Create SMB threads type 2
2015-04-28 13:34:15: SMB: 6: Start SMB threads type 0

2015-04-28 13:34:15: SMB: 6: Start SMB threads type 2
2015-04-28 13:34:15: SMB: 6: Start VDM configuration
2015-04-28 13:34:15: SMB: 6: ShareDB: DART database started
2015-04-28 13:34:15: SMB: 3: No CIFS server found for IP=10.70.7.18
2015-04-28 13:34:15: SECMAP: 6: server_2 database started
2015-04-28 13:34:15: LGDB: 6: Database started
2015-04-28 13:34:15: SMB: 6: End VDM configuration
2015-04-28 13:34:15: ADMIN: 6: Command succeeded: cifs start
2015-04-28 13:34:15: ADMIN: 6: Command succeeded: vdm begin
2015-04-28 13:34:15: VDM: 3: The file system 25 is not mounted.
2015-04-28 13:34:15: ADMIN: 3: Command failed: vdm add id=2 fs=25 name=vdm_1
2015-04-28 13:34:15: DPSVC: 6: DpService::RecoveryReplicationVdmState() begin
2015-04-28 13:34:15: DPSVC: 6: DpService::RecoveryReplicationVdmState() end
2015-04-28 13:34:15: ADMIN: 6: Command succeeded: vdm end
2015-04-28 13:34:15: ADMIN: 6: Added the following 4 interface(s) to the list of
secure MAC clients: 128.221.252.100 128.221.252.101 128.221.253.100 128.221.253
.101
2015-04-28 13:34:15: ADMIN: 3: Indication using sequence starting: 0
2015-04-28 13:34:15: ADMIN: 6: Command succeeded: mac allow=128.221.252.100 128
.221.252.101 128.221.253.100 128.221.253.101
2015-04-28 13:34:15: CIC: 6: DIC_D SSL Client context initialized
2015-04-28 13:34:15: ADMIN: 6: Command succeeded: httpd secmac port=5082
2015-04-28 13:34:15: ADMIN: 6: Command succeeded: dpinit
2015-04-28 13:34:15: PERFSTATS: 6: The statmonService started on port: 7777
2015-04-28 13:34:15: ADMIN: 6: Command succeeded: statmonService start port=777
7 allow=128.221.252.100 128.221.252.101 128.221.253.100 128.221.253.101
2015-04-28 13:34:15: DPSVC: 6: BDatabase:: InitDatabase
2015-04-28 13:34:15: DPSVC: 6: BDatabase::initialize getEnvironment
2015-04-28 13:34:15: DPSVC: 6: BDatabase::initialize Register
2015-04-28 13:34:15: BOOT: 3: No processors to boot
2015-04-28 13:34:15: DPSVC: 6: BDatabase::initialize getDatabase
2015-04-28 13:34:15: DPSVC: 6: DpThrot::queryRemSystem 'APM000930014170000' not
found
2015-04-28 13:34:15: DPSVC: 3: loopback:20001:APM000930014170000:1:1:127.0.0.1:1
27.0.0.1
2015-04-28 13:34:15: DPSVC: 3: ControlStation:20002:APM000930014170000:2:4:128.2
21.252.2:128.221.252.100
2015-04-28 13:34:15: DPSVC: 6: DpInit::versionInit() Recover Done
2015-04-28 13:34:15: DPSVC: 6: DpInit::policyInit() Recover Done
2015-04-28 13:34:15: DPSVC: 6: DpInit::replicaInit() Recover Done
2015-04-28 13:34:15: DPSVC: 6: DpInit::sessionInit() Recover Done
2015-04-28 13:34:15: DPSVC: 6: DpInit::snapSchedulerInit() Done
2015-04-28 13:34:15: DPSVC: 6: DpInit::taskManagerInit() Done
2015-04-28 13:34:15: CMD: 3: DicXmlAsyncService::initDicXmlShortMsgService()
2015-04-28 13:34:15: CMD: 3: DicXmlAsyncService::initDicXmlLongMsgService()
2015-04-28 13:34:15: CMD: 3: DicXmlAsyncService::initDicXmlDefMsgService()
2015-04-28 13:34:15: CMD: 3: DicXmlAsyncService::initDicXmlDartMsgService()
2015-04-28 13:34:15: CMD: 6: Cmd timer thread starting ...
2015-04-28 13:34:15: CMD: 6: Initializing Indication Threads
2015-04-28 13:34:15: CMD: 6: CmdRecover:: Initdatabase Starting
2015-04-28 13:34:15: CMD: 6: CmdDatabase:: InitDatabase
2015-04-28 13:34:15: CMD: 6: CmdDatabase::initialize getEnvironment
2015-04-28 13:34:15: CMD: 6: CmdDatabase::initialize Register
2015-04-28 13:34:16: DPSVC: 6: BDatabase:: InitDatabase
2015-04-28 13:34:16: DPSVC: 6: BDatabase::initialize getEnvironment
2015-04-28 13:34:16: DPSVC: 6: BDatabase::initialize Register
2015-04-28 13:34:16: DPSVC: 6: BDatabase::initialize getDatabase
2015-04-28 13:34:16: CMD: 6: CMD Context: Recovering the Cmd Contexts :
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
lished
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
13:34:16:
13:34:16:
13:34:16:
13:34:16:
13:34:16:
13:34:18:
13:34:18:
CMD: 6:
Task_Id
RefCount
CMD: 3: TaskRecovery::destroy() Object not instantiated.
CMD: 6: CmdRecover:Recovery Completed.
DPSVC: 3: DpInit:Recovery Completed.
KERNEL: 6: cpu stat started
SMB: 3: No CIFS server found for IP=10.70.7.18
SYSTEM: 6: Setting SIB attention status to 5, contact estab
13:34:18:
13:34:21:
13:36:06:
13:36:06:
13:36:06:
13:36:06:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:53:
13:36:54:
13:36:54:
13:36:57:
13:42:00:
13:42:02:
13:47:05:
13:47:08:
13:52:11:
13:52:13:
13:57:16:
13:57:18:
14:02:21:
ADMIN: 6: Command succeeded: sib contacted

SMB: 3: last message repeated 34 times
NFS: 6: Processed options:
NFS: 6: Export / @ 20792804, rootuid 0
CAM: 6: Scan All Buses
CAM: 3: 2: Warning: The SCSI HBA 32 has failed.
ADMIN: 6: xml_dns domaincount :1
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
14:02:24:
14:07:27:
14:07:29:
14:12:32:
14:12:34:
14:17:37:
14:17:39:
14:22:42:
14:22:44:
14:27:47:
14:27:49:
14:32:52:
14:32:55:
14:38:07:
14:38:09:
14:43:12:
14:43:15:
14:48:18:
14:48:20:
14:53:23:
14:53:25:
14:58:28:
14:58:31:
15:03:34:
15:03:36:
15:08:39:
15:08:42:
15:13:45:
15:13:47:
15:18:50:
15:18:52:
15:23:55:
15:23:57:
15:29:00:
15:29:02:
15:34:05:
15:34:08:
15:39:20:
15:39:22:
15:44:25:
15:44:28:
15:49:31:
15:49:33:
15:54:36:
15:54:38:
15:59:41:
15:59:43:
16:04:46:
16:04:48:
16:09:51:
16:09:54:
16:14:57:
16:14:59:
16:20:02:
16:20:05:
16:25:08:
16:25:10:
16:30:13:
16:30:15:
16:35:23:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
SMB:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
3:
No CIFS server found for IP=10.70.7.18

last message repeated 101 times
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
2015-04-28
16:35:25:
16:36:21:
16:36:21:
16:36:22:
16:36:24:
16:41:27:
16:41:29:
16:46:32:
16:46:34:
16:51:37:
16:51:40:
16:56:43:
16:56:45:
17:01:48:
17:01:51:
17:06:54:
17:06:56:
17:11:59:
17:12:01:

CAM: 6: Scan All Buses
ADMIN: 6: xml_dns domaincount :1
[nasadmin@ws1-whs-nascont1 dump]$ server_mount ALL

server_2 :
root_fs_2 on / uxfs,perm,rw
root_fs_vdm_vdm_1 on /root_vdm_2/.etc uxfs,perm,rw,<unmounted>
fs01 on /root_vdm_2/fs01 uxfs,perm,rw,<unmounted>
root_fs_common on /.etc_common uxfs,perm,ro
server_3 :
root_fs_3 on / uxfs,perm,rw,<unmounted>
root_fs_common on /.etc_common uxfs,perm,ro,<unmounted>
vdm_1 :
Error 4032: vdm_1 : VDM is not loaded
[nasadmin@ws1-whs-nascont1 ~]$ .server_config ALL -v "fcp bind show"
server_2 : commands processed: 1
command(s) succeeded
output is complete
*** Persistent Binding Table ***
Chain 0000: WWN 50060160446009b5 HBA 0 SP-a00 Bound
Chain 0016: WWN 50060168446009b5 HBA 1 SP-b00 Bound
Chain 0032: WWN 5006048000000000 HBA 2 N_PORT Bind Pending
Existing CRC: 9865a9a7, Actual: 9865a9a7, CRC Matchs
*** Dynamic Binding Table ***
Chain 0000: WWN 50060160446009b5 HBA 0 ID 0 Inx 00:00 Pid 0000
Chain 0032: WWN 0000000000000000 HBA 2 ID 2 Inx 02:81 Pid 0032
FCP Base Chain: 0 Dump Slot: 2 Dump Chain: 0 16
Adapter Chain Offset 0:0 1:16 2:32 3:48 dumpInit 1
1430251235: ADMIN: 6: Command succeeded: fcp bind show
server_3 : commands processed: 1
command(s) succeeded
output is complete
D_ID
D_ID
D_ID
D_ID
0000ef
0000ef
000000
000000
Sys
Non
Non
Non
*** Persistent Binding Table ***

Chain 0000: WWN 50060161446009b5 HBA 0 SP-a01 Bound
Chain 0016: WWN 50060169446009b5 HBA 1 SP-b01 Bound
Existing CRC: 0eb8af06, Actual: 0eb8af06, CRC Matchs
*** Dynamic Binding Table ***
FCP Base Chain: 0 Dump Slot: 3 Dump Chain: 0 16
Adapter Chain Offset 0:0 1:16 2:32 3:48 dumpInit 1
1430263900: ADMIN: 6: Command succeeded: fcp bind show
D_ID
D_ID
D_ID
D_ID
vdm_1 :
Error 4032: vdm_1 : VDM is not loaded
[nasadmin@ws1-whs-nascont1 ~]$ /nas/sbin/enclosure_status -e 0 -v
DEVICE A
DEVICE B
-------------------PRESENCE-------------------00 60 16 31 A4 35
MAC
00 60 16 31 A6 2C
Scorpion
Hardware Platform
Scorpion
Present
Peer Compute Blade Present
Present
Compute Blade
Present
Powered On Compute Blade
Powered On
Present
Power Supply A
Present
Present
Power Supply B
Present
Inactive
Manufacturing Mode Inactive
Inactive
Margin High Mode
Inactive
Inactive
Margin Low Mode
Inactive
Inactive
MP I2C Bus in Reset Inactive
---------------BROADCOM STATUS----------------Active
Dnlink/P4 FullDuplex Active
Active
Dnlink/P4 100 MBPS Active
Active
Dnlink/P4 Up
Active
Active
PeerSP/P3 FullDuplex Active
Active
PeerSP/P3 100 MBPS Active
Active
PeerSP/P3 Up
Active
Active
Uplink/P1 FullDuplex Active
Active
Uplink/P1 100 MBPS Active
Active
Uplink/P1 Up
Active
Active
SP
/P0 FullDuplex Active
Active
SP
/P0 100 MBPS Active
Active
SP
/P0 Up
Active
---------------ENCLOSURE ALARMS---------------Pass
PSA OK
Pass
Pass
PSA Overtemp
Pass
Pass
PSA AC OK
Pass
Pass
PSB OK
Pass
Pass
PSB Overtemp
Pass
Pass
PSB AC OK
Pass
----------------BROADCOM ALARMS---------------Pass
5325 Comm Fail
Pass
Pass
5325 MIB RAM
Pass
Pass
5325 MIN MEM
Pass
Pass
5325 BUFF Con
Pass
--------------RESUME CSUM ERRORS--------------Pass
PSA Csum Error
Pass
Pass
PSB Csum Error
Pass
0000ef
0000ef
000000
000000
Sys
Non
Non
Non
Pass
CB x Csum Error
Pass
Pass
Midplane Csum Error Pass
----------------COLDFIRE ALARMS---------------Pass
P On Self Flash Fail Pass
Pass
P On Self RAM Fail Pass
Pass
P On Self FEC Fail Pass
Pass
P On Self 5325 Fail Pass
Pass
Run Self Flash Fail Pass
Pass
Run Self RAM Fail
Pass
Pass
Diag 5325 Fail
Pass
Pass
Encl Resume Block
Pass
Pass
Peer RS232 traffic Pass
Pass
MidPlane ID Read
Pass
Pass
I2C Arbit Error
Pass
Pass
I2C MP Bus Error
Pass
-------------STATUS CONDITIONS----------------Valid
Peer Coldfire Status Valid
Valid
Fault Expander
Valid
Valid
Data Mover Status
Valid
Valid
PSB Status
Valid
Valid
PSA Status
Valid
------------------FRU STATUS------------------Pass
FRU CPU DIMM 0
Pass
Pass
FRU CPU DIMM 1
Pass
Pass
FRU CPU DIMM 2
Pass
Pass
FRU CPU DIMM 3
Pass
Pass
FRU CPU Module
Pass
Pass
FRU CPU IO Module
Pass
Pass
NAS Personality Card Pass
Pass
FRU Enclosure
Pass
Pass
FRU Coldfire
Pass
Pass
FRU Power Supply B Pass
Pass
FRU Power Supply A Pass
---------------SYSTEM VARIABLES---------------02
Slot ID
03
00
Enclosure ID
00
00
BackPlane ID
01
61
Post Code
61
05
Reason Code
05
2C
Blade Status Code
2C
80
Post Middle 8 bits 80
00
I2C Error Mask
00
00
I2C MP Error Mask
00
[nasadmin@ws1-whs-nascont1 ~]$ /nas/sbin/navicli -h spb getagent
Agent Rev:
Name:
Desc:
Node:
Physical Node:
Signature:
Peer Signature:
Revision:
SCSI Id:
Model:
Model Type:
Prom Rev:
SP Memory:
Serial No:
6.29.5 (0.55)
K10
B-APM00093001417
K10
2285660
2183213
04.29.000.5.006
0
CX4-480
Rackmount
4.80.00
8160
APM00093001417
SP Identifier:
Cabinet:
B
SPE5
[nasadmin@ws1-whs-nascont1 ~]$ /nas/sbin/navicli -h spb getcache

SP Read Cache State
Enabled
SP Write Cache State
Disabled
Cache Page size:
8
Write Cache Mirrored:
YES
Low Watermark:
70
High Watermark:
90
SPA Cache pages:
247039
SPB Cache pages:
247040
Unassigned Cache Pages:
0
Read Hit Ratio:
N/A
Write Hit Ratio:
N/A
Prct Dirty Cache Pages =
0
Prct Cache Pages Owned =
50
SPA Read Cache State
Enabled
SPB Read Cache State
Enabled
SPA Write Cache State
Disabled
SPB Write Cache State
Disabled
System Buffer (spA):
3662 MB
System Buffer (spB):
3662 MB
SPS Test Day:
Saturday
SPS Test Time:
21:00
SPA Physical Memory Size =
8160
SPB Physical Memory Size =
8160
Physical memory size of Front-End = Switch not supported
Physical memory size of Back-End = Switch not supported
SPA Free Memory Size =
0
SPB Free Memory Size =
0
Free Memory Size of Front-End =
Switch not supported
Free Memory Size of Back-End =
Switch not supported
SPA Read Cache Size =
512
SPB Read Cache Size =
512
SPA Write Cache Size =
3986
SPB Write Cache Size =
3986
SPA Optimized Raid 3 Memory Size = 0
SPB Optimized Raid 3 Memory Size = 0
[nasadmin@ws1-whs-nascont1 ~]$ /nas/sbin/navicli -h spb getcrus
SPE5 Enclosure SPE
*FAULT*
(Enclosure SPE : Faulted; Enclosure SPE SPS A : Faulted; Enclosure SPE SPS B :
Faulted)
SP A State:
Present
SP B State:
Present
Enclosure SPE Power A0 State: Present
Enclosure SPE Power B0 State: Present
Enclosure SPE SPS A State: Present
Enclosure SPE SPS B State: Present
Enclosure SPE SPS A Cabling State: Invalid Multiple Cable
Enclosure SPE SPS B Cabling State: Invalid Power Cable 2
DAE3P Bus 0 Enclosure 0
*FAULT*
(Bus 0 Enclosure 0 : Failed - Fault reported by SP A : Enclosure reported a g
eneral fault ; Fault reported by SP B : Enclosure reported a general fault ; )
Bus 0 Enclosure 0 Fan A State: Present
Bus
Bus
Bus
Bus
Bus
Bus
Bus
Bus
Bus
0
0
0
0
0
0
0
0
0
Enclosure
Enclosure
Enclosure
Enclosure
Enclosure
Enclosure
Enclosure
Enclosure
Enclosure
0
0
0
0
0
0
0
0
0
Fan B
Power
Power
LCC A
LCC B
LCC A
LCC B
LCC A
LCC B
State: Present
A State: Present
B State: Faulted
State: Present
State: Present
Revision: 255.255
Revision: 255.255
Serial #: N/A
Serial #: N/A
UNKNOWN Bus 0 Enclosure 1

*FAULT*
(Bus 0 Enclosure 1 : Failed - Fault reported by SP A : Faulted enclosure befo
re this in the chain ; Fault reported by SP B : Faulted enclosure before this i
n the chain ; )
[nasadmin@ws1-whs-nascont1 ~]$ nas_storage -f id=1
id
= 1
serial_number
= APM00093001417
name
= APM00093001417
acl
= 0
done
[nasadmin@ws1-whs-nascont1 ~]$ nas_storage -c -a
Discovering storage (may take several minutes)
Error 5017: storage health check failed
APM00093001417 write cache disabled
APM00093001417 SPA is faulted/removed
APM00093001417 SPB is faulted/removed
APM00093001417 root_disk is faulted/removed
APM00093001417 root_ldisk is faulted/removed
APM00093001417 d3 is faulted/removed
APM00093001417 d11, RAID5(8+1), doesn't match any storage profile
[nasadmin@ws1-whs-nascont1 ~]$ server_mount server_2 |grep -i unmount
[nasadmin@ws1-whs-nascont1 ~]$ /nas/sbin/navicli -h spa faults -list
Faulted Subsystem: APM00093001417
Enclosure SPE : Faulted
Enclosure SPE SPS A : Faulted
Enclosure SPE SPS B : Faulted
Bus 0 Enclosure 0 : Failed - Fault
ral fault ; Fault reported by SP B
this in the chain ; Fault reported
reported by SP A : Enclosure reported a gene

: Enclosure reported a general fault ;
reported by SP A : Faulted enclosure before
by SP B : Faulted enclosure before this in t
he chain ;
Bus 0 Enclosure
Bus 0 Enclosure
Bus 0 Enclosure
Bus 0 Enclosure
Bus 0 Enclosure
Bus 0 Enclosure
Bus 0 Enclosure
Bus 0 Enclosure
Bus 0 Enclosure
Bus 0 Enclosure
0
0
0
0
0
0
0
0
0
0
Disk
Disk
Disk
Disk
Disk
Disk
Disk
Disk
Disk
Disk
4 : Removed
6 : Removed
7 : Removed
8 : Removed
9 : Removed
10 : Removed
11 : Removed
12 : Removed
13 : Removed
14 : Removed
[nasadmin@ws1-whs-nascont1 ~]$ /nas/sbin/navicli -h spb faults -list

Faulted Subsystem: APM00093001417
Enclosure SPE : Faulted
Enclosure SPE SPS A : Faulted
Enclosure SPE SPS B : Faulted
ral fault ; Fault reported by SP B
this in the chain ; Fault reported
he chain ;
Bus 0 Enclosure 0 Disk 4 : Removed
reported by SP A : Enclosure reported a gene

: Enclosure reported a general fault ;
reported by SP A : Faulted enclosure before
by SP B : Faulted enclosure before this in t
[nasadmin@ws1-whs-nascont1 ~]$
[nasadmin@ws1-whs-nascont1 ~]$ server_mount server_2 |grep -i unmount
[nasadmin@ws1-whs-nascont1 ~]$ /nas/sbin/navicli -h spa getcrus
SPE5 Enclosure SPE
*FAULT*
(Enclosure SPE : Faulted; Enclosure SPE SPS A : Faulted; Enclosure SPE SPS B :
Faulted)
SP A State:
Present
SP B State:
Present
Enclosure SPE SPS A State: Present
Enclosure SPE SPS B State: Present
Enclosure SPE SPS A Cabling State: Invalid Multiple Cable
Enclosure SPE SPS B Cabling State: Invalid Power Cable 2
DAE3P Bus 0 Enclosure 0
*FAULT*
(Bus 0 Enclosure 0 : Failed - Fault reported by SP A : Enclosure reported a g
eneral fault ; Fault reported by SP B : Enclosure reported a general fault ; )
Bus 0 Enclosure 0 Fan A State: Present
Bus 0 Enclosure 0 Fan B State: Present
Bus
Bus
Bus
Bus
Bus
Bus
Bus
Bus
0
0
0
0
0
0
0
0
Enclosure
Enclosure
Enclosure
Enclosure
Enclosure
Enclosure
Enclosure
Enclosure
0
0
0
0
0
0
0
0
Power
Power
LCC A
LCC B
LCC A
LCC B
LCC A
LCC B
A State: Present
B State: Present
State: Present
State: Present
Revision: 255.255
Revision: 255.255
Serial #: N/A
Serial #: N/A
UNKNOWN Bus 0 Enclosure 1

*FAULT*
(Bus 0 Enclosure 1 : Failed - Fault reported by SP A : Faulted enclosure befo
re this in the chain ; Fault reported by SP B : Faulted enclosure before this i
n the chain ; )`

CLARiON Panic

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

CLARiON Panic

Caricato da

Copyright:

Formati disponibili

>

Both the DMs panicked and hence no failover

*** slot_0 primary control station ***

Apr 28 13:34:45 2015:CS_PLATFORM:LogCollect:INFO:311::::1430242485:automaticcoll

[nasadmin@ws1-whs-nascont1 dump]$ /nas/sbin/dump_slot -v -F -x

ADMIN: 6: Command succeeded: logsys add output disk=root_l

[nasadmin@ws1-whs-nascont1 dump]$ server_log server_2 -s -a |grep -i 2015-04-28

2015-04-28 13:33:17: DRIVERS: 6: FCDMTL 0 [9.0.0] DVM:PRLI 6 id=0000ef

2015-04-28 13:33:46: DRIVERS: 6: cge3: Current Mac Address -> 0:60:16:26:cb:c3

2015-04-28 13:33:50: ADMIN: 6: Command succeeded: ifconfig el31 protocol=IP dev

2015-04-28 13:33:52: DRIVERS: 4: 5: cge0: link up

2015-04-28 13:34:12: ADMIN: 6: Command succeeded: volume hyper SNBS6 1 NBS6

2015-04-28 13:34:15: SMB: 6: Start SMB threads type 0

ADMIN: 6: Command succeeded: sib contacted

No CIFS server found for IP=10.70.7.18

SMB: 3: No CIFS server found for IP=10.70.7.18

[nasadmin@ws1-whs-nascont1 dump]$ server_mount ALL

*** Persistent Binding Table ***

[nasadmin@ws1-whs-nascont1 ~]$ /nas/sbin/navicli -h spb getcache

UNKNOWN Bus 0 Enclosure 1

reported by SP A : Enclosure reported a gene

[nasadmin@ws1-whs-nascont1 ~]$ /nas/sbin/navicli -h spb faults -list

reported by SP A : Enclosure reported a gene

UNKNOWN Bus 0 Enclosure 1

Potrebbero piacerti anche

* slot_0 primary control station *

* Persistent Binding Table *