The accompanying log indicates the problem I have since upgraded to 6.0 esxi
Can someone help me out what to do about it
2015-06-25T10:00:32.712Z cpu4:32795)WARNING: ScsiDeviceIO: 1243: Device naa.60014050099f4f1d930ad3999d85f1d5 performance has deteriorated. I/O latency increased from average value of 12579 microseconds to 253893 microseconds.
2015-06-25T10:00:34.048Z cpu4:32795)WARNING: ScsiDeviceIO: 1243: Device naa.60014050099f4f1d930ad3999d85f1d5 performance has deteriorated. I/O latency increased from average value of 12586 microseconds to 517602 microseconds.
2015-06-25T10:00:42.790Z cpu5:32796)WARNING: ScsiDeviceIO: 1243: Device naa.60014050099f4f1d930ad3999d85f1d5 performance has deteriorated. I/O latency increased from average value of 12601 microseconds to 1040631 microseconds.
2015-06-25T10:00:43.991Z cpu4:32795)WARNING: ScsiDeviceIO: 1243: Device naa.60014050099f4f1d930ad3999d85f1d5 performance has deteriorated. I/O latency increased from average value of 12606 microseconds to 2285662 microseconds.
2015-06-25T10:00:48.037Z cpu5:32796)WARNING: ScsiDeviceIO: 1243: Device naa.60014050099f4f1d930ad3999d85f1d5 performance has deteriorated. I/O latency increased from average value of 12702 microseconds to 944142 microseconds.
2015-06-25T10:00:48.176Z cpu5:32796)WARNING: ScsiDeviceIO: 1243: Device naa.60014050099f4f1d930ad3999d85f1d5 performance has deteriorated. I/O latency increased from average value of 12707 microseconds to 1944411 microseconds.
2015-06-25T10:00:50.866Z cpu6:32797)WARNING: ScsiDeviceIO: 1243: Device naa.60014050099f4f1d930ad3999d85f1d5 performance has deteriorated. I/O latency increased from average value of 12728 microseconds to 858876 microseconds.
2015-06-25T10:00:50.930Z cpu6:32797)WARNING: ScsiDeviceIO: 1243: Device naa.60014050099f4f1d930ad3999d85f1d5 performance has deteriorated. I/O latency increased from average value of 12728 microseconds to 1800284 microseconds.
2015-06-25T10:02:07.795Z cpu4:32795)WARNING: ScsiDeviceIO: 1243: Device t10.ATA_____WDC_WD1001FALS2D00J7B1________________________WD2DWMATV2213307 performance has deteriorated. I/O latency increased from average value of 11645 microseconds to 35$
2015-06-25T16:34:20.399Z cpu0:87682)WARNING: Heartbeat: 493: One or more PCPUs didn't perform a heartbeat check for 93 seconds.
2015-06-25T16:34:30.404Z cpu5:33251)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:1 L:0 : Task mgmt "Abort Task" with itt=0x16531d (refITT=0x16531c) timed out.
2015-06-25T16:34:41.107Z cpu5:33251)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:0 T:1 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2015-06-25T16:34:41.107Z cpu5:33251)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2000-01.com.synology:ds214plus.target-1.8b0573ad70 TPGT: 0 TSIH: 0]
2015-06-25T16:34:41.107Z cpu5:33251)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.2.150:54195 R: 192.168.1.214:3260]
2015-06-25T16:34:41.107Z cpu5:33251)WARNING: iscsi_vmk: iscsivmk_TaskMgmtIssue: vmhba37:CH:0 T:1 L:0 : Task mgmt "Abort Task" with itt=0x16531f (refITT=0x16531c) timed out.
2015-06-25T16:34:41.107Z cpu0:32810)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-06-25T16:34:41.866Z cpu0:32786)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-06-25T16:34:42.831Z cpu0:35481)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-06-25T16:34:43.862Z cpu0:35471)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-06-25T16:34:44.124Z cpu5:33251)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba37:CH:0 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4)
2015-06-25T16:34:44.124Z cpu5:33251)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2000-01.com.synology:ds212.esxi TPGT: 0 TSIH: 0]
2015-06-25T16:34:44.124Z cpu5:33251)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.2.150:10632 R: 192.168.2.201:3260]
2015-06-25T16:34:44.838Z cpu0:32836)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-06-25T16:34:45.830Z cpu0:35481)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-06-25T16:34:46.872Z cpu0:35859)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-06-25T16:34:47.866Z cpu0:35479)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-06-25T16:34:48.837Z cpu0:37018)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-06-25T16:34:49.866Z cpu0:35471)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-06-25T16:34:50.842Z cpu0:32785)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.60014050099f4f1d930ad3999d85f1d5" state in doubt; requested fast path state update...
2015-06-25T16:34:51.129Z cpu0:35479)WARNING: NMP: nmp_IssueCommandToDevice:4431: I/O could not be issued to device "naa.60014050099f4f1d930ad3999d85f1d5" due to Not found
2015-06-25T16:34:51.129Z cpu0:35479)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.60014050099f4f1d930ad3999d85f1d5": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2015-06-25T16:34:51.129Z cpu0:35479)WARNING: NMP: nmp_DeviceStartLoop:725: NMP Device "naa.60014050099f4f1d930ad3999d85f1d5" is blocked. Not starting I/O from device.
2015-06-25T16:34:51.826Z cpu3:33162)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - issuing command 0x439d806a1840
2015-06-25T16:34:51.826Z cpu3:33162)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:34:51.826Z cpu3:33162)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014050099f4f1d930ad3999d85f1d5": awaiting fast path state update...
2015-06-25T16:34:52.826Z cpu3:33162)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - issuing command 0x439d806a1840
2015-06-25T16:34:52.826Z cpu3:33162)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:34:52.826Z cpu3:33162)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014050099f4f1d930ad3999d85f1d5": awaiting fast path state update...
2015-06-25T16:35:21.108Z cpu2:32995)WARNING: NMP: nmp_DeviceStartLoop:725: NMP Device "naa.60014050099f4f1d930ad3999d85f1d5" is blocked. Not starting I/O from device.
2015-06-25T16:35:21.108Z cpu0:32810)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: Timeout:
2015-06-25T16:35:21.108Z cpu0:32810)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:21.108Z cpu7:32811)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:21.108Z cpu7:32811)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:21.826Z cpu3:87762)WARNING: NMP: nmpDeviceAttemptFailover:566: Retry world restore device "naa.60014050099f4f1d930ad3999d85f1d5" - no more commands to retry
2015-06-25T16:35:21.840Z cpu2:32809)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:21.840Z cpu2:32809)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:24.888Z cpu7:32810)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:24.888Z cpu7:32810)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:27.835Z cpu0:32811)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:27.835Z cpu0:32811)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:30.835Z cpu0:32809)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:30.835Z cpu0:32809)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:33.835Z cpu0:32810)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:33.835Z cpu0:32810)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:36.835Z cpu2:32811)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:36.835Z cpu2:32811)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:39.835Z cpu0:32809)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:39.835Z cpu0:32809)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:42.835Z cpu0:32810)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:42.835Z cpu0:32810)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:45.835Z cpu2:32811)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:45.835Z cpu2:32811)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:48.835Z cpu0:32809)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:48.835Z cpu0:32809)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:51.835Z cpu0:32810)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:51.835Z cpu0:32810)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:54.835Z cpu2:32811)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:54.835Z cpu2:32811)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:35:57.835Z cpu0:32809)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:35:57.835Z cpu0:32809)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:36:00.835Z cpu0:32810)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:36:00.835Z cpu0:32810)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:36:03.835Z cpu2:32811)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:36:03.835Z cpu2:32811)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:36:06.835Z cpu0:32809)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:36:06.835Z cpu0:32809)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:36:09.835Z cpu0:32810)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:36:09.835Z cpu0:32810)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:36:12.835Z cpu0:32811)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:36:12.835Z cpu0:32811)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:36:32.212Z cpu4:34580)WARNING: NMP: nmp_IssueCommandToDevice:4431: I/O could not be issued to device "naa.60014050099f4f1d930ad3999d85f1d5" due to Not found
2015-06-25T16:36:32.212Z cpu4:34580)WARNING: Vol3: 2427: Failed to refresh FS 53785dce-fff67823-650b-0025b302c14c descriptor: No connection
2015-06-25T16:36:32.212Z cpu4:34580)WARNING: NMP: nmp_IssueCommandToDevice:4431: I/O could not be issued to device "naa.60014050099f4f1d930ad3999d85f1d5" due to Not found
2015-06-25T16:36:40.828Z cpu7:32948)WARNING: NMP: nmp_IssueCommandToDevice:4431: I/O could not be issued to device "naa.60014050099f4f1d930ad3999d85f1d5" due to Not found
2015-06-25T16:36:40.828Z cpu7:32948)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.60014050099f4f1d930ad3999d85f1d5": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2015-06-25T16:36:40.828Z cpu7:32948)WARNING: NMP: nmp_DeviceStartLoop:725: NMP Device "naa.60014050099f4f1d930ad3999d85f1d5" is blocked. Not starting I/O from device.
2015-06-25T16:36:41.827Z cpu0:33162)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - issuing command 0x439d80641d40
2015-06-25T16:36:41.827Z cpu0:33162)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:36:41.827Z cpu0:33162)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014050099f4f1d930ad3999d85f1d5": awaiting fast path state update...
2015-06-25T16:36:42.827Z cpu2:33162)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - issuing command 0x439d80641d40
2015-06-25T16:36:42.827Z cpu2:33162)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:36:42.827Z cpu2:33162)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014050099f4f1d930ad3999d85f1d5": awaiting fast path state update...
2015-06-25T16:37:12.827Z cpu7:33162)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - issuing command 0x439d80641d40
2015-06-25T16:37:12.827Z cpu7:33162)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:37:12.827Z cpu7:33162)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014050099f4f1d930ad3999d85f1d5": awaiting fast path state update...
2015-06-25T16:37:20.829Z cpu2:32995)WARNING: NMP: nmp_DeviceStartLoop:725: NMP Device "naa.60014050099f4f1d930ad3999d85f1d5" is blocked. Not starting I/O from device.
2015-06-25T16:37:21.828Z cpu3:87910)WARNING: NMP: nmpDeviceAttemptFailover:566: Retry world restore device "naa.60014050099f4f1d930ad3999d85f1d5" - no more commands to retry
2015-06-25T16:37:21.828Z cpu6:33161)WARNING: NMP: nmp_IssueCommandToDevice:4431: I/O could not be issued to device "naa.60014050099f4f1d930ad3999d85f1d5" due to Not found
2015-06-25T16:37:21.828Z cpu6:33161)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.60014050099f4f1d930ad3999d85f1d5": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2015-06-25T16:37:21.828Z cpu6:33161)WARNING: NMP: nmp_DeviceStartLoop:725: NMP Device "naa.60014050099f4f1d930ad3999d85f1d5" is blocked. Not starting I/O from device.
2015-06-25T16:37:22.827Z cpu7:33162)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - issuing command 0x439d80641d40
2015-06-25T16:37:22.827Z cpu7:33162)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:37:22.827Z cpu7:33162)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014050099f4f1d930ad3999d85f1d5": awaiting fast path state update...
2015-06-25T16:37:23.827Z cpu5:87910)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - issuing command 0x439d80641d40
2015-06-25T16:37:23.827Z cpu5:87910)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:37:23.827Z cpu5:87910)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014050099f4f1d930ad3999d85f1d5": awaiting fast path state update...
2015-06-25T16:37:53.827Z cpu4:87910)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - issuing command 0x439d80641d40
2015-06-25T16:37:53.827Z cpu4:87910)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014050099f4f1d930ad3999d85f1d5" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:37:53.827Z cpu4:87910)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014050099f4f1d930ad3999d85f1d5": awaiting fast path state update...
2015-06-25T16:38:00.831Z cpu2:32995)WARNING: NMP: nmp_DeviceStartLoop:725: NMP Device "naa.60014050099f4f1d930ad3999d85f1d5" is blocked. Not starting I/O from device.
2015-06-25T16:38:00.831Z cpu7:32948)WARNING: NMP: nmp_IssueCommandToDevice:4431: I/O could not be issued to device "naa.60014057364b892d53e6d3961d8141d4" due to Not found
2015-06-25T16:38:00.831Z cpu7:32948)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.60014057364b892d53e6d3961d8141d4": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2015-06-25T16:38:00.831Z cpu7:32948)WARNING: NMP: nmp_DeviceStartLoop:725: NMP Device "naa.60014057364b892d53e6d3961d8141d4" is blocked. Not starting I/O from device.
2015-06-25T16:38:01.827Z cpu0:87910)WARNING: NMP: nmpDeviceAttemptFailover:566: Retry world restore device "naa.60014050099f4f1d930ad3999d85f1d5" - no more commands to retry
2015-06-25T16:38:01.827Z cpu3:33162)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014057364b892d53e6d3961d8141d4" - issuing command 0x439d80641d40
2015-06-25T16:38:01.827Z cpu3:33162)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014057364b892d53e6d3961d8141d4" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:38:01.827Z cpu3:33162)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014057364b892d53e6d3961d8141d4": awaiting fast path state update...
2015-06-25T16:38:02.827Z cpu1:87910)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014057364b892d53e6d3961d8141d4" - issuing command 0x439d80641d40
2015-06-25T16:38:02.827Z cpu1:87910)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014057364b892d53e6d3961d8141d4" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:38:02.827Z cpu1:87910)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014057364b892d53e6d3961d8141d4": awaiting fast path state update...
2015-06-25T16:38:32.828Z cpu4:87910)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014057364b892d53e6d3961d8141d4" - issuing command 0x439d80641d40
2015-06-25T16:38:32.828Z cpu4:87910)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014057364b892d53e6d3961d8141d4" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:38:32.828Z cpu4:87910)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014057364b892d53e6d3961d8141d4": awaiting fast path state update...
2015-06-25T16:38:40.833Z cpu0:32995)WARNING: NMP: nmp_DeviceStartLoop:725: NMP Device "naa.60014057364b892d53e6d3961d8141d4" is blocked. Not starting I/O from device.
2015-06-25T16:38:41.828Z cpu6:33162)WARNING: NMP: nmpDeviceAttemptFailover:566: Retry world restore device "naa.60014057364b892d53e6d3961d8141d4" - no more commands to retry
2015-06-25T16:38:41.828Z cpu1:33161)WARNING: NMP: nmp_IssueCommandToDevice:4431: I/O could not be issued to device "naa.60014057364b892d53e6d3961d8141d4" due to Not found
2015-06-25T16:38:41.828Z cpu1:33161)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.60014057364b892d53e6d3961d8141d4": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2015-06-25T16:38:41.828Z cpu1:33161)WARNING: NMP: nmp_DeviceStartLoop:725: NMP Device "naa.60014057364b892d53e6d3961d8141d4" is blocked. Not starting I/O from device.
2015-06-25T16:38:42.828Z cpu7:87910)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014057364b892d53e6d3961d8141d4" - issuing command 0x439d80641d40
2015-06-25T16:38:42.828Z cpu7:87910)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014057364b892d53e6d3961d8141d4" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:38:42.828Z cpu7:87910)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014057364b892d53e6d3961d8141d4": awaiting fast path state update...
2015-06-25T16:38:43.828Z cpu5:33162)WARNING: NMP: nmpDeviceAttemptFailover:603: Retry world failover device "naa.60014057364b892d53e6d3961d8141d4" - issuing command 0x439d80641d40
2015-06-25T16:38:43.828Z cpu5:33162)WARNING: NMP: nmpDeviceAttemptFailover:678: Retry world failover device "naa.60014057364b892d53e6d3961d8141d4" - failed to issue command due to Not found (APD), try again...
2015-06-25T16:38:43.828Z cpu5:33162)WARNING: NMP: nmpDeviceAttemptFailover:728: Logical device "naa.60014057364b892d53e6d3961d8141d4": awaiting fast path state update...
2015-06-25T16:39:04.127Z cpu7:32959)WARNING: NMP: nmp_DeviceStartLoop:725: NMP Device "naa.60014057364b892d53e6d3961d8141d4" is blocked. Not starting I/O from device.
2015-06-25T16:39:04.828Z cpu0:87910)WARNING: NMP: nmpDeviceAttemptFailover:566: Retry world restore device "naa.60014057364b892d53e6d3961d8141d4" - no more commands to retry
2015-06-25T16:40:42.197Z cpu5:87696)WARNING: Vol3: 2427: Failed to refresh FS 53785dce-fff67823-650b-0025b302c14c descriptor: No connection
2015-06-25T16:41:12.838Z cpu4:32809)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:41:12.838Z cpu4:32809)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:46:12.840Z cpu4:32810)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:46:12.840Z cpu4:32810)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:51:12.843Z cpu0:32811)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:51:12.843Z cpu0:32811)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T16:56:12.845Z cpu0:32809)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T16:56:12.845Z cpu0:32809)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T17:01:12.848Z cpu1:32810)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T17:01:12.848Z cpu1:32810)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T17:06:12.850Z cpu3:32811)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T17:06:12.850Z cpu3:32811)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
2015-06-25T17:11:12.853Z cpu0:32809)WARNING: HBX: 3278: 'DS214plus': HB at offset 3969024 - Reclaiming timed out HB failed: No connection:
2015-06-25T17:11:12.853Z cpu0:32809)WARNING: [HB state abcdef02 offset 3969024 gen 143 stampUS 77765009680 uuid 558afd87-d3f58c7e-c8d6-6805ca0fedec jrnl <FB 2405421> drv 14.61 lockImpl 3]
0:00:00:06.362 cpu0:32768)WARNING: VMKAcpi: 754: No IPMI PNP id found
2015-06-25T17:16:11.612Z cpu0:33128)WARNING: LinuxSignal: 541: ignored unexpected signal flags 0x2 (sig 17)