Issue
VPLEX Online help indicates the following cause for a degrade connectivity status:
| Degraded |
A LUN exported from the array does not have two paths to each |
On a CLARiiON:
If you have collect diagnostics, look at the debugTowerDump files. Search for "scsi view array" and you should see an output similar to below:
COMMAND scsi view array * all
DIRECTOR /engines/engine-1-1/directors/director-1-1-A
a: EMC~CLARiiON~APM00111000XXX name APM00111000XXX vend "DGC "
prod "VRAID " rev "0532" type EMC~CLARiiON
c: APM00111000XXX.SPA
c: APM00111000XXX.SPB
u: VPD83T3:600601607ee028006115390b2d25e311
u: VPD83T3:600601607ee02800bf8e45f52c25e311
u: VPD83T3:50060160bde007a9
l: "x fcp i 0x5000144280728a13 t 0x5006016c3de007a9"
l: "x fcp i 0x5000144280728a12 t 0x500601683de007a9"
l: "x fcp i 0x5000144280728a13 t 0x500601643de007a9"
l: "x fcp i 0x5000144280728a12 t 0x500601603de007a9"
p: "x fcp i 0x5000144280728a13 t 0x500601643de007a9 l 0x0000000000000000" AAO
p: "x fcp i 0x5000144280728a12 t 0x500601683de007a9 l 0x0000000000000000" DL0
p: "x fcp i 0x5000144280728a13 t 0x500601643de007a9 l 0x0001000000000000" AAO
p: "x fcp i 0x5000144280728a12 t 0x500601603de007a9 l 0x0001000000000000" AAO
p: "x fcp i 0x5000144280728a13 t 0x5006016c3de007a9 l 0x0000000000000000" DL0
p: "x fcp i 0x5000144280728a12 t 0x500601603de007a9 l 0x0000000000000000" AAO
Note the paths which end in a DL0. These represent Default LUN 0 because VPLEX does not see storage on these paths and assigns a DL0 (similar to a LUNZ). With proper backend connectivity, all paths should show as AAN and AAO.
a: EMC~CLARiiON~APM00051002XXX name APM00051002XXX vend "DGC "
prod "RAID 5 " rev "0226" type EMC~CLARiiON
c: APM00051002XXX.SPB
c: APM00051002XXX.SPA
u: VPD83T3:60060160fd221100f5e474acaadfe211
l: "x fcp i 0x5000144260072011 t 0x50060169106026fe"
l: "x fcp i 0x5000144260072010 t 0x50060169106026fe"
l: "x fcp i 0x5000144260072010 t 0x50060160106026fe"
l: "x fcp i 0x5000144260072011 t 0x50060160106026fe"
l: "x fcp i 0x5000144260072011 t 0x50060161106026fe"
l: "x fcp i 0x5000144260072010 t 0x50060161106026fe"
l: "x fcp i 0x5000144260072011 t 0x50060168106026fe"
l: "x fcp i 0x5000144260072010 t 0x50060168106026fe"
p: "x fcp i 0x5000144260072010 t 0x50060161106026fe l 0x0000000000000000" AAN
p: "x fcp i 0x5000144260072010 t 0x50060169106026fe l 0x0000000000000000" AAO
p: "x fcp i 0x5000144260072010 t 0x50060160106026fe l 0x0000000000000000" AAN
p: "x fcp i 0x5000144260072010 t 0x50060168106026fe l 0x0000000000000000" AAO
p: "x fcp i 0x5000144260072011 t 0x50060161106026fe l 0x0000000000000000" AAN
p: "x fcp i 0x5000144260072011 t 0x50060169106026fe l 0x0000000000000000" AAO
p: "x fcp i 0x5000144260072011 t 0x50060160106026fe l 0x0000000000000000" AAN
p: "x fcp i 0x5000144260072011 t 0x50060168106026fe l 0x0000000000000000" AAO
On a Symmetrix:
Array and logical-units show as degraded, but all storage-volumes are healthy.
EMC-SYMMETRIX-190100XXX degraded - 0x50060482d52cc607, 474
0x50060482d52cc608,
0x50060482d52cc617,
0x50060482d52cc618,
0x50060482d52cc627,
0x50060482d52cc628,
/clusters/cluster-1/storage-elements/storage-arrays/EMC-SYMMETRIX-190100XXX/logical-units:
Name Connectivity Active/AAO Passive/AAN Visibility LUNs ALUA
---------------------------------------- Status Controllers Controllers ---------- ------------------- Support
---------------------------------------- ------------ ----------- ----------- ---------- ------------------- -------
VPD83T3:60060480000190100248533030344442 degraded 190100XXX active 0x004f000000000000 none
VPD83T3:60060480000190100248533030344445 degraded 190100XXX active 0x0050000000000000 none
VPD83T3:60060480000190100248533030344530 degraded 190100XXX active 0x0051000000000000 none
VPD83T3:60060480000190100248533030344534 degraded 190100XXX active 0x0052000000000000 none
VPD83T3:60060480000190100248533030344537 degraded 190100XXX active 0x0053000000000000 none
VPD83T3:60060480000190100248533030344543 degraded 190100XXX active 0x0055000000000000 none
VPD83T3:60060480000190100248533030344630 degraded 190100XXX active 0x0056000000000000 none
VPlexcli:/> connectivity validate-be
Storage volumes that are dead or unreachable:
Storage volumes which do not meet the high availability requirement for storage volume paths*:
Cluster Director Array
Storage volumes which do not meet the high availability requirement for storage volume paths*
---------- -------------- ------------------------------------ ---------------------------------------------------------------------------------------------
cluster-1 director-1-1-B (190100XXX, EMC~SYMMETRIX~190100XXX) VPD83T3:60060480000190100248533031364442 VPD83T3:60060480000190100248533030354634
VPD83T3:60060480000190100248533031364438
VPD83T3:60060480000190100248533030363138
VPD83T3:60060480000190100248533031333441
VPD83T3:60060480000190100248533031373731
VPD83T3:60060480000190100248533030353543
VPD83T3:60060480000190100248533030353141
VPD83T3:60060480000190100248533031333439
...
Summary
Cluster cluster-1
5 storage-volumes which are dead or unreachable.
104 storage-volumes which do not meet the high availability requirement for storage volume paths*.
Director director-1-1-B
Storage array: ('190100XXX', 'EMC~SYMMETRIX~190100XXX') has 104 storage-volumes which do not meet the high availability requirement for storage volume paths*.
0 storage-volumes which are not visible from all directors.
0 storage-volumes which have more than supported (4) active paths from same director.
*To meet the high availability requirement for storage volume paths each storage volume must be accessible from each of the directors through 2 or more VPlex backend ports, and 2 or more Array target ports, and there should be 2 or more ITLs.
Environment
EMC Hardware: VPLEX Series
EMC Hardware: VPLEX VS1
EMC Hardware: VPLEX VS2
EMC Hardware: VPLEX-Local
EMC Hardware: VPLEX-Metro
EMC Hardware: VPLEX-Geo
EMC Software: GeoSynchrony 4.0
EMC Software: GeoSynchrony 4.0.1
EMC Software: GeoSynchrony 4.1
EMC Software: GeoSynchrony 4.2
EMC Software: GeoSynchrony 4.2 Patch1
EMC Software: GeoSynchrony 5.0
EMC Software: GeoSynchrony 5.0.1
EMC Software: GeoSynchrony 5.0.1 Patch 1
EMC Software: GeoSynchrony 5.0.1 Patch 2
EMC Software: GeoSynchrony 5.1
EMC Software: GeoSynchrony 5.1 Patch 1
EMC Software: GeoSynchrony 5.1 Patch 2
EMC Software: GeoSynchrony 5.1 Patch 3
EMC Software: GeoSynchrony 5.1 Patch 4
EMC Software: GeoSynchrony 5.2
EMC Software: GeoSynchrony 5.2 Patch 1
EMC Software: GeoSynchrony 5.2 Service Pack 1
EMC Hardware: Symmetrix Series
EMC Hardware: VNX Series
EMC Hardware: CLARiiON Series
All supported third party array types
Cause
VPLEX Online help indicates the following cause for a degrade connectivity status:
Degraded
A LUN exported from the array does not have two paths to each
director.
However, a LUN will not enter a degraded status if only one path exists from the array to the VPLEX director,
For example, if the zoning is not present, or if the VPLEX director or array ports are not enabled.
This condition is caused when there are multiple paths from the array to a given VPLEX director, but the LUN is not mapped/masked to the necessary VPLEX ports.
It is possible to hit this issue on any array type, but is less common on array types where all initiators are registered to a single host entry. This article uses a Symmetrix as an example.
In the case of active/passive arrays, confirm that all initiators are registered and in the storage-group. Any VPLEX initiators which are unregistered and not in a storage-group will cause the array status to be degraded
Change
User had made zoning changes in the fabric which requires the initiators to be manually unregistered on the array (change the mapping and masking), then once the rezoning is done and the initiators have logged back onto the VNX they will be on different ports and so needed to be registered and added back to the storage-group and these two steps were missed.
Resolution
To verify this issue, pick one of the degraded devices and check the mapping and masking. For a Symmetrix, use the following commands:
VPlexcli:/clusters/cluster-1/storage-elements/storage-arrays/EMC-SYMMETRIX-190100XXX/logical-units>
VPD83T3:60060480000190100248533031333434 degraded 190100XXX active 0x006e000000000000 none
[root@sanadm]# symdev -sid 0XXX show -wwn 60060480000190100248533031333434
Device Physical Name : Not Visible
Device Symmetrix Name : 1344
Device Serial ID : N/A
Symmetrix ID : 000190100XXX
Attached BCV Device : N/A
[root@sanadm]# symmaskdb -sid 0XXX list assignment -dev 1344
Symmetrix ID : 000190100XXX
Device Identifier Type Dir:P
------ ---------------- ----- ----------------
1344 50001442806cbd12 FIBRE FA-8A:0
50001442804c2f12 FIBRE FA-8A:0
50001442904c2f12 FIBRE FA-8A:0
50001442806cbd13 FIBRE FA-9A:0
50001442906cbd13 FIBRE FA-9A:0
50001442804c2f13 FIBRE FA-9A:0
50001442904c2f13 FIBRE FA-9A:0
Note: The pwwns listed under the Identifier column. These are the VPLEX initiator ports. Check these against the available VPLEX initiator ports, by running 'll /engines/**/ports' from vplexcli. Ensure that two ports from each director are present in the symmaskdb output. In this example, director-1-1-B port B1-FC02 is missing from the symmaskdb output. This matches the output of 'connectivity validate-be' which indicates that director-1-1-B did not have the required number of paths.
VPlexcli:/> ll engines/**/ports
/engines/engine-1-1/directors/director-1-1-A/hardware/ports:
Name Address Role Port Status
------- ------------------ --------- -----------
A0-FC00 0x50001442806cbd00 front-end up
A0-FC01 0x50001442806cbd01 front-end up
A0-FC02 0x50001442806cbd02 front-end up
A0-FC03 0x50001442806cbd03 front-end up
A1-FC00 50001442806cbd10 back-end up
A1-FC01 50001442806cbd11 back-end up
A1-FC02 50001442806cbd12 back-end up
A1-FC03 50001442806cbd13 back-end up
A3-FC00 0x50001442806cbd30 local-com up
A3-FC01 0x50001442806cbd31 local-com up
A3-FC02 0x0000000000000000 - down
A3-FC03 0x0000000000000000 - down
/engines/engine-1-1/directors/director-1-1-B/hardware/ports:
Name Address Role Port Status
------- ------------------ --------- -----------
B0-FC00 0x50001442906cbd00 front-end up
B0-FC01 0x50001442906cbd01 front-end up
B0-FC02 0x50001442906cbd02 front-end up
B0-FC03 0x50001442906cbd03 front-end up
B1-FC00 50001442906cbd10 back-end up
B1-FC01 50001442906cbd11 back-end up
B1-FC02 50001442906cbd12 back-end up <---
B1-FC03 50001442906cbd13 back-end up
B3-FC00 0x50001442906cbd30 local-com up
B3-FC01 0x50001442906cbd31 local-com up
B3-FC02 0x0000000000000000 - down
B3-FC03 0x0000000000000000 - down
/engines/engine-1-2/directors/director-1-2-A/hardware/ports:
Name Address Role Port Status
------- ------------------ --------- -----------
A0-FC00 0x50001442804c2f00 front-end up
A0-FC01 0x50001442804c2f01 front-end up
A0-FC02 0x50001442804c2f02 front-end up
A0-FC03 0x50001442804c2f03 front-end up
A1-FC00 50001442804c2f10 back-end up
A1-FC01 50001442804c2f11 back-end up
A1-FC02 50001442804c2f12 back-end up
A1-FC03 50001442804c2f13 back-end up
A3-FC00 0x50001442804c2f30 local-com up
A3-FC01 0x50001442804c2f31 local-com up
A3-FC02 0x0000000000000000 - down
A3-FC03 0x0000000000000000 - down
/engines/engine-1-2/directors/director-1-2-B/hardware/ports:
Name Address Role Port Status
------- ------------------ --------- -----------
B0-FC00 0x50001442904c2f00 front-end up
B0-FC01 0x50001442904c2f01 front-end up
B0-FC02 0x50001442904c2f02 front-end up
B0-FC03 0x50001442904c2f03 front-end up
B1-FC00 50001442904c2f10 back-end up
B1-FC01 50001442904c2f11 back-end up
B1-FC02 50001442904c2f12 back-end up
B1-FC03 50001442904c2f13 back-end up
B3-FC00 0x50001442904c2f30 local-com up
B3-FC01 0x50001442904c2f31 local-com up
B3-FC02 0x0000000000000000 - down
B3-FC03 0x0000000000000000 - down
For VNX or CLARiiON, log in to Unisphere, go to the initiators tab and look for any VPLEX initiators which are logged into the array, but not registered AND in the VPLEX storage-group.
To resolve the issue, add the missing port(s) to the mapping/masking/storage-group on the array. Once this is done, run an array re-discover for VPLEX to pick up the new path.