hp ilo 服务器 hpssacli 做 raid
环境： hp ilo4 DL380 Gen9，ilo 页面检测到磁盘 degrade，但是操作系统 centos7 中显示磁盘（做过 raid 后的逻辑盘）正常。更换磁盘后，系统显示不出新的磁盘符 /dev/sdx，需要手动重启操作系统, 做 raid
目的：不停机，直接使用 hpssacli 做 raid0
需要提前删除更换的 ceph osd 对应的 lv、pv、vg
1、不重新对更换后的磁盘做 raid，系统显示不出更换后的磁盘设备
[root@cmp21 osd]# lsblk -d
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 558.9G 0 disk
sdb 8:16 0 1.7T 0 disk
sdc 8:32 0 1.7T 0 disk
sdd 8:48 0 1.7T 0 disk
sde 8:64 0 1.7T 0 disk
sdf 8:80 0 1.7T 0 disk
sdg 8:96 0 1.7T 0 disk
sdh 8:112 0 1.7T 0 disk
sdi 8:128 0 1.7T 0 disk
sdk 8:160 0 1.7T 0 disk
sdl 8:176 0 1.7T 0 disk
sdm 8:192 0 1.7T 0 disk
sdn 8:208 0 1.7T 0 disk
sdo 8:224 0 1.7T 0 disk
sdp 8:240 0 1.7T 0 disk
sdq 65:0 0 1.7T 0 disk
sdr 65:16 0 1.7T 0 disk
sds 65:32 0 1.7T 0 disk
sdt 65:48 0 1.7T 0 disk
sdu 65:64 0 1.7T 0 disk
缺少 /dev/sdj
2、使用 hpssacli 检查逻辑盘，发现 logicaldrive 10 failed
[root@cmp21 osd]# hpssacli ctrl slot=0 ld all show status
logicaldrive 1 (558.9 GB, 1): OK
logicaldrive 2 (1.6 TB, 0): OK
logicaldrive 3 (1.6 TB, 0): OK
logicaldrive 4 (1.6 TB, 0): OK
logicaldrive 5 (1.6 TB, 0): OK
logicaldrive 6 (1.6 TB, 0): OK
logicaldrive 7 (1.6 TB, 0): OK
logicaldrive 8 (1.6 TB, 0): OK
logicaldrive 9 (1.6 TB, 0): OK
logicaldrive 10 (1.6 TB, 0): Failed
logicaldrive 11 (1.6 TB, 0): OK
logicaldrive 12 (1.6 TB, 0): OK
logicaldrive 13 (1.6 TB, 0): OK
logicaldrive 14 (1.6 TB, 0): OK
logicaldrive 15 (1.6 TB, 0): OK
logicaldrive 16 (1.6 TB, 0): OK
logicaldrive 17 (1.6 TB, 0): OK
logicaldrive 18 (1.6 TB, 0): OK
logicaldrive 19 (1.6 TB, 0): OK
logicaldrive 20 (1.6 TB, 0): OK
logicaldrive 21 (1.6 TB, 0): OK
而物理盘状态显示没有问题
[root@cmp21 osd]# hpssacli ctrl slot=0 pd all show status
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, 600 GB): OK
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, 600 GB): OK
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, 1800.3 GB): OK
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, 1800.3 GB): OK
physicaldrive 1I:1:5 (port 1I:box 1:bay 5, 1800.3 GB): OK
physicaldrive 1I:1:6 (port 1I:box 1:bay 6, 1800.3 GB): OK
physicaldrive 1I:1:7 (port 1I:box 1:bay 7, 1800.3 GB): OK
physicaldrive 1I:1:8 (port 1I:box 1:bay 8, 1800.3 GB): OK
physicaldrive 1I:1:9 (port 1I:box 1:bay 9, 1800.3 GB): OK
physicaldrive 1I:1:10 (port 1I:box 1:bay 10, 1800.3 GB): OK
physicaldrive 1I:1:11 (port 1I:box 1:bay 11, 1800.3 GB): OK
physicaldrive 1I:1:12 (port 1I:box 1:bay 12, 1800.3 GB): OK
physicaldrive 1I:1:13 (port 1I:box 1:bay 13, 1800.3 GB): OK
physicaldrive 1I:1:14 (port 1I:box 1:bay 14, 1800.3 GB): OK
physicaldrive 1I:1:15 (port 1I:box 1:bay 15, 1800.3 GB): OK
physicaldrive 1I:1:16 (port 1I:box 1:bay 16, 1800.3 GB): OK
physicaldrive 1I:1:17 (port 1I:box 1:bay 17, 1800.3 GB): OK
physicaldrive 1I:1:18 (port 1I:box 1:bay 18, 1800.3 GB): OK
physicaldrive 1I:1:19 (port 1I:box 1:bay 19, 1800.3 GB): OK
physicaldrive 1I:1:20 (port 1I:box 1:bay 20, 1800.3 GB): OK
physicaldrive 1I:1:21 (port 1I:box 1:bay 21, 1800.3 GB): OK
physicaldrive 1I:1:22 (port 1I:box 1:bay 22, 1800.3 GB): OK
3、删除 logicaldrive 10
[root@cmp21 osd]# hpssacli ctrl slot=0 ld 10 delete
Warning: Deleting an array can cause other array letters to become renamed.
E.g. Deleting array A from arrays A,B,C will result in two remaining
arrays A,B ... not B,C
Warning: Deleting the specified device(s) will result in data being lost.
Continue? (y/n) y
4、重新创建 raid0
[root@cmp21 osd]# hpssacli ctrl slot=0 create type=ld drives=1I:1:11 raid=0
Warning: Creation of this logical drive has caused array letters to become
renamed.
5、查看系统磁盘信息
[root@cmp21 osd]# lsblk -d
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 558.9G 0 disk
sdb 8:16 0 1.7T 0 disk
sdc 8:32 0 1.7T 0 disk
sdd 8:48 0 1.7T 0 disk
sde 8:64 0 1.7T 0 disk
sdf 8:80 0 1.7T 0 disk
sdg 8:96 0 1.7T 0 disk
sdh 8:112 0 1.7T 0 disk
sdi 8:128 0 1.7T 0 disk
sdj 8:144 0 1.7T 0 disk
sdk 8:160 0 1.7T 0 disk
sdl 8:176 0 1.7T 0 disk
sdm 8:192 0 1.7T 0 disk
sdn 8:208 0 1.7T 0 disk
sdo 8:224 0 1.7T 0 disk
sdp 8:240 0 1.7T 0 disk
sdq 65:0 0 1.7T 0 disk
sdr 65:16 0 1.7T 0 disk
sds 65:32 0 1.7T 0 disk
sdt 65:48 0 1.7T 0 disk
sdu 65:64 0 1.7T 0 disk
注意事项：
需要提前删除更换的 ceph osd 对应的 lv、pv、vg，否则磁盘符会增加，不会恢复到原来的磁盘符
[root@cmp13 ~]# lsblk -d
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 558.9G 0 disk
sdb 8:16 0 1.7T 0 disk
sdc 8:32 0 1.7T 0 disk
sdd 8:48 0 1.7T 0 disk
sde 8:64 0 1.7T 0 disk
sdf 8:80 0 1.7T 0 disk
sdg 8:96 0 1.7T 0 disk
sdh 8:112 0 1.7T 0 disk
sdi 8:128 0 1.7T 0 disk
sdj 8:144 0 1.7T 0 disk
sdk 8:160 0 1.7T 0 disk
sdl 8:176 0 1.7T 0 disk
sdm 8:192 0 1.7T 0 disk
sdn 8:208 0 1.7T 0 disk
sdo 8:224 0 1.7T 0 disk
sdp 8:240 0 1.7T 0 disk
sdq 65:0 0 1.7T 0 disk
sds 65:32 0 1.7T 0 disk
sdt 65:48 0 1.7T 0 disk
sdu 65:64 0 1.7T 0 disk
sdv 65:80 0 1.7T 0 disk
这时使用 hpssacli 清除 ld，再重新创建 ld，新的磁盘符不会出现，还会出现 multiple uuid 错误，pvscan --cache 会报错
pvs
pvscan --cache
hpssacli 清除 ld
hpssacli ctrl slot=0 ld x delete
刷新磁盘管理
echo "- - -" > /sys/class/scsi_host/hostx/scan
清除 /dev/sdv
echo 1 > /sys/block/sdx/device/delete
清除错误 Error reading device /dev/ceph-3310ea32-d1d3-4b15-86ef-ef3047684498/osd-data-99e5b417-19f9-427c-aa77-d8dc7cece135 at 0 length 512.
[root@cmp13 ~]# pvs
Error reading device /dev/ceph-3310ea32-d1d3-4b15-86ef-ef3047684498/osd-data-99e5b417-19f9-427c-aa77-d8dc7cece135 at 0 length 512.
Error reading device /dev/ceph-3310ea32-d1d3-4b15-86ef-ef3047684498/osd-data-99e5b417-19f9-427c-aa77-d8dc7cece135 at 0 length 4.
Error reading device /dev/ceph-3310ea32-d1d3-4b15-86ef-ef3047684498/osd-data-99e5b417-19f9-427c-aa77-d8dc7cece135 at 4096 length 4.
PV VG Fmt Attr PSize PFree
/dev/sdb ceph-8fb04441-4899-4470-979f-b82319df1c04 lvm2 a-- <1.64t 0
/dev/sdc ceph-dea3360a-dea5-45fd-918b-86b530750d51 lvm2 a-- <1.64t 0
/dev/sdd ceph-91a54c77-1d74-4c7a-96c7-d97cda9ce269 lvm2 a-- <1.64t 0
/dev/sde ceph-76e9dd42-35ac-426d-9733-1c123be18a4c lvm2 a-- <1.64t 0
/dev/sdf ceph-d8ed0ac2-07e2-47ca-8be2-2517064336fe lvm2 a-- <1.64t 0
/dev/sdg ceph-f82bd637-daf2-4ad2-9660-95df9b1dfca9 lvm2 a-- <1.64t 0
/dev/sdh ceph-c688cfe8-0488-4646-942c-bba590eb01c5 lvm2 a-- <1.64t 0
/dev/sdi ceph-70ae3747-d758-4b28-973f-0acd2d21e6b2 lvm2 a-- <1.64t 0
/dev/sdj ceph-44c4d98f-1940-4d9f-b132-3fdf5178233c lvm2 a-- <1.64t 0
/dev/sdk ceph-a1b13b3b-b8d7-40d4-b7d4-ca5174adb012 lvm2 a-- <1.64t 0
/dev/sdl ceph-2f15820c-0e12-4518-a103-26bd3ee35d4e lvm2 a-- <1.64t 0
/dev/sdm ceph-ed5d6897-d93c-4037-a3e8-5a7fce070ba8 lvm2 a-- <1.64t 0
/dev/sdn ceph-b8502c37-69b9-45a9-bbaf-d29d6d322d991 lvm2 a-- <1.64t 0
/dev/sdo ceph-6968d0c9-6b29-4457-8e08-b292324675f4 lvm2 a-- <1.64t 0
/dev/sdp ceph-a102a5f5-a59e-4dd0-9c88-8c721603af6e lvm2 a-- <1.64t 0
/dev/sdq ceph-49543ed2-4b3e-43d9-aa25-7693ded3c898 lvm2 a-- <1.64t 0
/dev/sds ceph-15fb9166-e800-4556-bfee-4a97b7f598b5 lvm2 a-- <1.64t 0
/dev/sdt ceph-f7f0916c-385d-4f41-a041-0009d914bd0e lvm2 a-- <1.64t 0
/dev/sdu ceph-42315175-e55c-4bc3-8db6-c184986f28b4 lvm2 a-- <1.64t 0
[root@cmp13 ~]# dmsetup remove /dev/ceph-3310ea32-d1d3-4b15-86ef-ef3047684498/osd-data-99e5b417-19f9-427c-aa77-d8dc7cece135
参考文章：
17 hpacucli Command Examples for Linux on HP Servers
Opening the CLI in Console mode
Physically removed a disk before deactivating volume group and can’t get LVM to stop printing errors about it
Linux 不重启扫描存储磁盘