【LVM】系统重装后LVM分区挂载的坑
遇到的问题：
系统重装前有个LVM 做过扩容。重装后后手动挂载是可以的。系统重启后挂载失败。
原ubuntu12.04 server 版本分区如下。
磁盘配置
由于在升级glibc 导致系统重启后无法启动，做系统修复操作后还是错。于是重装系统。
重装工程中对原 ext4分区进行格式化，LVM 未做改动。
新系统磁盘分配
系统安装后安装完。执行以下命令后可以正常挂载访问。
e2label /dev/vgjenkins/lvjenkins /jenkins
e2label /dev/vgbuild/lvlocaldata /localdata
mkdir /mnt/jenkins/
mkdir /mnt/localdata
mount -L /jenkins /mnt/jenkins
mount -L /localdata /mnt/localdata修改 /etc/fstab
vim /etc/fstab
#--add below
LABEL=/localdata /mnt/localdata ext4 defaults,relatime,usrquota,data=writeback 0 2
LABEL=/jenkins /mnt/jenkins ext4 defaults 0 2
重启系统提示 /locadaldata 无法挂载。按S后跳过挂载开机。检查LVM 配置如下
vgdisplay
Couldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.
--- Volume group ---
VG Name vgbuild
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 2
Act PV 1
VG Size 6.94 TiB
PE Size 4.00 MiB
Total PE 1819968
Alloc PE / Size 1819592 / 6.94 TiB
Free PE / Size 376 / 1.47 GiB
VG UUID qM5sXD-YUFC-Ylwq-0VM1-qpI7-OvAc-CLdFJt
--- Volume group ---
VG Name vgjenkins
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 465.66 GiB
PE Size 4.00 MiB
Total PE 119209
Alloc PE / Size 119209 / 465.66 GiB
Free PE / Size 0 / 0
VG UUID dVGlj7-nF52-csCj-8oMg-tddl-1WtY-Zn7IeW
root@xmbuilder02:~# vgscan
Reading all physical volumes. This may take a while...
Couldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.
Found volume group "vgbuild" using metadata type lvm2
Found volume group "vgjenkins" using metadata type lvm2
root@xmbuilder02:~# lvscan
Couldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.
inactive '/dev/vgbuild/lvlocaldata' [6.94 TiB] inherit
ACTIVE '/dev/vgjenkins/lvjenkins' [465.66 GiB] inherit
查看磁盘的物理卷组
# pvdisplay
Couldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.
--- Physical volume ---
PV Name /dev/sda9
VG Name vgbuild
PV Size 4.22 TiB / not usable 1.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 1105249
Free PE 0
Allocated PE 1105249
PV UUID 0YRTc9-dBEp-Q6Oc-sqyX-PTzh-oqzj-AVUXHM
--- Physical volume ---
PV Name unknown device
VG Name vgbuild
PV Size 2.73 TiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 714719
Free PE 376
Allocated PE 714343
PV UUID n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc
--- Physical volume ---
PV Name /dev/sda8
VG Name vgjenkins
PV Size 465.66 GiB / not usable 1.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 119209
Free PE 0
Allocated PE 119209
PV UUID cTpMy1-ml54-jMFu-UN09-KR9E-4XT7-gStZ49
可以确定是 /dev/sdb 丢了
#lvscan
Couldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.
inactive '/dev/vgbuild/lvlocaldata' [6.94 TiB] inherit
ACTIVE '/dev/vgjenkins/lvjenkins' [465.66 GiB] inherit
# pvdisplay
Couldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.
--- Physical volume ---
PV Name unknown device
VG Name vgbuild
PV Size 2.73 TiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 714719
Free PE 376
Allocated PE 714343
PV UUID n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc
fdisk 命令查看磁盘
#fdisk -l
WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sdb: 2997.8 GB, 2997752954880 bytes
255 heads, 63 sectors/track, 364456 cylinders, total 5854986240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sdb1 1 4294967295 2147483647+ ee GPT
WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sda: 5396.0 GB, 5395955318784 bytes
255 heads, 63 sectors/track, 656020 cylinders, total 10538975232 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sda1 1 4294967295 2147483647+ ee GPT
Disk /dev/mapper/vgjenkins-lvjenkins: 500.0 GB, 499998785536 bytes
255 heads, 63 sectors/track, 60788 cylinders, total 976560128 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/mapper/vgjenkins-lvjenkins doesn't contain a valid partition table
分析：
1. 找了不少信息，也在不少技术群里问了，都没人回答，其实 想知道我要是用pvcreate 创建/dev/sdb1会导致之前数据丢失么。要不我很想试试这个命令 来新建丢失的PV
pvcreate /dev/sdb1 --u n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc --norestorefile2. 或者考虑 vgreduce或者 pvremove 来移除这个 PV Name unknown devicec，然后再添加上去。
但是vgreduce和命令不是太熟悉 pvremove
| vgreduce | pvremove | |
|---|---|---|
| 作用 | vgreduce命令通过删除LVM卷组中的物理卷来减少卷组容量。不能删除LVM卷组中剩余的最后一个物理卷。 | pvremove命令用于删除一个存在的物理卷。使用pvremove指令删除物理卷时， 它将LVM分区上的物理卷信息删除，使其不再被视为一个物理卷。 |
| 语法 | vgreduce(选项)(参数) | pvremove(选项)(参数) |
| 选项 | -a：如果命令行中没有指定要删除的物理卷，则删除所有的空物理卷； --removemissing：删除卷组中丢失的物理卷，使卷组恢复正常状态。 |
-d：调试模式； -f：强制删除； -y：对提问回答“yes”。 |
| 实例 | 使用vgreduce命令从卷组"vg2000"中移除物理卷/dev/sdb2。 在命令行中输入下面的命令： [root@localhost ~]# vgreduce vg2000 /dev/sdb2 #将物理卷"/dev/sdb2"从卷组"vg2000"中删除 输出信息如下： Removed "/dev/sdb2" from volume group "vg2000" |
使用pvremove指令删除物理卷/dev/sdb2。 在命令行中输入下面的命令： pvremove /dev/sdb2 #删除物理卷 Labels on physical volume "/dev/sdb2" successfully wiped |
3. 害怕数据丢失，还是来回顾一下之前添加扩容的过程吧
解决方法：
1 单纯的移除 vgbuild 中丢失的部分是不行的因为 VG vgbuild 中有一部分扩容，出问题的刚好是扩容的那部分。
root@xmbuilder02:/etc/lvm/backup# pvs
Couldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.
PV VG Fmt Attr PSize PFree
/dev/sda8 vgjenkins lvm2 a-- 465.66g 0
/dev/sda9 vgbuild lvm2 a-- 4.22t 0
unknown device vgbuild lvm2 a-m 2.73t 1.47g
root@xmbuilder02:/etc/lvm/backup# vgreduce --removemissing vgbuild
Couldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.
WARNING: Partial LV lvlocaldata needs to be repaired or removed.
There are still partial LVs in VG vgbuild.
To remove them unconditionally use: vgreduce --removemissing --force.
Proceeding to remove empty missing PVs.
打算重建PV了,用pvcreate 后sdb 是有了。
#pvcreate -u n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc --restorefile /etc/lvm/backup/vgbuild /dev/sdb
Couldn't find device with uuid n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc.
Physical volume "/dev/sdb" successfully created
root@xmbuilder02:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda8 vgjenkins lvm2 a-- 465.66g 0
/dev/sda9 vgbuild lvm2 a-- 4.22t 0
/dev/sdb vgbuild lvm2 a-m 2.73t 1.47g
于是在做个reteore试试
root@xmbuilder02:~#vgcfgrestore vgbuild
root@xmbuilder02:~# vgcfgrestore vgbuild
Restored volume group vgbuild
root@xmbuilder02:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda8 vgjenkins lvm2 a-- 465.66g 0
/dev/sda9 vgbuild lvm2 a-- 4.22t 0
/dev/sdb vgbuild lvm2 a-- 2.73t 1.47g最终的记录
root@xmbuilder02:~# pvs
PV VG Fmt Attr PSize PFree
/dev/sda8 vgjenkins lvm2 a-- 465.66g 0
/dev/sda9 vgbuild lvm2 a-- 4.22t 0
/dev/sdb vgbuild lvm2 a-- 2.73t 1.47g
root@xmbuilder02:~# vgscan
Reading all physical volumes. This may take a while...
Found volume group "vgbuild" using metadata type lvm2
Found volume group "vgjenkins" using metadata type lvm2
root@xmbuilder02:~# lvscan
ACTIVE '/dev/vgbuild/lvlocaldata' [6.94 TiB] inherit
ACTIVE '/dev/vgjenkins/lvjenkins' [465.66 GiB] inherit
root@xmbuilder02:~# vgdisplay
--- Volume group ---
VG Name vgbuild
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 10
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 2
Act PV 2
VG Size 6.94 TiB
PE Size 4.00 MiB
Total PE 1819968
Alloc PE / Size 1819592 / 6.94 TiB
Free PE / Size 376 / 1.47 GiB
VG UUID qM5sXD-YUFC-Ylwq-0VM1-qpI7-OvAc-CLdFJt
--- Volume group ---
VG Name vgjenkins
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 465.66 GiB
PE Size 4.00 MiB
Total PE 119209
Alloc PE / Size 119209 / 465.66 GiB
Free PE / Size 0 / 0
VG UUID dVGlj7-nF52-csCj-8oMg-tddl-1WtY-Zn7IeW
root@xmbuilder02:~# pvdisplay
--- Physical volume ---
PV Name /dev/sda9
VG Name vgbuild
PV Size 4.22 TiB / not usable 1.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 1105249
Free PE 0
Allocated PE 1105249
PV UUID 0YRTc9-dBEp-Q6Oc-sqyX-PTzh-oqzj-AVUXHM
--- Physical volume ---
PV Name /dev/sdb
VG Name vgbuild
PV Size 2.73 TiB / not usable 4.00 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 714719
Free PE 376
Allocated PE 714343
PV UUID n3lryH-nt0A-8H1g-6Dz3-gR43-8DKE-BvRXcc
--- Physical volume ---
PV Name /dev/sda8
VG Name vgjenkins
PV Size 465.66 GiB / not usable 1.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 119209
Free PE 0
Allocated PE 119209
PV UUID cTpMy1-ml54-jMFu-UN09-KR9E-4XT7-gStZ49
最终的结果，东西没丢，舒心一笑。
root@xmbuilder02:~# e2label /dev/vgbuild/lvlocaldata /localdata
root@xmbuilder02:~# mount -L /localdata /mnt/localdata
root@xmbuilder02:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 63G 12K 63G 1% /dev
tmpfs 13G 1.9M 13G 1% /run
/dev/sda2 184G 400M 174G 1% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 63G 0 63G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda5 9.1G 21M 8.6G 1% /tmp
/dev/sda3 181M 38M 131M 23% /boot
/dev/sda6 9.1G 557M 8.1G 7% /usr
/dev/sda7 9.1G 350M 8.3G 4% /var
/dev/mapper/vgjenkins-lvjenkins 459G 324G 136G 71% /mnt/jenkins
/dev/mapper/vgbuild-lvlocaldata 6.9T 6.8T 156G 98% /mnt/localdata