rh436 - Facing "Volume group for uuid not found" error while run lvcreate on shared storage

Version:
[root@node2 ~]# cat /proc/version 
Linux version 2.6.32-220.el6.x86_64 ([email protected]) (gcc version 4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Wed Nov 9 08:03:13 EST 2011
[root@node2 ~]#


Issue description:
I was doing a HA LVM on test in a three node cluster, but got "Volume group for uuid not found" error while run lvcreate on multipathed shared storage, please find below for detail steps:
[root@node2 ~]# fdisk -cu /dev/mapper/netstorage


Command (m for help): p


Disk /dev/mapper/netstorage: 4294 MB, 4294967296 bytes
171 heads, 40 sectors/track, 1226 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xee9a74ab


                 Device Boot      Start         End      Blocks   Id  System
/dev/mapper/netstoragep1            2048     2099199     1048576   8e  Linux LVM


Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 
Value out of range.
Partition number (1-4): 2
First sector (2099200-8388607, default 2099200): 
Using default value 2099200
Last sector, +sectors or +size{K,M,G} (2099200-8388607, default 8388607): +2G


Command (m for help): w
The partition table has been altered!


Calling ioctl() to re-read partition table.


WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@node2 ~]# ll /dev/mapper/
total 0
lrwxrwxrwx. 1 root root      7 Jun 16 18:11 clusteredvg-webdata -> ../dm-5
crw-rw----. 1 root root 10, 58 Jun 16 18:10 control
lrwxrwxrwx. 1 root root      7 Jun 16 18:11 netstorage -> ../dm-3
lrwxrwxrwx. 1 root root      7 Jun 16 18:11 netstoragep1 -> ../dm-4
lrwxrwxrwx. 1 root root      7 Jun 16 18:10 vgsrv-home -> ../dm-2
lrwxrwxrwx. 1 root root      7 Jun 16 18:10 vgsrv-root -> ../dm-1
lrwxrwxrwx. 1 root root      7 Jun 16 18:10 vgsrv-swap -> ../dm-0
[root@node2 ~]# partprobe 
device-mapper: remove ioctl failed: Device or resource busy
Warning: parted was unable to re-read the partition table on /dev/mapper/netstorage (Device or resource busy).  This means Linux won't know anything about the modifications you made. 
device-mapper: create ioctl failed: Device or resource busy
device-mapper: remove ioctl failed: Device or resource busy
Warning: WARNING: the kernel failed to re-read the partition table on /dev/vda (Device or resource busy).  As a result, it may not reflect all of your changes until after reboot.
[root@node2 ~]# ll /dev/mapper/
total 0
lrwxrwxrwx. 1 root root       7 Jun 16 18:11 clusteredvg-webdata -> ../dm-5
crw-rw----. 1 root root  10, 58 Jun 16 18:10 control
lrwxrwxrwx. 1 root root       7 Jun 16 18:11 netstorage -> ../dm-3
lrwxrwxrwx. 1 root root       7 Jun 16 18:11 netstoragep1 -> ../dm-4
brw-rw----. 1 root disk 253,  6 Jun 16 18:24 netstoragep2
lrwxrwxrwx. 1 root root       7 Jun 16 18:10 vgsrv-home -> ../dm-2
lrwxrwxrwx. 1 root root       7 Jun 16 18:10 vgsrv-root -> ../dm-1
lrwxrwxrwx. 1 root root       7 Jun 16 18:10 vgsrv-swap -> ../dm-0
[root@node2 ~]# multipath -r
Jun 16 18:24:16 | /lib/udev/scsi_id exitted with 1
Jun 16 18:24:16 | /lib/udev/scsi_id exitted with 1
reload: netstorage (1IET_00010001) undef IET,VIRTUAL-DISK
size=4.0G features='0' hwhandler='0' wp=undef
|-+- policy='round-robin 0' prio=1 status=undef
| `- 2:0:0:1 sda 8:0   active ready running
`-+- policy='round-robin 0' prio=1 status=undef
  `- 3:0:0:1 sdb 8:16  active ready running
[root@node2 ~]# ll /dev/mapper/
total 0
lrwxrwxrwx. 1 root root      7 Jun 16 18:11 clusteredvg-webdata -> ../dm-5
crw-rw----. 1 root root 10, 58 Jun 16 18:10 control
lrwxrwxrwx. 1 root root      7 Jun 16 18:24 netstorage -> ../dm-3
lrwxrwxrwx. 1 root root      7 Jun 16 18:24 netstoragep1 -> ../dm-4
lrwxrwxrwx. 1 root root      7 Jun 16 18:24 netstoragep2 -> ../dm-6
lrwxrwxrwx. 1 root root      7 Jun 16 18:10 vgsrv-home -> ../dm-2
lrwxrwxrwx. 1 root root      7 Jun 16 18:10 vgsrv-root -> ../dm-1
lrwxrwxrwx. 1 root root      7 Jun 16 18:10 vgsrv-swap -> ../dm-0
[root@node2 ~]# pvcreate /dev/mapper/netstoragep2 
  Writing physical volume data to disk "/dev/mapper/netstoragep2"
  Physical volume "/dev/mapper/netstoragep2" successfully created
[root@node2 ~]# vgcreate -cy havg /dev/mapper/netstoragep2
  Clustered volume group "havg" successfully created
[root@node2 ~]# vgs
  VG          #PV #LV #SN Attr   VSize    VFree
  clusteredvg   1   1   0 wz--nc 1020.00m    0 
  havg          1   0   0 wz--nc    2.00g 2.00g
  vgsrv         1   3   0 wz--n-    8.78g 4.69g
[root@node2 ~]# pvs
p  PV                       VG          Fmt  Attr PSize    PFree
  /dev/mapper/netstoragep1 clusteredvg lvm2 a--  1020.00m    0 
  /dev/mapper/netstoragep2 havg        lvm2 a--     2.00g 2.00g
  /dev/vda2                vgsrv       lvm2 a--     8.78g 4.69g
[root@node2 ~]# lvcreate -l 100%free -n nfsdata /dev/havg
  Error locking on node node1.private.cluster20.example.com: Volume group for uuid not found: j6NzWCQCDIfr9oPaxFXYHYSNleMSGJNmEesWct32FM1W09X2qG15nq72RYfgm0Fe
  Failed to activate new LV.
[root@node2 ~]# lvs
  LV      VG          Attr   LSize    Origin Snap%  Move Log Copy%  Convert
  webdata clusteredvg -wi-ao 1020.00m                                      
  home    vgsrv       -wi-ao  256.00m                                      
  root    vgsrv       -wi-ao    3.31g                                      
  swap    vgsrv       -wi-ao  544.00m                                      
[root@node2 ~]# tail /var/log/messages
Jun 16 18:18:40 node2 modclusterd: startup succeeded
Jun 16 18:21:00 node2 ntpd[2299]: synchronized to 192.168.0.254, stratum 11
Jun 16 18:24:12 node2 kernel: sda: sda1 sda2
Jun 16 18:24:12 node2 kernel: sdb: sdb1 sdb2
Jun 16 18:24:12 node2 multipathd: dm-6: remove map (uevent)
Jun 16 18:24:12 node2 multipathd: dm-6: devmap not registered, can't remove
Jun 16 18:34:51 node2 multipathd: dm-7: remove map (uevent)
Jun 16 18:34:51 node2 multipathd: dm-7: devmap not registered, can't remove
Jun 16 18:34:51 node2 multipathd: dm-7: remove map (uevent)
Jun 16 18:34:51 node2 multipathd: dm-7: devmap not registered, can't remove
[root@node2 ~]# 


Cause:
I forgot to run partprobe on node1 before vgcreate.


Solution:
Please run below commands before vgcreate on every node shares the same sharestorage in cluster.
[root@node1 ~]# partprobe ; multipath -r


Before creating lvm, we must fistly recreate vg, since the old vg created before partprobe did not be recognized on other node.
[root@node2 ~]# vgremove /dev/havg
  Volume group "havg" successfully removed
[root@node2 ~]# vgcreate -cy havg /dev/mapper/netstoragep2 
  Clustered volume group "havg" successfully created
[root@node2 ~]# lvcreate -l 100%free -n nfsdata havg
  Logical volume "nfsdata" created
[root@node2 ~]# 




REF:
1. Re: [Linux-cluster] adding volume to cluster
http://www.redhat.com/archives/linux-cluster/2008-October/msg00019.html