raid5 + lvm 实验

    今天的实验是raid5 + lvm，环境是centos6.0，VMware® Workstation 8.0

新添加3块5G的硬盘，sdbb,sdc,sdd组成raid5.

先格式化硬盘

[root@shanker ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xa3349e6d.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-652, default 1): 1
Last cylinder, +cylinders or +size{K,M,G} (1-652, default 652): //选择默认
Using default value 652

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): L

 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris
 1  FAT12           39  Plan 9          82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      3c  PartitionMagic  83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       40  Venix 80286     84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      41  PPC PReP Boot   85  Linux extended  c7  Syrinx
 5  Extended        42  SFS             86  NTFS volume set da  Non-FS data
 6  FAT16           4d  QNX4.x          87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS       4e  QNX4.x 2nd part 88  Linux plaintext de  Dell Utility
 8  AIX             4f  QNX4.x 3rd part 8e  Linux LVM       df  BootIt
 9  AIX bootable    50  OnTrack DM      93  Amoeba          e1  DOS access
 a  OS/2 Boot Manag 51  OnTrack DM6 Aux 94  Amoeba BBT      e3  DOS R/O
 b  W95 FAT32       52  CP/M            9f  BSD/OS          e4  SpeedStor
 c  W95 FAT32 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi eb  BeOS fs
 e  W95 FAT16 (LBA) 54  OnTrackDM6      a5  FreeBSD         ee  GPT
 f  W95 Ext'd (LBA) 55  EZ-Drive        a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            56  Golden Bow      a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    5c  Priam Edisk     a8  Darwin UFS      f1  SpeedStor
12  Compaq diagnost 61  SpeedStor       a9  NetBSD          f4  SpeedStor
14  Hidden FAT16 <3 63  GNU HURD or Sys ab  Darwin boot     f2  DOS secondary
16  Hidden FAT16    64  Novell Netware  af  HFS / HFS+      fb  VMware VMFS
17  Hidden HPFS/NTF 65  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
18  AST SmartSleep  70  DiskSecure Mult b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 75  PC/IX           bb  Boot Wizard hid fe  LANstep
1c  Hidden W95 FAT3 80  Old Minix       be  Solaris boot    ff  BBT
1e  Hidden W95 FAT1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

当recover这行数据到到100%的时候就代表同步结束

可以用mdadm --detail /dev/md0来查看刚才新建的设备状态

[root@shanker ~]# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Fri Mar 23 05:18:44 2012
     Raid Level : raid5
     Array Size : 10471424 (9.99 GiB 10.72 GB)
  Used Dev Size : 5235712 (4.99 GiB 5.36 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Fri Mar 23 05:20:54 2012
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : shanker:0  (local to host shanker)
           UUID : deb85091:1f514788:eafc2e11:99b0a7b2
         Events : 18

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       3       8       49        2      active sync   /dev/sdd1

然后配置raid的配置文件

[root@shanker ~]# echo device /dev/sdb1 /dev/sdc1 /dev/sdd1 > /etc/mdadm.conf
[root@shanker ~]# mdadm --detail --scan >> /etc/mdadm.conf

或者mdadm -D -s >> /etc/mdadm.conf

mdadm -S /dev/md0 //停止raid5

mdadm -As /dev/md0 //启动

然后我们来做LVM实验

[root@shanker ~]# pvcreate  /dev/md0
  Physical volume "/dev/md0" successfully created
[root@shanker ~]# vgcreate vg /dev/md0
  Volume group "vg" successfully created
[root@shanker ~]# pvscan
  PV /dev/md0   VG vg   lvm2 [9.98 GiB / 9.98 GiB free]
  Total: 1 [9.98 GiB] / in use: 1 [9.98 GiB] / in no VG: 0 [0   ]
[root@shanker ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "vg" using metadata type lvm2
[root@shanker ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md0
  VG Name               vg
  PV Size               9.99 GiB / not usable 2.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              2556
  Free PE               2556
  Allocated PE          0
  PV UUID               qWZXxe-DJ7H-K8xN-vPY2-uT7x-vt2M-yPNvhh

[root@shanker ~]# vgdisplay
  --- Volume group ---
  VG Name               vg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               9.98 GiB
  PE Size               4.00 MiB
  Total PE              2556
  Alloc PE / Size       0 / 0
  Free  PE / Size       2556 / 9.98 GiB
  VG UUID               DS0q5R-H00g-A5Au-6mHp-uNM5-5MkS-a1tDmq

然后创建lvm

[root@shanker ~]# lvcreate -L 1g -n lvm1 vg
  Logical volume "lvm1" created
[root@shanker ~]# lvcreate -L 500m -n lvm2 vg
  Logical volume "lvm2" created
[root@shanker ~]# lvscan
  ACTIVE            '/dev/vg/lvm1' [1.00 GiB] inherit
  ACTIVE            '/dev/vg/lvm2' [500.00 MiB] inherit
[root@shanker ~]# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg/lvm1
  VG Name                vg
  LV UUID                p1MvrS-Dxse-vVYb-uO6O-hhY5-tvKk-ONEu1Y
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                1.00 GiB
  Current LE             256
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/vg/lvm2
  VG Name                vg
  LV UUID                d0jopc-JGbI-Colb-sa0b-aUxf-zpqg-X6vCrf
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                500.00 MiB
  Current LE             125
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     4096
  Block device           253:1

[root@shanker ~]# mkfs.ext3 /dev/vg
vg/          vga_arbiter
[root@shanker ~]# mkfs.ext3 /dev/vg/lvm*
mke2fs 1.41.12 (17-May-2010)
mkfs.ext3: invalid blocks count - /dev/vg/lvm2
[root@shanker ~]# mkfs.ext3 /dev/vg/lvm1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=256 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@shanker ~]# mkfs.ext3 /dev/vg/lvm2
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=512 blocks, Stripe width=1024 blocks
128016 inodes, 512000 blocks
25600 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
63 block groups
8192 blocks per group, 8192 fragments per group
2032 inodes per group
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 30 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@shanker ~]#

mkdir /mnt/test{1..2}

mount /dev/vg/lvm1 /mnt/test1

mount /dev/vg/lvm2 /mnt/test2

然后vi /etc/fstab 将挂载点写入fstab，实现开机自动挂载

/dev/vg/lvm1/ /mnt/test1 ext3 defaults 0 0
/dev/vg/lvm2/ /mnt/test2 ext3 defaults 0 0

然后实验raid5设备有故障

[root@shanker tmp]# mdadm /dev/md0 -f /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0
[root@shanker tmp]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sdc1[1] sdb1[0](F)
      10471424 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]

有故障的设备有个(F)标志

删除故障盘

[root@shanker tmp]# mdadm /dev/md0 -r /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md0
[root@shanker tmp]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[3] sdc1[1]
      10471424 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]

此时pvdisplay 发现容量并没有减少，我们添加新硬盘/dev/sde然后格式化，添加到rai5，其实用之前移出的sdb盘也可以

[root@shanker tmp]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md0
  VG Name               vg
  PV Size               9.99 GiB / not usable 2.00 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              2556
  Free PE               2175
  Allocated PE          381
  PV UUID               qWZXxe-DJ7H-K8xN-vPY2-uT7x-vt2M-yPNvhh

[root@shanker tmp]# fdisk /dev/sde
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xd18a15b7.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-652, default 1): 1
Last cylinder, +cylinders or +size{K,M,G} (1-652, default 652):
Using default value 652

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@shanker tmp]# mdadm /dev/md0 -a /dev/sde1
mdadm: added /dev/sde1
[root@shanker tmp]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sde1[4] sdd1[3] sdc1[1]
      10471424 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
      [=>...................]  recovery =  6.6% (346112/5235712) finish=1.8min speed=43264K/sec

最后来做LVM扩容

[root@shanker tmp]# df -h /mnt/test1
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg-lvm1  1008M   34M  924M   4% /mnt/test1
[root@shanker tmp]# lvextend -L +1g /dev/vg
vg/          vga_arbiter
[root@shanker tmp]# lvextend -L +1g /dev/vg/lvm1
  Extending logical volume lvm1 to 2.00 GiB
  Logical volume lvm1 successfully resized
[root@shanker tmp]# resize
resize2fs   resizecons
[root@shanker tmp]# resize2fs /dev/vg/lvm1
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/vg/lvm1 is mounted on /mnt/test1; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/vg/lvm1 to 524288 (4k) blocks.
The filesystem on /dev/vg/lvm1 is now 524288 blocks long.

[root@shanker tmp]# df -h /mnt/test1/
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg-lvm1   2.0G   34M  1.9G   2% /mnt/test1

done～！