rh436 - 软件raid的配置(mdadm)
软件环境：
[root@station18 ~]# lsb_release -a
LSB Version: :core-3.1-ia32:core-3.1-noarch:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 5.4 (Tikanga)
Release: 5.4
Codename: Tikanga
先将sdb分成8个大小均为1G的分区备用：
fdisk /dev/sdb
partprobe
[root@station18 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 107.3 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 123 987966 83 Linux
/dev/sdb2 124 246 987997+ 83 Linux
/dev/sdb3 247 369 987997+ 83 Linux
/dev/sdb4 370 13054 101892262+ 5 Extended
/dev/sdb5 370 492 987966 83 Linux
/dev/sdb6 493 615 987966 83 Linux
/dev/sdb7 616 738 987966 83 Linux
/dev/sdb8 739 861 987966 83 Linux
/dev/sdb9 862 984 987966 83 Linux
创建raid设备md1，raid级别为5，由3块镜像盘和一块热备盘组成，可用容量为2G：
mdadm: array /dev/md1 started.
[root@station18 ~]# mdadm -D /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Tue Jul 3 00:53:00 2012
Raid Level : raid5
Array Size : 1975680 (1929.70 MiB 2023.10 MB)
Used Dev Size : 987840 (964.85 MiB 1011.55 MB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Tue Jul 3 00:53:05 2012
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
UUID : 267059c1:45b5e0d0:84d94faf:cc7eefaa
Events : 0.2
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 18 1 active sync /dev/sdb2
2 8 19 2 active sync /dev/sdb3
3 8 21 - spare /dev/sdb5
[root@station18 ~]#
mdadm -D实际上是通过查询/sys/block/mdX/md/得出raid信息的：
[root@station18 ~]# cat /sys/block/md1/md/
array_state metadata_version stripe_cache_size
bitmap_set_bits mismatch_cnt suspend_hi
chunk_size new_dev suspend_lo
component_size raid_disks sync_action
dev-sdb1/ rd0/ sync_completed
dev-sdb2/ rd1/ sync_speed
dev-sdb3/ rd2/ sync_speed_max
dev-sdb5/ resync_start sync_speed_min
layout safe_mode_delay
level stripe_cache_active
[root@station18 ~]# cat /sys/block/md1/md/level
raid5
[root@station18 ~]#
通过mdadm的配置文件/etc/mdadm.conf可以保存当前raid的状态，方便停止后恢复：
[root@station18 ~]# ll /etc/mdadm.conf
ls: /etc/mdadm.conf: No such file or directory
[root@station18 ~]# mdadm -D -s > /etc/mdadm.conf
[root@station18 ~]# cat /etc/mdadm.conf
ARRAY /dev/md1 level=raid5 num-devices=3 metadata=0.90 spares=1 UUID=267059c1:45b5e0d0:84d94faf:cc7eefaa
[root@station18 ~]# mv /etc/mdadm.conf{,.orig}
[root@station18 ~]# ll /etc/mdadm*
-rw-r--r-- 1 root root 105 Jul 3 00:56 /etc/mdadm.conf.orig
[root@station18 ~]# mdadm --stop /dev/md1
mdadm: stopped /dev/md1
[root@station18 ~]# mdadm --assemble --scan
mdadm: No arrays found in config file
[root@station18 ~]# cp /etc/mdadm.conf.orig /etc/mdadm.conf
[root@station18 ~]# mdadm --assemble --scan
mdadm: /dev/md1 has been started with 3 drives and 1 spare.
即便没有配置文件，如果记得当时raid的有哪些磁盘组成，也可在stop后恢复：
[root@station18 ~]# rm /etc/mdadm.conf -f
[root@station18 ~]# mdadm --stop /dev/md1
mdadm: stopped /dev/md1
[root@station18 ~]# mdadm -D /dev/md1
mdadm: md device /dev/md1 does not appear to be active.
[root@station18 ~]# mdadm --assemble /dev/md1 /dev/sdb{1,2,3,5}
mdadm: /dev/md1 has been started with 3 drives and 1 spare.
[root@station18 ~]# mdadm -D /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Tue Jul 3 00:53:00 2012
Raid Level : raid5
Array Size : 1975680 (1929.70 MiB 2023.10 MB)
Used Dev Size : 987840 (964.85 MiB 1011.55 MB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Tue Jul 3 00:53:05 2012
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
UUID : 267059c1:45b5e0d0:84d94faf:cc7eefaa
Events : 0.2
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 18 1 active sync /dev/sdb2
2 8 19 2 active sync /dev/sdb3
3 8 21 - spare /dev/sdb5
[root@station18 ~]#
监视raid的异常并通过MAILADDR发送邮件：
[root@station18 ~]# cp /etc/mdadm.conf.orig /etc/mdadm.conf
[root@station18 ~]# vim /etc/mdadm.conf
ARRAY /dev/md1 level=raid5 num-devices=3 metadata=0.90 spares=1 UUID=267059c1:45b5e0d0:84d94faf:cc7eefaa
MAILADDR root@localhost
[root@station18 ~]# cp /etc/mdadm.conf.orig /etc/mdadm.conf
[root@station18 ~]# vim /etc/mdadm.conf
[root@station18 ~]# service sendmail status
sendmail (pid 3471) is running...
[root@station18 ~]# service mdmonitor restart
Starting mdmonitor: [ OK ]
[root@station18 ~]# chkconfig mdmonitor on
[root@station18 ~]# mail
No mail for root
[root@station18 ~]# mail
No mail for root
[root@station18 ~]# mdadm /dev/md1 -f /dev/sdb3 #模拟sdb3损坏
mdadm: set /dev/sdb3 faulty in /dev/md1
[root@station18 ~]# mdadm -D /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Tue Jul 3 00:53:00 2012
Raid Level : raid5
Array Size : 1975680 (1929.70 MiB 2023.10 MB)
Used Dev Size : 987840 (964.85 MiB 1011.55 MB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Tue Jul 3 01:15:04 2012
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
UUID : 267059c1:45b5e0d0:84d94faf:cc7eefaa
Events : 0.6
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 18 1 active sync /dev/sdb2
2 8 21 2 active sync /dev/sdb5
3 8 19 - faulty spare /dev/sdb3
[root@station18 ~]# mail
Mail version 8.1 6/6/93. Type ? for help.
"/var/spool/mail/root": 1 message 1 new
>N 1 [email protected] Tue Jul 3 01:14 31/1161 "Fail event on /dev/md"
& 1
Message 1:
From [email protected] Tue Jul 3 01:14:59 2012
Date: Tue, 3 Jul 2012 01:14:59 +0800
From: mdadm monitoring <[email protected]>
To: [email protected]
Subject: Fail event on /dev/md1:station18.example.com
This is an automatically generated mail message from mdadm
running on station18.example.com
A Fail event had been detected on md device /dev/md1.
It could be related to component device /dev/sdb3.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid6] [raid5] [raid4]
md1 : active raid5 sdb1[0] sdb5[3](S) sdb3[2](F) sdb2[1]
1975680 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]
unused devices: <none>
移除已损坏的raid磁盘成员：
[root@station18 ~]# mdadm /dev/md1 -r /dev/sdb3
mdadm: hot removed /dev/sdb3
[root@station18 ~]# mdadm -D -s
ARRAY /dev/md1 level=raid5 num-devices=3 metadata=0.90 UUID=267059c1:45b5e0d0:84d94faf:cc7eefaa
[root@station18 ~]# mdadm -D /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Tue Jul 3 00:53:00 2012
Raid Level : raid5
Array Size : 1975680 (1929.70 MiB 2023.10 MB)
Used Dev Size : 987840 (964.85 MiB 1011.55 MB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Tue Jul 3 01:16:44 2012
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
UUID : 267059c1:45b5e0d0:84d94faf:cc7eefaa
Events : 0.8
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 18 1 active sync /dev/sdb2
2 8 21 2 active sync /dev/sdb5
[root@station18 ~]#
向raid中增加新的磁盘成员：
[root@station18 ~]# mdadm /dev/md1 -a /dev/sdb3 #这里假设sdb3是块好盘
mdadm: added /dev/sdb3
[root@station18 ~]# mdadm -D /dev/sdb
sdb sdb1 sdb2 sdb3 sdb4 sdb5 sdb6 sdb7 sdb8 sdb9
[root@station18 ~]# mdadm -D /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Tue Jul 3 00:53:00 2012
Raid Level : raid5
Array Size : 1975680 (1929.70 MiB 2023.10 MB)
Used Dev Size : 987840 (964.85 MiB 1011.55 MB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Tue Jul 3 01:16:44 2012
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
UUID : 267059c1:45b5e0d0:84d94faf:cc7eefaa
Events : 0.8
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 18 1 active sync /dev/sdb2
2 8 21 2 active sync /dev/sdb5
3 8 19 - spare /dev/sdb3
[root@station18 ~]#
在线拉伸raid的大小：
[root@station18 ~]# mkfs.ext3 /dev/md1
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
247296 inodes, 493920 blocks
24696 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=507510784
16 block groups
32768 blocks per group, 32768 fragments per group
15456 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 26 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@station18 ~]# mkdir /mnt/md1
[root@station18 ~]# mount /dev/md1 /mnt/md1/
[root@station18 ~]# cp /boot/* /mnt/md1/
cp: omitting directory `/boot/grub'
cp: omitting directory `/boot/lost+found'
[root@station18 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vol0-root
7.8G 2.4G 5.0G 33% /
/dev/mapper/vol0-home
465M 11M 431M 3% /home
/dev/sda1 99M 28M 66M 30% /boot
tmpfs 885M 0 885M 0% /dev/shm
none 885M 208K 885M 1% /var/lib/xenstored
/dev/md1 1.9G 57M 1.8G 4% /mnt/md1
[root@station18 ~]#
[root@station18 ~]# mdadm /dev/md1 -a /dev/sdb{6,7} #为md1增加磁盘
mdadm: added /dev/sdb6
mdadm: added /dev/sdb7
[root@station18 ~]# mdadm -D /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Tue Jul 3 00:53:00 2012
Raid Level : raid5
Array Size : 1975680 (1929.70 MiB 2023.10 MB)
Used Dev Size : 987840 (964.85 MiB 1011.55 MB)
Raid Devices : 3
Total Devices : 6
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Tue Jul 3 01:20:24 2012
State : clean
Active Devices : 3
Working Devices : 6
Failed Devices : 0
Spare Devices : 3
Layout : left-symmetric
Chunk Size : 64K
UUID : 267059c1:45b5e0d0:84d94faf:cc7eefaa
Events : 0.8
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 18 1 active sync /dev/sdb2
2 8 21 2 active sync /dev/sdb5
3 8 19 - spare /dev/sdb3
4 8 22 - spare /dev/sdb6
5 8 23 - spare /dev/sdb7
[root@station18 ~]# mdadm --grow /dev/md1 --raid-devices=5 --backup-file=/tmp/md1.bu #把sdb6,sdb7由热备盘加到镜像盘，增加前先备份当前配置到/tmp/md1.bu，如果奇偶校验过程中失败，可通过该文件进行恢复（mdadm --assemble /dev/md1 --backup-file=/tmp/md1.bu），如果没有遇到错误，该文件会在完成后被自动删除。
mdadm: Need to backup 256K of critical section..
mdadm: ... critical section passed.
[root@station18 ~]# ll /tmp/md1.bu
ls: /tmp/md1.bu: No such file or directory
[root@station18 ~]# mdadm -D /dev/md1
/dev/md1:
Version : 0.91
Creation Time : Tue Jul 3 00:53:00 2012
Raid Level : raid5
Array Size : 1975680 (1929.70 MiB 2023.10 MB)
Used Dev Size : 987840 (964.85 MiB 1011.55 MB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Tue Jul 3 01:23:18 2012
State : clean, recovering
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
Reshape Status : 83% complete
Delta Devices : 2, (3->5)
UUID : 267059c1:45b5e0d0:84d94faf:cc7eefaa
Events : 0.574
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 18 1 active sync /dev/sdb2
2 8 21 2 active sync /dev/sdb5
3 8 23 3 active sync /dev/sdb7
4 8 22 4 active sync /dev/sdb6
5 8 19 - spare /dev/sdb3
[root@station18 ~]#
[root@station18 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vol0-root
7.8G 2.4G 5.0G 33% /
/dev/mapper/vol0-home
465M 11M 431M 3% /home
/dev/sda1 99M 28M 66M 30% /boot
tmpfs 885M 0 885M 0% /dev/shm
none 885M 208K 885M 1% /var/lib/xenstored
/dev/md1 1.9G 57M 1.8G 4% /mnt/md1
[root@station18 ~]# resize2fs /dev/md1 ＃拉伸文件系统
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/md1 is mounted on /mnt/md1; on-line resizing required
Performing an on-line resize of /dev/md1 to 987840 (4k) blocks.
The filesystem on /dev/md1 is now 987840 blocks long.
[root@station18 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vol0-root
7.8G 2.4G 5.0G 33% /
/dev/mapper/vol0-home
465M 11M 431M 3% /home
/dev/sda1 99M 28M 66M 30% /boot
tmpfs 885M 0 885M 0% /dev/shm
none 885M 208K 885M 1% /var/lib/xenstored
/dev/md1 3.8G 58M 3.5G 2% /mnt/md1
[root@station18 ~]#
两个raid磁盘组（/dev/md1，/dev/md2）共用热备盘：
[root@station18 ~]# mdadm -C /dev/md2 -a yes -l 1 -n 2 /dev/sdb{8,9} ＃创建一个新的raid，md2，初始没有热备盘
mdadm: array /dev/md2 started.
[root@station18 ~]# mdadm -D /dev/md2
/dev/md2:
Version : 0.90
Creation Time : Tue Jul 3 01:26:00 2012
Raid Level : raid1
Array Size : 987840 (964.85 MiB 1011.55 MB)
Used Dev Size : 987840 (964.85 MiB 1011.55 MB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 2
Persistence : Superblock is persistent
Update Time : Tue Jul 3 01:26:05 2012
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
UUID : 99492308:7c84fa1e:7ad03c15:73b342db
Events : 0.2
Number Major Minor RaidDevice State
0 8 24 0 active sync /dev/sdb8
1 8 25 1 active sync /dev/sdb9
[root@station18 ~]#
[root@station18 ~]# cat /etc/mdadm.conf
ARRAY /dev/md1 level=raid5 num-devices=3 metadata=0.90 spares=1 UUID=267059c1:45b5e0d0:84d94faf:cc7eefaa
MAILADDR root@localhost
[root@station18 ~]# mdadm -D -s >> /etc/mdadm.conf
[root@station18 ~]# vim /etc/mdadm.conf ＃在ARRAY后增加spare-group=share1以共享热备盘
ARRAY /dev/md1 level=raid5 num-devices=3 metadata=0.90 spares=1 UUID=267059c1:45b5e0d0:84d94faf:cc7eefaa spare-group=share1
MAILADDR root@localhost
ARRAY /dev/md1 level=raid5 num-devices=5 metadata=0.90 spares=1 UUID=267059c1:45b5e0d0:84d94faf:cc7eefaa
ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=99492308:7c84fa1e:7ad03c15:73b342db spare-group=share1
[root@station18 ~]# mdadm -D /dev/md1
/dev/md1:
Version : 0.90
Creation Time : Tue Jul 3 00:53:00 2012
Raid Level : raid5
Array Size : 3951360 (3.77 GiB 4.05 GB)
Used Dev Size : 987840 (964.85 MiB 1011.55 MB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 1
Persistence : Superblock is persistent
Update Time : Tue Jul 3 01:25:00 2012
State : clean
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
UUID : 267059c1:45b5e0d0:84d94faf:cc7eefaa
Events : 0.684
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 18 1 active sync /dev/sdb2
2 8 21 2 active sync /dev/sdb5
3 8 23 3 active sync /dev/sdb7
4 8 22 4 active sync /dev/sdb6
5 8 19 - spare /dev/sdb3
[root@station18 ~]# mdadm -D /dev/md2
/dev/md2:
Version : 0.90
Creation Time : Tue Jul 3 01:26:00 2012
Raid Level : raid1
Array Size : 987840 (964.85 MiB 1011.55 MB)
Used Dev Size : 987840 (964.85 MiB 1011.55 MB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 2
Persistence : Superblock is persistent
Update Time : Tue Jul 3 01:26:05 2012
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
UUID : 99492308:7c84fa1e:7ad03c15:73b342db
Events : 0.2
Number Major Minor RaidDevice State
0 8 24 0 active sync /dev/sdb8
1 8 25 1 active sync /dev/sdb9
[root@station18 ~]# mdadm /dev/md2 -f /dev/sdb8 ＃模拟md2中的sdb8损坏，会发现md1的热备盘自动顶上
mdadm: set /dev/sdb8 faulty in /dev/md2
[root@station18 ~]# mdadm -D /dev/md2
/dev/md2:
Version : 0.90
Creation Time : Tue Jul 3 01:26:00 2012
Raid Level : raid1
Array Size : 987840 (964.85 MiB 1011.55 MB)
Used Dev Size : 987840 (964.85 MiB 1011.55 MB)
Raid Devices : 2
Total Devices : 3
Preferred Minor : 2
Persistence : Superblock is persistent
Update Time : Tue Jul 3 01:36:23 2012
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 1
Spare Devices : 0
UUID : 99492308:7c84fa1e:7ad03c15:73b342db
Events : 0.8
Number Major Minor RaidDevice State
0 8 19 0 active sync /dev/sdb3
1 8 25 1 active sync /dev/sdb9
2 8 24 - faulty spare /dev/sdb8
[root@station18 ~]#