KVM虚拟机CentOS5.5 升级ext3->ext4

工作中遇到一个性能瓶颈问题，发现KVM中环境Guest为RHEL5.5时，IO性能比RHEL6.3差很多，于是着手研究性能差距的原因。由于操作系统版本不一样，所以重点考虑的是内核与文件系统，通过升级内核对比试验，发现性能差距不大，但是ext3升级为ext4的时候性能明显提升很多。

考虑到qcow2文件格式可能对性能的影响，所以分别额外采用了raw格式做对比试验。下面是测试结果对比：

RAW + ext3：

[root@localhost ~]# iozone -a -i0 -i1 -s6G -r64k
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.420 $
		Compiled for 64 bit mode.
		Build: linux-ia64 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
	             Vangel Bojaxhi, Ben England, Vikentsi Lapa.

	Run began: Fri Apr 25 11:42:43 2014

	Auto Mode
	File size set to 6291456 KB
	Record Size 64 KB
	Command line used: iozone -a -i0 -i1 -s6G -r64k
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         6291456      64  422636  286223   352009   369563                                                                          

iozone test complete.
[root@localhost ~]# mount -l
/dev/mapper/rootvg-rootlv on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/vda1 on /boot type ext3 (rw) [/boot1]
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

RAW + ext4：

[root@localhost ~]# iozone -a -i0 -i1 -s6G -r64k
	Iozone: Performance Test of File I/O
	        Version $Revision: 3.420 $
		Compiled for 64 bit mode.
		Build: linux-ia64 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
	             Vangel Bojaxhi, Ben England, Vikentsi Lapa.

	Run began: Fri Apr 25 11:46:51 2014

	Auto Mode
	File size set to 6291456 KB
	Record Size 64 KB
	Command line used: iozone -a -i0 -i1 -s6G -r64k
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
         6291456      64  674409  823374   443455   443929                                                                          

iozone test complete.
[root@localhost ~]# mount -l
/dev/mapper/rootvg-rootlv on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/vda1 on /boot type ext4 (rw) [/boot1]
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

针对qcow2格式测试结果如下：（为了尽量环境公平，cp a.qcow2 -> b.qcow2 ; b.qcow2 升级到ext4，然后再做比对试验）

qcow2 + ext3:

Run began: Fri Apr 25 16:05:48 2014
 Auto Mode
 File size set to 6291456 KB
 Record Size 64 KB
 Command line used: iozone -a -i0 -i1 -s6G -r64k
 Output is in Kbytes/sec
 Time Resolution = 0.000001 seconds.
 Processor cache size set to 1024 Kbytes.
 Processor cache line size set to 32 bytes.
 File stride size set to 17 * record size.
                                                            random random bkwd record stride
              KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
         6291456 64 518274 392571 262093 268514
iozone test complete.

qcow2 + ext4 ：

Auto Mode
 File size set to 2097152 KB
 Record Size 64 KB
 Command line used: iozone -a -i0 -i1 -s2G -r64k
 Output is in Kbytes/sec
 Time Resolution = 0.000001 seconds.
 Processor cache size set to 1024 Kbytes.
 Processor cache line size set to 32 bytes.
 File stride size set to 17 * record size.
                                                            random random bkwd record stride
              KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread
         2097152 64 633332 776716 312232 278960
iozone test complete.
[root@localhost ~]# mount -l
/dev/mapper/rootvg-rootlv on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/vda1 on /boot type ext4 (rw) [/boot1]
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

从对比可以发现，不管哪种文件格式，读写性能ext4都比ext3有很大幅度的提升。

之后面临的问题就是如何将线上的rhel5.5无损的从ext3升级到ext4。 我拿了一台虚拟机做实验，步骤如下：

1. 检查所需要的软件包，尽量为最新版本：

yum install -y upgrade mkinitrd e2fsprogs 
yum update -y upgrade mkinitrd e2fsprogs

2. 备份fstab，以防万一

cp /etc/fstab /etc/fstab.bak

3.  将ext3全部换成ext4

sed -i "s/ext3/ext4/g" /etc/fstab

4. 备份系统initrd，重新生成新的。因为centos5.5默认不加载ext4，所以要主动 --with=ext4

mv /boot/initrd-`uname -r`.img /boot/initrd-`uname -r`.img.bak

mkinitrd -v --with=ext4 /boot/initrd-`uname -r`.img `uname -r`

5. 拿到非跟文件系统的ext4分区：

[root@localhost ~]# grep  -v -w / /etc/fstab |grep ext4|awk '{print $2}'
/boot
/home

6. 查到5中对应的设备：

[root@localhost ~]# df -h|egrep "boot|home"
/dev/sda1             485M   67M  393M  15% /boot
/dev/mapper/VolGroup-lv_home
                      394G   23G  352G   6% /home

umount /home

umount /boot

7. 增加ext4的特性支持：

tune4fs -O extents,uninit_bg,dir_index,flex_bg -m 1 /dev/mapper/VolGroup-lv_home
tune4fs -O extents,uninit_bg,dir_index,flex_bg -m 1 /dev/sda1

8. 检查文件系统是否有错误，自动修复

fsck -vfyDC0 /dev/sda1
fsck -vfyDC0 /dev/mapper/VOlGroup-lv_home

9, 重启系统，检查是否ext4已经生效

[root@localhost ~]# mount -l
/dev/mapper/rootvg-rootlv on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/vda1 on /boot type ext4 (rw) [/boot1]
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)