工作中遇到一个性能瓶颈问题,发现KVM中环境Guest为RHEL5.5时,IO性能比RHEL6.3差很多,于是着手研究性能差距的原因。由于操作系统版本不一样,所以重点考虑的是内核与文件系统,通过升级内核对比试验,发现性能差距不大,但是ext3升级为ext4的时候性能明显提升很多。
考虑到qcow2文件格式可能对性能的影响,所以分别额外采用了raw格式做对比试验。下面是测试结果对比:
RAW + ext3:
[root@localhost ~]# iozone -a -i0 -i1 -s6G -r64k Iozone: Performance Test of File I/O Version $Revision: 3.420 $ Compiled for 64 bit mode. Build: linux-ia64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, Vangel Bojaxhi, Ben England, Vikentsi Lapa. Run began: Fri Apr 25 11:42:43 2014 Auto Mode File size set to 6291456 KB Record Size 64 KB Command line used: iozone -a -i0 -i1 -s6G -r64k Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 6291456 64 422636 286223 352009 369563 iozone test complete. [root@localhost ~]# mount -l /dev/mapper/rootvg-rootlv on / type ext3 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/vda1 on /boot type ext3 (rw) [/boot1] tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
[root@localhost ~]# iozone -a -i0 -i1 -s6G -r64k Iozone: Performance Test of File I/O Version $Revision: 3.420 $ Compiled for 64 bit mode. Build: linux-ia64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, Vangel Bojaxhi, Ben England, Vikentsi Lapa. Run began: Fri Apr 25 11:46:51 2014 Auto Mode File size set to 6291456 KB Record Size 64 KB Command line used: iozone -a -i0 -i1 -s6G -r64k Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 6291456 64 674409 823374 443455 443929 iozone test complete. [root@localhost ~]# mount -l /dev/mapper/rootvg-rootlv on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/vda1 on /boot type ext4 (rw) [/boot1] tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
针对qcow2格式测试结果如下:(为了尽量环境公平,cp a.qcow2 -> b.qcow2 ; b.qcow2 升级到ext4,然后再做比对试验)
qcow2 + ext3:
Run began: Fri Apr 25 16:05:48 2014 Auto Mode File size set to 6291456 KB Record Size 64 KB Command line used: iozone -a -i0 -i1 -s6G -r64k Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 6291456 64 518274 392571 262093 268514 iozone test complete.
Auto Mode File size set to 2097152 KB Record Size 64 KB Command line used: iozone -a -i0 -i1 -s2G -r64k Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 2097152 64 633332 776716 312232 278960 iozone test complete. [root@localhost ~]# mount -l /dev/mapper/rootvg-rootlv on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/vda1 on /boot type ext4 (rw) [/boot1] tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
之后面临的问题就是如何将线上的rhel5.5无损的从ext3升级到ext4。 我拿了一台虚拟机做实验,步骤如下:
1. 检查所需要的软件包,尽量为最新版本:
yum install -y upgrade mkinitrd e2fsprogs yum update -y upgrade mkinitrd e2fsprogs
2. 备份fstab,以防万一
cp /etc/fstab /etc/fstab.bak
sed -i "s/ext3/ext4/g" /etc/fstab
4. 备份系统initrd,重新生成新的。因为centos5.5默认不加载ext4,所以要主动 --with=ext4
mv /boot/initrd-`uname -r`.img /boot/initrd-`uname -r`.img.bak mkinitrd -v --with=ext4 /boot/initrd-`uname -r`.img `uname -r`
[root@localhost ~]# grep -v -w / /etc/fstab |grep ext4|awk '{print $2}' /boot /home
[root@localhost ~]# df -h|egrep "boot|home" /dev/sda1 485M 67M 393M 15% /boot /dev/mapper/VolGroup-lv_home 394G 23G 352G 6% /home
umount /home
umount /boot
tune4fs -O extents,uninit_bg,dir_index,flex_bg -m 1 /dev/mapper/VolGroup-lv_home
tune4fs -O extents,uninit_bg,dir_index,flex_bg -m 1 /dev/sda1
8. 检查文件系统是否有错误,自动修复
fsck -vfyDC0 /dev/sda1 fsck -vfyDC0 /dev/mapper/VOlGroup-lv_home
[root@localhost ~]# mount -l /dev/mapper/rootvg-rootlv on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/vda1 on /boot type ext4 (rw) [/boot1] tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)