使用cgroups来控制磁盘IO带宽
磨砺技术珠矶，践行数据之道，追求卓越价值
回到上一级页面：PostgreSQL内部结构与源代码研究索引页 回到顶级页面：PostgreSQL索引页
[作者 高健@博客园 [email protected]]
可资参考的资料：
http://kaivanov.blogspot.com/2012/07/setting-up-linux-cgroups-control-groups.html
首先用 hdparm测试磁盘读取的最大带宽:
hdparm --direct -t /dev/sda
测试结果为200MB/s以上。
然后给用户postgres设置参数，保持磁盘读写I/O为10MB/秒以下（此用户名下所有进程总和）。
# Configuration file generated by cgsnapshot mount { cpuset = /cgroup/cpuset; cpu = /cgroup/cpu; cpuacct = /cgroup/cpuacct; memory = /cgroup/memory; devices = /cgroup/devices; freezer = /cgroup/freezer; net_cls = /cgroup/net_cls; blkio = /cgroup/blkio; } group io-test { perm { task{ uid=postgres; gid=postgres; } admin{ uid=root; gid=root; } } blkio { blkio.throttle.write_iops_device=""; blkio.throttle.read_iops_device=""; blkio.throttle.write_bps_device="8:0 10485760"; blkio.throttle.read_bps_device="8:0 10485760"; blkio.reset_stats=""; blkio.weight="500"; blkio.weight_device=""; } } [postgres@cent6 Desktop]$
再看
[postgres@cent6 Desktop]$ cat /etc/cgrules.conf # /etc/cgrules.conf # #Each line describes a rule for a user in the forms: # #<user> <controllers> <destination> #<user>:<process name> <controllers> <destination> # #Where: # <user> can be: # - an user name # - a group name, with @group syntax # - the wildcard *, for any user or group. # - The %, which is equivalent to "ditto". This is useful for # multiline rules where different cgroups need to be specified # for various hierarchies for a single user. # # <process name> is optional and it can be: # - a process name # - a full command path of a process # # <controller> can be: # - comma separated controller names (no spaces) # - * (for all mounted controllers) # # <destination> can be: # - path with-in the controller hierarchy (ex. pgrp1/gid1/uid1) # # Note: # - It currently has rules based on uids, gids and process name. # # - Don't put overlapping rules. First rule which matches the criteria
# will be executed. # # - Multiline rules can be specified for specifying different cgroups # for multiple hierarchies. In the example below, user "peter" has # specified 2 line rule. First line says put peter's task in test1/
# dir for "cpu" controller and second line says put peter's tasks in
# test2/ dir for memory controller. Make a note of "%" sign in second line. # This is an indication that it is continuation of previous rule. # # #<user> <controllers> <destination> # #john cpu usergroup/faculty/john/ #john:cp cpu usergroup/faculty/john/cp #@student cpu,memory usergroup/student/ #peter cpu test1/ #% memory test2/ #@root * admingroup/ #* * default/ postgres blkio io-test/ # End of file # # # [postgres@cent6 Desktop]$
先用dd命令实际测试看看：
[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=4K count=1024 oflag=direct 1024+0 records in
1024+0 records out 4194304 bytes (4.2 MB) copied, 0.372953 s, 11.2 MB/s [postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=8K count=1024 oflag=direct 1024+0 records in
1024+0 records out 8388608 bytes (8.4 MB) copied, 0.733823 s, 11.4 MB/s [postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=8K count=1024 oflag=direct 1024+0 records in
1024+0 records out 8388608 bytes (8.4 MB) copied, 0.733256 s, 11.4 MB/s [postgres@cent6 Desktop]$ [postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile2 bs=16K count=1024 oflag=direct 1024+0 records in
1024+0 records out [postgres@cent6 Desktop]$
当然，如果dd执行时，开数据块太多，又是这种完全空的块，误差就会增大：
[postgres@cent6 Desktop]$ dd if=/dev/zero of=testfile1 bs=10240 count=300000
300000+0 records in
300000+0 records out 3072000000 bytes (3.1 GB) copied, 57.6779 s, 53.3 MB/s [postgres@cent6 Desktop]$
用实际的下载动作进行测试：
[postgres@cent6 Desktop]$ wget http://centos.arcticnetwork.ca/6.4/isos/x86_64/CentOS-6.4-x86_64-LiveCD.iso
再看看IO状态：
[postgres@cent6 Desktop]$ iostat -x 20 Linux 2.6.32-279.el6.x86_64 (cent6.gao) 09/10/2013 _x86_64_ (1 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 1.38 0.00 3.55 12.16 0.00 82.91 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 7.50 1747.20 7.77 24.88 535.75 14224.09 452.17 12.54 384.25 4.41 14.40 dm-0 0.00 0.00 13.79 1771.99 524.65 14224.07 8.26 1975.37 1106.16 0.09 16.73 dm-1 0.00 0.00 0.38 0.00 3.03 0.00 8.00 0.00 3.68 2.34 0.09 avg-cpu: %user %nice %system %iowait %steal %idle 3.65 0.00 2.14 0.73 0.00 93.48 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 1.82 1.09 1.56 49.64 25.44 28.24 0.03 11.78 3.57 0.95 dm-0 0.00 0.00 1.04 3.18 48.80 25.44 17.58 0.03 7.35 2.15 0.91 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 1.65 0.00 1.65 0.98 0.00 95.72 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util sda 0.00 60.28 0.00 1.44 0.00 492.12 341.14 0.02 11.79 8.86 1.28 dm-0 0.00 0.00 0.00 61.51 0.00 492.12 8.00 0.59 9.54 0.21 1.28 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
除了一开始的峰值，util长期在2%-3%之间，基本达到效果。
[作者 高健@博客园 [email protected]]
磨砺技术珠矶，践行数据之道，追求卓越价值
回到上一级页面：PostgreSQL内部结构与源代码研究索引页 回到顶级页面：PostgreSQL索引页