Linux HugePage for MySQL Server(MySQL中大页的使用)
网上大页的资料抄来抄去, 没一个是正确的 :shock: . 本人经过查Linux说明和实践, 现将线上的配置总结一下. :lol:
[b]效果对比[/b]
未配置HugePage:
[img]http://dl2.iteye.com/upload/attachment/0097/8982/fa99d7fc-9610-3efe-aec8-b518b1b71433.png[/img]
配置HugePage后 (并加多客户端个数向MySQL施压):
[img]http://dl2.iteye.com/upload/attachment/0097/8984/c4d4661d-727e-3cde-8d0a-17853adb7a47.png[/img]
总结: 从对比数据看出来,开启HugePage后,QPS 2~3倍,提升下, UserCPU状态只是多消耗0.2~0.5而已. Load低成3~5倍. 长短时间的运行,表情都很稳定. 大幅度减少重启后MySQL预热时间(Hit列提高几个百分点).
[b]原理资料[/b]
大家知道,OS HugePage开启后,内存置换大幅度减少.这没什么高深的技术难度,但是很考验原理的理解和配置项的细心程度.
根据: linux kernel描述 [i][url]https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt[/url][/i]
[list]
[*][b]HugePages_Total[/b] is the size of the pool of huge pages.
[*][b]HugePages_Free[/b] is the number of huge pages in the pool that are not yet
allocated.
[*][b]HugePages_Rsvd[/b] is short for "reserved," and is the number of huge pages for
which a commitment to allocate from the pool has been made,
but no allocation has yet been made. Reserved huge pages
guarantee that an application will be able to allocate a
huge page from the pool of huge pages at fault time.
[*][b]HugePages_Surp[/b] is short for "surplus," and is the number of huge pages in
the pool above the value in /proc/sys/vm/nr_hugepages. The
maximum number of surplus huge pages is controlled by
/proc/sys/vm/nr_overcommit_hugepages.
[/list]
HugePages_Total , HugePages_Free , HugePages_Rsvd , HugePages_Surp 都是页的个数. 其中每页的大小和Linux Kernel密切相关. 例如: centos 6 x64上是2MB 一个HugePage.
HugePages_Rsvd 上面说明是保留的页数.
内存页置换原理: [i][url]http://www.ibm.com/developerworks/cn/linux/l-cn-hugetlb[/url][/i]
Kernel的内存页: [i][url]https://www.kernel.org/doc/Documentation/sysctl/kernel.txt[/url][/i]
MySQL使用HugePage: [i][url]http://dev.mysql.com/doc/refman/5.5/en/large-page-support.html[/url][/i] [i][url]http://dev.mysql.com/doc/refman/5.6/en/large-page-support.html[/url][/i]
我们在Linux上,使用命令查看相关的Huge Page在OS上的参数.
cat /proc/meminfo | fgrep -i Huge
AnonHugePages: 5775360 kB
HugePages_Total: 47872
HugePages_Free: 4304
HugePages_Rsvd: 361
HugePages_Surp: 0
Hugepagesize: 2048 kB
[b]环境[/b]
前面属述的效果图相关环境如下:
CentOS release 6.5 (Final)
Linux 2.6 x86_64
MySQL 5.5 (5.6也有一样有2~3倍提升)
[b]配置[/b]
Linux OS
[list]
[*]vm.nr_hugepages = 47872 #开启47872个大页. 相当于
[*]vm.hugetlb_shm_group = 499 #可以使用的HugePage,Linux进程相关的用户组ID
[*]kernel.shmmax = 94489280512 # Controls the maximum shared segment size, in bytes #单个进程下最大能使用的HugePage个数
[*]kernel.shmall = 230686720 # Controls the maximum number of shared memory segments, in pages. default page size : 4KB #Linux System整个所能使用的个数
[/list]
我喜欢一次性把几个参数在文件中配置完. 记住用命令 sysctl -p生效之.
MySQL --defaults-file 中开启
[mysqld]
large-pages
[b]参数大小计算说明和示例[/b]
*numastat 可以看到当前系统下numa状态.
[img]http://dl2.iteye.com/upload/attachment/0097/8986/90210b03-e275-3605-8d43-77f5c725a518.png[/img]
*注意MySQL manual提示 : For MySQL usage, you normally want the value of shmmax to be close to the value of shmall. 我们尽量的把shmmax -> 接近 shmall的值.
[list]
[*][b]1st step[/b]确定机器的CPU processors(指我们在Linux命令看到的个数. cat /proc/cpuinfo | fgrep -i 'processor' | wc -l )
[*][b]2nd step[/b]确定MySQL每个connection所消耗的内存, 我们可以使用mysqltuner.pl 查看到 [i][url]http://mysqltuner.com/[/url][/i]
[/list]
网上很多的资料说BufferPool 配置到80%的物理内存, 是有道理的, 同时这是有前提的max_connections是很少的.
总的使用内存不能超过物理内存的90%. (这个原理需要单独写篇BLOG)
我的配置示例是:
CPU 32 processors .
选择 mysql my.cnf
max_connections = 3000 # 约47GB
innodb_buffer_pool_size = 84480M
innodb_buffer_pool_instances = 30 #每个instance是2.75GB. 因为每个在2.5GB~3GB是很高效的, 体现在MySQL RT上. 而30(CPU 32)这个数量容易计算.留一两个processors给系统. 而[b]每个实例会多占用近100MB的HugePage[/b]
加上其他的conns消耗:
Total buffers: 83.0G global + 16.2M per thread (3000 max threads)
[OK] Maximum possible memory usage: 130.7G (82% of installed RAM)
这样30+4(留足一定的Huge Pages给MySQL和系统其他用).
vm.nr_hugepages = 47872 ( HugePages_Total : 47872 34*2.75GB / 2MB ) <<< 物理内存的80%.
kernel.shmmax = 94489280512 ( 32*2.75GB = 94489280512 bytes )
kernel.shmall = 230686720 ( 32*2.75 GB / 4KB = 230686720 ) 其中: 4KB 是LinuxKernelPage基本大小.
--
如果不留足vm.nr_hugepages , MySQL启动会报错误,启动不了!
看看MySQL 占用效果:
ipcs | fgrep mysql
[img]http://dl2.iteye.com/upload/attachment/0097/9028/20fe10f1-4833-3237-8e56-a85d258eb77c.png[/img]
使用top (请忽略RES). [b]关注 VIRT一项[/b].
[img]http://dl2.iteye.com/upload/attachment/0097/9030/b3da134a-a23f-3353-883e-6a2dda182207.png[/img]
----
[list]
[*]MySQL 最好根据实际需要配置上大的内存的.(反正现在内存很便宜的.)
[*]如果是一机器上的单实例, 直接关闭 numa. 或是设置kernerl启动项 在/etc/grub.conf , numa=off
[*]关闭swap. swapoff -a 执行. 或是在 /etc/rc.local 追加swapoff -a.
[/list]
--------
最后我们可以按照MySQL defaults-file和机器的物理内存大小配置线上的HugePage参数.
我们只是配置正确,不是在"调优". <高性能MySQL>提示.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
有应聘"MySQL/MariaDB,运维/开发", 请发邮件至: [email protected], 并取得联系.
工程师招聘: [url]http://job.youzan.com[/url] 请联系我: [email protected] .
[b]效果对比[/b]
未配置HugePage:
[img]http://dl2.iteye.com/upload/attachment/0097/8982/fa99d7fc-9610-3efe-aec8-b518b1b71433.png[/img]
配置HugePage后 (并加多客户端个数向MySQL施压):
[img]http://dl2.iteye.com/upload/attachment/0097/8984/c4d4661d-727e-3cde-8d0a-17853adb7a47.png[/img]
总结: 从对比数据看出来,开启HugePage后,QPS 2~3倍,提升下, UserCPU状态只是多消耗0.2~0.5而已. Load低成3~5倍. 长短时间的运行,表情都很稳定. 大幅度减少重启后MySQL预热时间(Hit列提高几个百分点).
[b]原理资料[/b]
大家知道,OS HugePage开启后,内存置换大幅度减少.这没什么高深的技术难度,但是很考验原理的理解和配置项的细心程度.
根据: linux kernel描述 [i][url]https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt[/url][/i]
[list]
[*][b]HugePages_Total[/b] is the size of the pool of huge pages.
[*][b]HugePages_Free[/b] is the number of huge pages in the pool that are not yet
allocated.
[*][b]HugePages_Rsvd[/b] is short for "reserved," and is the number of huge pages for
which a commitment to allocate from the pool has been made,
but no allocation has yet been made. Reserved huge pages
guarantee that an application will be able to allocate a
huge page from the pool of huge pages at fault time.
[*][b]HugePages_Surp[/b] is short for "surplus," and is the number of huge pages in
the pool above the value in /proc/sys/vm/nr_hugepages. The
maximum number of surplus huge pages is controlled by
/proc/sys/vm/nr_overcommit_hugepages.
[/list]
HugePages_Total , HugePages_Free , HugePages_Rsvd , HugePages_Surp 都是页的个数. 其中每页的大小和Linux Kernel密切相关. 例如: centos 6 x64上是2MB 一个HugePage.
HugePages_Rsvd 上面说明是保留的页数.
内存页置换原理: [i][url]http://www.ibm.com/developerworks/cn/linux/l-cn-hugetlb[/url][/i]
Kernel的内存页: [i][url]https://www.kernel.org/doc/Documentation/sysctl/kernel.txt[/url][/i]
MySQL使用HugePage: [i][url]http://dev.mysql.com/doc/refman/5.5/en/large-page-support.html[/url][/i] [i][url]http://dev.mysql.com/doc/refman/5.6/en/large-page-support.html[/url][/i]
我们在Linux上,使用命令查看相关的Huge Page在OS上的参数.
cat /proc/meminfo | fgrep -i Huge
AnonHugePages: 5775360 kB
HugePages_Total: 47872
HugePages_Free: 4304
HugePages_Rsvd: 361
HugePages_Surp: 0
Hugepagesize: 2048 kB
[b]环境[/b]
前面属述的效果图相关环境如下:
CentOS release 6.5 (Final)
Linux 2.6 x86_64
MySQL 5.5 (5.6也有一样有2~3倍提升)
[b]配置[/b]
Linux OS
[list]
[*]vm.nr_hugepages = 47872 #开启47872个大页. 相当于
[*]vm.hugetlb_shm_group = 499 #可以使用的HugePage,Linux进程相关的用户组ID
[*]kernel.shmmax = 94489280512 # Controls the maximum shared segment size, in bytes #单个进程下最大能使用的HugePage个数
[*]kernel.shmall = 230686720 # Controls the maximum number of shared memory segments, in pages. default page size : 4KB #Linux System整个所能使用的个数
[/list]
我喜欢一次性把几个参数在文件中配置完. 记住用命令 sysctl -p生效之.
MySQL --defaults-file 中开启
[mysqld]
large-pages
[b]参数大小计算说明和示例[/b]
*numastat 可以看到当前系统下numa状态.
[img]http://dl2.iteye.com/upload/attachment/0097/8986/90210b03-e275-3605-8d43-77f5c725a518.png[/img]
*注意MySQL manual提示 : For MySQL usage, you normally want the value of shmmax to be close to the value of shmall. 我们尽量的把shmmax -> 接近 shmall的值.
[list]
[*][b]1st step[/b]确定机器的CPU processors(指我们在Linux命令看到的个数. cat /proc/cpuinfo | fgrep -i 'processor' | wc -l )
[*][b]2nd step[/b]确定MySQL每个connection所消耗的内存, 我们可以使用mysqltuner.pl 查看到 [i][url]http://mysqltuner.com/[/url][/i]
[/list]
网上很多的资料说BufferPool 配置到80%的物理内存, 是有道理的, 同时这是有前提的max_connections是很少的.
总的使用内存不能超过物理内存的90%. (这个原理需要单独写篇BLOG)
我的配置示例是:
CPU 32 processors .
选择 mysql my.cnf
max_connections = 3000 # 约47GB
innodb_buffer_pool_size = 84480M
innodb_buffer_pool_instances = 30 #每个instance是2.75GB. 因为每个在2.5GB~3GB是很高效的, 体现在MySQL RT上. 而30(CPU 32)这个数量容易计算.留一两个processors给系统. 而[b]每个实例会多占用近100MB的HugePage[/b]
加上其他的conns消耗:
Total buffers: 83.0G global + 16.2M per thread (3000 max threads)
[OK] Maximum possible memory usage: 130.7G (82% of installed RAM)
这样30+4(留足一定的Huge Pages给MySQL和系统其他用).
vm.nr_hugepages = 47872 ( HugePages_Total : 47872 34*2.75GB / 2MB ) <<< 物理内存的80%.
kernel.shmmax = 94489280512 ( 32*2.75GB = 94489280512 bytes )
kernel.shmall = 230686720 ( 32*2.75 GB / 4KB = 230686720 ) 其中: 4KB 是LinuxKernelPage基本大小.
--
如果不留足vm.nr_hugepages , MySQL启动会报错误,启动不了!
看看MySQL 占用效果:
ipcs | fgrep mysql
[img]http://dl2.iteye.com/upload/attachment/0097/9028/20fe10f1-4833-3237-8e56-a85d258eb77c.png[/img]
使用top (请忽略RES). [b]关注 VIRT一项[/b].
[img]http://dl2.iteye.com/upload/attachment/0097/9030/b3da134a-a23f-3353-883e-6a2dda182207.png[/img]
----
[list]
[*]MySQL 最好根据实际需要配置上大的内存的.(反正现在内存很便宜的.)
[*]如果是一机器上的单实例, 直接关闭 numa. 或是设置kernerl启动项 在/etc/grub.conf , numa=off
[*]关闭swap. swapoff -a 执行. 或是在 /etc/rc.local 追加swapoff -a.
[/list]
--------
最后我们可以按照MySQL defaults-file和机器的物理内存大小配置线上的HugePage参数.
我们只是配置正确,不是在"调优". <高性能MySQL>提示.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
有应聘"MySQL/MariaDB,运维/开发", 请发邮件至: [email protected], 并取得联系.
工程师招聘: [url]http://job.youzan.com[/url] 请联系我: [email protected] .