Linux PCIe SSD NVME 性能调优篇
直接来干货！怎么调优 PBlaze IV PCIe SSD NVMe。 Go!
1. 中断绑定
在Redhat 6.5中的NVMe驱动会自动把全部的中断向量绑定到core0上，如果有多个SSD, core0将会成为瓶颈。
(1) Turns off the IRQ balancer:
|
[root@memblaze-lyk2 ~]# service irqbalance stop |
(2) Check nvme device IRQ vector number:
|
[root@memblaze-lyk2 ~]# cat /proc/interrupts | grep nvme 160: 0 1265046 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme admin, nvme 161: 0 10750058 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme 162: 0 917319 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge nvme ……… ………. |
(3) Assignand check the affinity of a single interrupt vector
[root@memblaze-lyk2~]# echo 0000aaaa > /proc/irq/161/smp_affinity
[root@memblaze-lyk2~]# cat /proc/irq/161/smp_affinity
0000,0000aaaa
[root@memblaze-lyk2~]# cat /proc/irq/161/smp_affinity_list
1,3,5,7,9,11,13,15
(4) You cat make a script to affinity all interrupt vector to all cores.
2. Multi device with Numa IO
PCIe的设备现在要考虑numa IO 架构啦。这里不详细介绍啦。简单方向就是让你应用的设备都在一个node上面。
(1) Plug multi Pblaze SSD to only one CPU PCIe slot or average plug to each CPU PCIe slot, itdependence on your requirement.
(3) Network and storage IO device on same node.
a) Check younetwork and SSD PCIe slot, they are on same node.
|
[root@memblaze-lyk2 numactl]# cat /sys/block/nvme0n1/device/numa_node 1 [root@memblaze-lyk2 numactl]# lspci | grep Mell 42:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] [root@memblaze-lyk2 numactl]# lspci -t [root@memblaze-lyk2 numactl]# cat /sys/devices/pci0000\:40/0000\:40\:02.0/numa_node 1 |
b) Check both device interrupt bind to node 1
c) Try numctl to bind application run on node 1, you can use cmd “ps -eo pid,args,psr”to check process running on which core.