公司短信服务器下发并发在2000条/s,需要先把短信写入到本地磁盘然后发送。同事反应发送短信慢,有事发送失败。
通过查看磁盘io发现:
[root@wmonstermt1] ~# iostat -x 1
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 34.5 63.6 141.8 1370.5 2143 737.3 30
pass0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 1.0 50.0 4.0 1243.7 143 1256 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 0.0 29.0 0.0 1038.3 2267 782 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
可以发现服务时间和等待时间超长。说明磁盘IO不够用了。如何解决?临时解决方案:
查看当前raid卡阵列IO策略:
[root@wmonstermt1] ~# MegaCli64 -LDInfo -Lall -aALL
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 2.178 TB
Sector Size : 512
Parity Size : 557.861 GB
State : Optimal
Strip Size : 256 KB
Number Of Drives : 5
Span Depth : 1
Default Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAhead, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disabled
Encryption Type : None
PI type: No PI
Is VD Cached: No
更改Raid卡策略吧:
[root@wmonstermt3] ~# MegaCli64 -LDSetProp WB -L0 -a0
[root@wmonstermt3] ~# MegaCli64 -LDSetProp CachedBadBBU -Lall -aALL
[root@wmonstermt3] ~# MegaCli64 -LDInfo -Lall -aALL
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-5, Secondary-0, RAID Level Qualifier-3
Size : 2.178 TB
Sector Size : 512
Parity Size : 557.861 GB
State : Optimal
Strip Size : 256 KB
Number Of Drives : 5
Span Depth : 1
Default Cache Policy: WriteBack, ReadAhead, Direct, Write Cache OK if Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Direct, Write Cache OK if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Disabled
Encryption Type : None
PI type: No PI
Is VD Cached: No
[root@wmonstermt3] ~# iostat -x 1
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 17.5 57.0 77.0 1348.1 0 537.2 28
pass0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 0.0 1.0 0.0 127.9 0 0.1 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 0.0 4.0 0.0 511.6 0 0.1 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
extended device statistics
device r/s w/s kr/s kw/s wait svc_t %b
da0 0.0 5.0 0.0 447.6 0 0.1 0
pass0 0.0 0.0 0.0 0.0 0 0.0 0
发现效果杠杠滴。