ES 性能测试
监控软件
marvel安装
kibana
测试环境
| host | 私网IP | IP | 配置 | 操作系统 | ES角色 |
|---|---|---|---|---|---|
| NM-WL-ES-01 | 10.37.30.2 | 100.66.137.152 | 4*8c,512G Mem,8T Disk | CentOS7.2 | MD |
| NM-WL-ES-02 | 10.37.30.3 | 100.66.137.153 | 4*8c,512G Mem,8T Disk | CentOS7.2 | MD |
| NM-WL-ES-03 | 10.37.30.4 | 100.66.137.154 | 4*8c,512G Mem,8T Disk | CentOS7.2 | MD |
| NM-WL-ES-04 | 10.37.30.5 | 100.66.137.155 | 4*8c,512G Mem,8T Disk | CentOS7.2 | DD |
| NM-WL-ES-05 | 10.37.30.6 | 100.66.137.156 | 4*8c,512G Mem,8T Disk | CentOS7.2 | DD |
| NM-WL-ES-06 | 10.37.30.7 | 100.66.137.157 | 4*8c,512G Mem,8T Disk | CentOS7.2 | DD |
每台起两个ES实例， 3个Master, 9个Data
ES 入库效率
8进程, 连接的数据节点有6个
单条数据 888
{
"T0": "2017-09-07 10:27:01.670",
"T1": "43",
"T10": "2",
"T11": "460-11",
"T12": "0x0003BD8B",
"T13": "0x0000",
"T14": "115.169.119.159",
"T15": "TOPON.S11.GW0202-B-HW.NJ.JS.NODE.EPC.MNC011.MCC460.3GPPNETWORK.ORG",
"T16": "460-11",
"T17": "0x3203",
"T18": "1",
"T19": "0xD3C56A04",
"T2": "64F011033201FFAB",
"T20": null,
"T21": "0xFFFF",
"T22": "255",
"T23": "0xFFFFFFFF",
"T24": "0x00C1CECD",
"T25": "2017-Sep-07 10:26:59",
"T26": "460-11",
"T27": "0x3B40",
"T28": null,
"T29": "0xFFFF",
"T3": "0x13707FEF",
"T30": null,
"T31": "4294967295",
"T32": "0xFFFFFFFF",
"T33": "1970-Jan-01 07:59:59",
"T34": null,
"T35": "0xFFFF",
"T36": "0x0000",
"T37": "0x0000",
"T38": "0x0000",
"T39": "0",
"T4": "0x95",
"T40": "00",
"T41": "0xFFFF",
"T42": "0xFF",
"T43": "0xFFFF",
"T44": "0",
"T45": "00",
"T46": "0xFFFF",
"T47": "0xFF",
"T48": "0x00005FD8",
"T49": "0x31",
"T5": "460110171455497",
"T50": null,
"T51": "F070000018",
"T6": "8618921771001",
"T7": "3585880708120215",
"T8": "0x1E",
"T9": "10"
}
查看template
curl http://100.66.137.157:9201/_template?pretty
Paste_Image.png
time /slview/logpro/bin/EpcFile2ES_Invoke.pl -f 8 -t test_mmelog -v epc -s TaskWLJ -c '/slview/test/liyr/mmelog/test_mmelog.yml' -lp 1
real 2m33.672s
echo "4072653/153" | bc
26618 /s
ES 索引占磁盘大小
CASE 1
"include_in_all" : false,
字段类型都为string，无分词
无正排索引
源文件大小 3.3G (json)
转为csv 1.77G
sed -i 's/\"//g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10
sed -i -E 's/T[0-9]+\://g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10
sed -i 's/{//g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10
sed -i 's/}//g' ./HuaweiUDN-MM01_20170907105102_000000.csv_007_10
入库后 1.78G
约为 1：1
CASE 2
"include_in_all" : ture,
string 带分词
store true
doc_values true
入库时间 real 2m54.699s
2.84Gi
约为 1: 1.5