Elastic+logstash+head绠�鍗曚粙缁�
涓�. 姒傝堪
ElasticSearch鏄竴涓熀浜嶭ucene鐨勬悳绱㈡湇鍔″櫒銆傚畠鎻愪緵浜嗕竴涓垎甯冨紡澶氱敤鎴疯兘鍔涚殑鍏ㄦ枃鎼滅储寮曟搸锛屽熀浜嶳ESTful web鎺ュ彛銆�
浜岋紟ElasticSearch鐨勫畨瑁呬笌绠�鍗曢厤缃�
1.1. 搴旇濮嬬粓杩愯鏈�鏂扮増鏈殑 Java 铏氭嫙鏈猴紙 JVM 锛夛紝 闄ら潪 Elasticsearch 缃戠珯涓婂彟鏈夎鏄庛�偮� Elasticsearch锛� 鐗瑰埆鏄� Lucene锛屾槸涓�涓珮瑕佹眰鐨勮蒋浠躲�侺ucene 鐨勫崟鍏冩祴璇曞拰闆嗘垚娴嬭瘯缁忓父鏆撮湶鍑� JVM 鏈韩鐨� bug銆傝繖浜� bug 鐨勮寖鍥翠粠杞诲井鐨勯夯鐑﹀埌涓ラ噸娈甸敊璇紝鎵�浠ワ紝鏈�濂藉敖鍙兘鐨勪娇鐢ㄦ渶鏂扮増鏈殑 JVM
1.2. 涓嬭浇ElasticSearch
涓嬭浇鍦板潃锛歨ttps://www.elastic.co/downloads/elasticsearch
濡備笅杞芥渶鏂扮増鏈鐐筎AR锛屽叾浠栫増鏈鐐瑰嚮 past releases閫夋嫨
1.3.聽 瑙ｅ帇瀹夎鍖�
1.4. 鐩綍璁茶В锛�
1. bin:鍖呭惈杩愯ElasticSearch瀹炰緥鍜岀鐞嗘彃浠剁殑涓�浜涜剼鏈�
2. Config: 涓昏鏄竴浜涜缃枃浠讹紝濡俵ogging锛宔lasticsearch.yml,jvm
3. Lib:鍖呭惈鐩稿叧鐨勫寘鏂囦欢绛�
4. Plugins:鍖呭惈鐩稿叧鐨勬彃浠舵枃浠剁瓑
5. Logs锛氭棩蹇楁枃浠�
6. Data锛氭暟鎹瓨鏀惧湴鍧�
1.5. 闆堕厤缃惎鍔�
鐩存帴杩涘叆bin鐩綍涓嬶紝鍚姩鑴氭湰elasticsearch
./elasticsearch
浼氭樉绀哄涓嬮敊璇細
(1).java.lang.RuntimeException: can not run elasticsearch as root
杩欐槸鐢变簬elastic涔嬪墠鍥犱负root瓒呯骇鏉冮檺涓嬪惎鍔ㄥ紩鍙戣繃琛�妗堬紝鎵�鏈夊己鍒朵笉璁╁湪root涓嬪惎鍔� 銆傝В鍐虫柟妗堬細
娣诲姞elastic鍚姩鐢ㄦ埛锛�
useradd syliu
passwd syliu
chown -R syliu:syliu elasticsearch-6.1.3
su syliu #
鐒跺悗杩涘叆bin鐩綍涓嬪惎鍔�./elasticsearch
(2).max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
杩欐槸鐢变簬linux涓嬫渶澶ф墦寮�鏂囦欢鏁伴噺姣攅lastic瑕佹眰鐨勬墦寮�鏁伴噺灏�
鏌ョ湅绯荤粺鏈�澶ф墦寮�鏂囦欢鏁伴噺
- ulimit -a (鏌ョ湅)
- ulimit -n 65536(璁剧疆)
鎴栬�咃細
鍒囨崲鍒皉oot鐢ㄦ埛淇敼
vim /etc/security/limits.conf
(3).max number of threads [3889] for user [syliu] is too low, increase to at least [4096]
vi /etc/security/limits.d/90-nproc.conf
vim /etc/security/limits.conf
(4).max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
淇敼/etc/sysctl.conf閰嶇疆鏂囦欢锛�
cat /etc/sysctl.conf | grep vm.max_map_count
vm.max_map_count=262144
濡傛灉涓嶅瓨鍦ㄥ垯娣诲姞
echo "vm.max_map_count=262144" >>/etc/sysctl.conf
1.6. 鍒版,鐜閰嶇疆瀹屾垚锛岄噸鏂板惎鍔ㄩ」鐩紝椤圭洰鍚姩鎴愬姛
璁块棶http://localhost:9200 鍚庝細鏄剧ず鎴愬姛鎻愮ず
(1) name:elastic瀹炰緥鍚嶏紝璁剧疆鍙互鍦╟onfig/elasticsearch.yml涓厤缃�
(2) Version:鐗堟湰鍙�,浠son鏍煎紡琛ㄧず鐨勪竴缁勪俊鎭��
鈶� Name: 褰撳墠鐗堟湰鍙�
鈶� build_snapshot锛氭槸鍚︿粠婧愮爜鏋勫缓鑰屾潵
鈶� lucene_version锛氬熀浜巐ucene鐨勭増鏈�
1.7. 绠�鍗曢厤缃甧lastic闆嗙兢
1. cluster.name:闆嗙兢鍚嶇О銆傝缃ソ浠ュ悗浼氭牴鎹悓涓�闆嗙兢鍚嶅瓧鑷姩鍙戠幇鍚屼竴缃戞涓嬬殑鑺傜偣锛屽鏋滃湪鍚屼竴缃戞涓嬫湁澶氫釜闆嗙兢锛屽彲浠ユ牴鎹繖涓瓧娈靛尯鍒嗛泦缇�
2. Node.name:鑺傜偣鍚嶇О锛屽彲浠ヨ嚜鍔ㄧ敓鎴愯妭鐐瑰悕绉帮紝杩欓噷閰嶇疆鏄湁鍒╀簬鍒╃敤api璁块棶鍏蜂綋鐨勮妭鐐癸紝寤鸿鑷繁閰嶇疆
3. Node.master:鑺傜偣鏄惁涓簃aster涓昏妭鐐�-----姣忎釜鑺傜偣閮藉彲琚厤缃负涓昏妭鐐广�傞粯璁ゅ�间负true锛岀洰鐨勬槸鎸囧畾璇ヨ妭鐐规槸鍚︽湁璧勬牸琚�変妇鎴愪负node銆傞粯璁ら泦缇や腑绗竴鍙伴泦缇や负master銆傚鏋滆繖鍙版満鍣ㄥ嚭鐜版晠闅滐紝闆嗙兢浼氳嚜鍔ㄩ噸鏂伴�変妇
4. Node.data:璁剧疆鑺傜偣鏄惁瀛樺偍鏁版嵁锛岄粯璁や负true锛屽鏋滀笉甯屾湜瀛樺偍锛屽垯璁剧疆涓篺alse
瀹㈡埛鑺傜偣閰嶇疆锛�
Node.master:true
Node.data:false
鏁版嵁鑺傜偣閰嶇疆锛�
Node.master:false
Node.data:true
5. Network.host: 缁戝畾鐩戝惉IP,鎻愪緵澶栫晫璁块棶
6. Transport.tcp.port:璁剧疆鑺傜偣闂翠氦浜掔殑tcp绔彛
7. Discovery.zen.ping.unicast.hosts:璁剧疆闆嗙兢涓璵aster鑺傜偣鐨勫垵濮嬪垪琛�-閫氳繃杩欎簺鑺傜偣鏉ヨ嚜鍔ㄥ彂鐜版柊鍔犲叆鐨勯泦缇�
8. discovery.zen.ping_timeout锛氳缃泦缇や腑鑷姩鍙戠幇鍏朵粬鑺傜偣鏃秔ing杩炴帴瓒呮椂鏃堕棿-榛樿涓�3绉掋�傚浜庢瘮杈冨樊鐨勭綉缁滅幆澧冨彲浠ュ姞澶у�兼潵闃叉鑷姩鍙戠幇鏃跺嚭閿�
9. client.transport.ping_timeout锛氬鎴风杩炴帴ping鐨勬渶澶ц秴鏃舵椂闂�
10. bootstrap.memory_lock锛歵rue 閿佸畾鍐呭瓨锛岄槻姝㈠唴瀛樹氦鎹�
11. http.port:缁戝畾鐩戝惉ip鐨勭鍙ｅ彿锛岄粯璁�9200
12. path.data:鏁版嵁瀛樻斁浣嶇疆锛屾渶濂戒笉瑕佹斁鍦ㄩ粯璁ゅ畨瑁呯洰褰曚笅锛岃繖鏍峰嵏杞戒簡浼氬紩璧锋暟鎹涪澶憋紝鍙互淇濆瓨鍒颁笉鍚岀殑鐩綍涓嬶紝鏈�濂芥寕杞藉埌涓嶅悓鐨勭鐩樹笂
閰嶇疆鏂规硶濡備笅锛歱ath.data: /path/to/data1,/path/to/data2
鍚姩elastic浼氬彂鐜版姤閿欙細
memory locking requested for elasticsearch process but memory is not locked
瑙ｅ喅鍔炴硶锛�
vim /etc/security/limits.conf
Syliu soft memlock unlimited
Syliu hard memlock unlimited
淇敼锛�
/etc/sysctl.conf
vm.swappiness=0
鍒版锛氶厤缃畬鎴愶紝鐒跺悗灏嗘湰閰嶇疆锛屽皢鏈厤缃嫹璐濆埌鍏朵粬鑺傜偣涓嬶紝娉ㄦ剰淇敼node.name.
濡傛灉閰嶇疆鍦ㄥ悓涓�鍙版湇鍔″櫒涓嬪叾浠栬妭鐐硅繕闇�瑕佷慨鏀圭鍙ｅ彿锛屼互鍏嶇鍙ｈ鍗犵敤
涓夛紟Logstash鐨勭畝鍗曢厤缃笌瀹夎
1.1聽 Logstash鏄竴涓兘鏈夋晥杩涜鏃ュ織澶勭悊鐨勫伐鍏凤紝鍙互瀵规棩蹇楄繘琛屾敹闆嗭紝鍒嗘瀽銆傚叾鏈韩骞朵笉浜х敓鏃ュ織锛屽畠鍙槸涓�涓唴缃垎鏋愬拰杞崲宸ュ叿鐨勬棩蹇楃鐞嗗伐鍏凤紝鏄竴涓帴鏀�,澶勭悊,杞彂鏃ュ織鐨勨�欑閬撯�欍��
1.2 涓嬭浇logstash
涓嬭浇鍦板潃锛歨ttps://www.elastic.co/downloads/logstash
涓嬭浇鍘嬬缉鍖咃紝瑙ｅ帇
1.3 杩涘叆bin鐩綍涓嬪垱寤�.conf鏂囦欢浣滀负鍚姩鏂囦欢
Logstash澶勭悊浜嬩欢鏈変笁涓樁娈碉細杈撳叆Inputs,杩囨护Filters,杈撳嚭OutPuts
鍏蜂綋鍙傞槄锛歨ttp://udn.yyuap.com/doc/logstash-best-practice-cn/input/stdin.html 鎴栬�卙ttps://www.elastic.co/guide/en/logstash/current/input-plugins.html
1.鎸囧畾file涓鸿鍙栨枃浠剁殑鏂瑰紡锛�
input {
file {
type => "guoan"
path => "F:/logs/guoanjia/guoanjia/*.log"
codec => json { charset => "GBK" }
start_position => "beginning"
}
}
type: 鏍囪浜嬩欢绫诲瀷聽 闆嗘垚java鐨勬椂鍊欓渶瑕佺敤鍒般�傝繕鍙互鏍规嵁type鍊煎仛涓�浜涜繃婊ょ殑鎿嶄綔
path:涓烘煇涓粷瀵硅矾寰勬枃浠跺す涓嬫墍鏈変互.log缁撳熬鐨勬枃浠朵负鏁版嵁婧�
codec: 缂栫爜鎻掍欢锛� codec 灏辨槸鐢ㄦ潵 decode銆乪ncode 浜嬩欢鐨�
鎸囧畾杈撳叆鍒發ogstash鐨勬牸寮忎负json鏍煎紡鐨勶紝鎸囧畾瀛楃闆嗕负涓枃
start_position锛氫粠浠�涔堜綅缃紑濮嬭鍙栨枃浠舵暟鎹紝榛樿鏄粨鏉熶綅缃�
鎶婅繖涓瀹氭敼鎴� "beginning"锛宭ogstash 杩涚▼灏变粠澶村紑濮嬭鍙�
闄ゆ涔嬪杩樻湁涓�浜涢厤缃湁鏃跺�欓渶瑕佹墜鍔ㄩ厤涓婏細
discover_interval锛歭ogstash 姣忛殧澶氫箙鍘绘鏌ヤ竴娆¤鐩戝惉鐨� path 涓嬫槸鍚︽湁鏂版枃浠躲�傞粯璁ゅ�兼槸 15 绉�
exclude锛氫笉鎯宠鐩戝惉鐨勬枃浠跺彲浠ユ帓闄ゅ嚭鍘�
stat_interval锛歭ogstash 姣忛殧澶氫箙妫�鏌ヤ竴娆¤鐩戝惉鏂囦欢鐘舵�侊紙鏄惁鏈夋洿鏂帮級锛岄粯璁ゆ槸 1 绉�
******娉ㄦ剰锛歴tart_position 浠呭湪璇ユ枃浠朵粠鏈鐩戝惉杩囩殑鏃跺�欒捣浣滅敤銆傚鏋� sincedb 鏂囦欢涓凡缁忔湁杩欎釜鏂囦欢鐨� inode 璁板綍浜嗭紝閭ｄ箞 logstash 渚濈劧浼氫粠璁板綍杩囩殑 pos 寮�濮嬭鍙栨暟鎹�傛墍浠ラ噸澶嶆祴璇曠殑鏃跺�欐瘡鍥為渶瑕佸垹闄� sincedb 鏂囦欢銆�
2. 閫氳繃TCP濂楁帴瀛楄鍙栦簨浠躲��
input {
tcp {
host =>127.0.0.1
port => 8999
mode => "server"
ssl_enable => false
type => "guoan3"
codec => json { charset => "GBK" }
}
}
鍙互鎺ュ彈鏉ヨ嚜瀹㈡埛绔殑杩炴帴鎴栬繛鎺ュ埌鏈嶅姟鍣紝鍏蜂綋鍙栧喅浜巑ode銆�
缂栬緫
mode
鍊煎彲浠ユ槸浠讳綍鐨勶細server锛宑lient
榛樿鍊间负 "server"
鎿嶄綔妯″紡銆俿erver鐩戝惉瀹㈡埛绔繛鎺ワ紝 client杩炴帴鍒版湇鍔″櫒銆�
ssl_enable => false
鍚敤SSL锛堝繀椤昏缃叾浠杝sl_閫夐」鎵嶈兘鐢熸晥锛�
host:
鍊肩被鍨嬫槸瀛楃涓�
榛樿鍊间负 "0.0.0.0"
褰撴ā寮忔槸server锛屽湴鍧�瑕佺洃鍚�傚綋妯″紡鏄痗lient锛岃繛鎺ュ埌鐨勫湴鍧�
鐒跺悗 闇�瑕佸湪SpringBoot椤圭洰涓姞鍏� 鐩稿簲鐨勪緷璧栵細
鏃ュ織閰嶇疆鏂囦欢(涓嶆槸SpringBoot涔熷彲浠�)锛�
鍒╃敤鏈ā寮忚繘琛屾暟鎹殑瀵煎叆闇�瑕佸湪springboot 閰嶇疆鏂囦欢涓姞鍏�
server:
context-path: /agenthouseCutomer
port:8084
max-http-header-size: 10024
3. 鍒╃敤logstash鐨� logstash-input-jdbc杩涜涓巑ysql鏁版嵁鐨勫悓姝�
杩涘叆logstash鐨刡in鐩綍涓嬪畨瑁呮彃浠�
Logstash-plugin install logstash-input-jdbc
Input閰嶇疆
4. 鍏充簬filter鐨勯厤缃�
4.1 grok filter
127.0.0.1 POST /logs/getLog 12345 0.123
grok { match => {鈥渕essage鈥�=>鈥滐紖{IP锛歝lient}锛厈WORD锛歮ethod}锛厈URIPATHPARAM:url}锛厈NUMBER锛歝ount}锛厈NUMBER锛歮oney}鈥潁 }
鍙互灏嗚偖鑴忕殑闈炵粨鏋勫寲鏃ュ織鏁版嵁瑙ｆ瀽鎴愮粨鏋勫寲鍜屽彲鏌ヨ鐨勬暟鎹�
Grok浣滀负鏁版嵁缁撴瀯鍖栧伐鍏�,鍦╨ogstash涓粯璁や笂鐧句釜grok鍙橀噺锛屽彲浠ョ洿鎺ユ嬁鏉ヤ娇鐢�
閫傚悎瀵箂yslog.apache log绛夊彲璇绘棩蹇楄繘琛屽垎鏋�
4.2 kv filter
瀵逛簬璇稿key-value 杩欑閿�煎鏁版嵁杩涜鍒嗘瀽
濡傦細user=鍥藉畨1&url=111&method=main&ip=124.65.164.98&args=null
kv {
source => "message"
field_split => "&?"
}
瑙ｆ瀽鍑烘潵鏍煎紡
"method": "main",
ip": "124.65.164.98",
"message": "user=鍥藉畨1&url=111&method=main&ip=124.65.164.98&args=null",
"url": "111",
"args": "null",
"user": "鍥藉畨1"
4.3 geoip
geoip {
source => "ip"
fields => ["city_name", "country_code2", "country_name", "latitude", "longitude", "region_name"]
remove_field => ["[geoip][country_code3]", "[geoip][region_name]", "[geoip][continent_code]", "[geoip][timezone]", "[geoip][ip]"]
}
鏍规嵁涓婇潰kv瑙ｆ瀽鍑烘潵鐨刬p瀛楁杩涜鑾峰彇璇︾粏鍦扮悊淇℃伅鍜岀粡绾害
鍝嶅簲锛�
"geoip": {
"city_name": "Beijing",
"latitude": 39.9289,
"country_code2": "CN",
"country_name": "China",
"longitude": 116.3883
},
5. 鍏充簬output閰嶇疆
5.1 elasticsearch
elasticsearch {
hosts => "127.0.0.1:9200"
#index => "guoan88881234"
index => "guoerror-%{+YYYY.MM.dd}"
user => elastic
password => changeme
retry_on_conflict => 5
}
灏唋ogstash涓暟鎹緭鍏ュ埌elasticsearch涓細
hosts:ip鍔犵鍙ｅ彿锛屾垨鑰呭煙鍚�
index锛氫唬琛ㄦ瘡澶╀互guoerror-寮�澶寸敓鎴愮储寮�
user:elasticsearch鐨勭敤鎴峰悕
password锛氫唬琛╡lasticsearch鐨勫瘑鐮�
retry_on_conflict锛氳緭鍑洪噸璇曟鏁�
5.2 email
email {
port => "25" 绔彛
address => "smtp.126.com" 鍦板潃
username => "[email protected]"聽 鐢ㄦ埛
password => "*****" 瀹㈡埛绔巿鏉冨瘑鐮� 閫傜敤浜庣櫥褰曚互涓嬫湇鍔�: POP3/IMAP/SMTP/Exchange/CardDAV/CalDAV鏈嶅姟
authentication => "plain" 鍥哄畾
use_tls => false
from => "[email protected]"
subject=> "Warning: 绯荤粺鍑洪敊浜�!%{@timestamp}-%{type}-%{logger_name}"
to => "[email protected]"
via => "smtp"
body => "%{stack_trace}"
聽 聽 聽 聽 }
6. 鍚姩logstash
閰嶇疆瀹屾垚鍚巜indons涓嬪垱寤� run.bat
鍐欏叆logstash -f guoan.conf 鍚姩鍛戒护
鎴栫洿鎺ュ懡浠よlogstash -f guoan.conf鍚姩
-f 鍙傛暟鎰忔�� 鈥樻枃浠垛��
浣滅敤鏄寚瀹歭ogstash閰嶇疆鏂囦欢
-e 鍙傛暟鐨勪綔鐢ㄤ负鎵ц聽 濡傦細logstash -e 鈥榠nput{stdin{}} output{stout{}}鈥�
闄ゆ涔嬪锛岃繕鏈� -t,-l,-w,-p,-v
鍚姩涔嬪墠娉ㄦ剰锛�
鍦╟onfig/logstash.yml涓厤缃細
#pipeline绠￠亾绾跨▼鏁帮紝瀹樻柟寤鸿鏄瓑浜嶤PU鍐呮牳鏁�
pipeline.workers: 8
#pipeline绠￠亾瀹為檯output鏃剁殑绾跨▼鏁帮紝涓�鑸皬浜庢垨绛変簬绠￠亾绾跨▼鏁帮紝寤鸿绛変簬cpu鍐呮牳鏁�
pipeline.output.workers: 8
#鍗曚釜宸ヤ綔绾跨▼鍦ㄥ皾璇曟墽琛岃繃婊ゅ櫒鍜岃緭鍑轰箣鍓嶆敹闆嗙殑鏈�澶т簨浠舵暟锛岄粯璁�125锛涖�� 鏁板�艰秺澶э紝澶勭悊鍒欓�氬父鏇撮珮鏁堬紝浣嗗鍔犱簡鍐呭瓨寮�閿�锛涖��鏌愪簺纭欢閰嶇疆瑕佹眰閫氳繃璁剧疆LS_HEAP_SIZE鍙橀噺鏉ュ鍔燡VM鍫嗗ぇ灏忥紝浠ラ伩鍏嶄娇鐢ㄦ閫夐」瀵艰嚧鎬ц兘涓嬮檷锛涖��姝ゅ弬鏁扮殑鍊艰秴杩囨渶浣宠寖鍥翠細瀵艰嚧鐢变簬棰戠箒鐨勫瀮鍦惧洖鏀舵垨涓庡唴瀛樹笉瓒冲紓甯哥浉鍏崇殑JVM宕╂簝鑰屽鑷存�ц兘涓嬮檷锛涖��璋冩暣pipeline.batch.size璁剧疆澶у皬鍙皟鏁村彂閫佸埌Elasticsearch鐨勬壒閲忚姹傜殑澶у皬
pipeline.batch.size: 3000
#姝よ缃皟鏁碙ogstash绠￠亾鐨勫欢杩燂紝榛樿5锛涖��娴佹按绾挎壒澶勭悊寤惰繜鏄疞ogstash鍦ㄥ綋鍓嶇閬撳伐浣滅嚎绋嬩腑鎺ユ敹鍒颁簨浠跺悗绛夊緟鏂版秷鎭殑鏈�澶ф椂闂达紙姣锛夛紱銆�鍦ㄦ鏃堕棿杩囧悗锛孡ogstash寮�濮嬫墽琛岃繃婊ゅ櫒鍜岃緭鍑�.Logstash鍦ㄦ帴鏀朵簨浠跺拰鍦ㄨ繃婊ゅ櫒涓鐞嗚浜嬩欢涔嬮棿绛夊緟鐨勬渶澶ф椂闂存槸pipeline.batch.delay鍜宲ipeline.batch.size璁剧疆鐨勪箻绉�
pipeline.batch.delay: 100
鍥涳紟Head 閰嶇疆
1.鍦╡lasticsearch.yml閰嶇疆璺ㄥ煙鏀寔
http.cors.enabled: true
http.cors.allow-origin: "*"
2. 涓嬭浇 elasticsearch-head
https://github.com/mobz/elasticsearch-head
3. Npm install
Npm run start
鎴栬�呭幓google鍟嗗簵鎼滅储ElasticSearch-head鎻掍欢锛岀洿鎺ュ畨瑁呭嵆鍙娇鐢�
鎴戝緱鐧惧害缃戠洏锛歨ttps://pan.baidu.com/s/1dkA1m9XECKSfpH65-qmE3Q
涓嬭浇瀹屼箣鍚庡姞鍏ヨ胺姝屾墿灞曠▼搴忓嵆鍙娇鐢�
浜旓紟ElasticSearch浼樺寲寤鸿
1.璋冨ぇ绯荤粺鐨勨�滄渶澶ф墦寮�鏂囦欢鏁扳�濓紝寤鸿32k鐢氳嚦64k
- ulimit -a (鏌ョ湅)
- ulimit -n 32000(璁剧疆)
2. 淇敼閰嶇疆鏂囦欢璋冩暣es鐨刯vm鍐呭瓨澶у皬
淇敼bin/elasticsearch.in.sh涓璄S_MIN_MEM鍜孍S_MAX_MEM鐨勫ぇ灏忥紝寤鸿璁剧疆涓�鏍峰ぇ锛岄伩鍏嶅钩鍑＄殑鍒嗛厤鍐呭瓨锛屾牴鎹湇鍔″櫒鍐呭瓨澶у皬閰嶇疆锛屼竴鑸垎閰�60%宸﹀彸锛堥粯璁�256锛�
3.璁剧疆mlockall鏉ラ攣瀹氳繘绋嬬殑鐗╃悊鍐呭瓨鍦板潃
- 閬垮厤浜ゆ崲锛坰wapped锛夋潵鎻愰珮鎬ц兘
-淇敼鏂囦欢 conf/elasticsearch.yml 涓� bootstrap.mlockall:true
4.鍒嗙墖澶氱殑璇濆彲浠ユ彁鍗囧缓绔嬬储寮曠殑鑳藉姏锛�5-20涓瘮杈冨悎閫傦紝榛樿5涓�
濡傛灉鍒嗙墖鏁拌繃灏戞垨杩囧锛岄兘浼氬鑷存绱㈡瘮杈冩參銆傚垎鐗囨暟杩囧浼氬鑷存绱㈡椂鎵撳紑姣旇緝澶氱殑鏂囦欢锛屽彟澶栦篃浼氬鑷村鍙版湇鍔″櫒涔嬮棿鐨勯�氳杩囨參銆傝�屽垎鐗囨暟杩囧皯浼氬鑷村崟涓垎鐗囩储寮曡繃澶э紝鎵�浠ユ绱㈤�熷害浼氭參涓嬫潵銆傚缓璁崟涓垎鐗囨渶澶氬偍瀛�20G宸﹀彸鐨勭储寮曟暟鎹紝鎵�浠ワ紝鍒嗙墖鏁�=鏁版嵁鎬婚噺/20G
5.鍓湰澶氱殑璇濓紝鍙互鎻愬崌鎼滅储鑳藉姏锛屼絾鏄鏋滆缃緢澶氬壇鏈殑璇濅篃浼氬鏈嶅姟鍣ㄩ�犳垚棰濆鐨勫帇鍔涳紝鍥犱负闇�瑕佸悓姝ユ暟鎹紝鎵�浠ュ缓璁缃�2-3涓�
PUT /gakj-1 { "settings": { "number_of_shards" : 5, "number_of_replicas" : 2 } }
6.瀹氭椂浼樺寲绱㈠紩锛宻egment瓒婂锛屾煡璇㈤�熷害瓒婂樊
max_num_segments=1
7.鍘绘帀mappings _all鍩燂紝绱㈠紩鍒涘缓鏃堕粯璁や細鏈塤all鍩燂紝杩欐牱浼氱粰鏌ヨ甯︽潵鏂归潰锛屼絾鏄鍔犵储寮曟椂闂村拰绱㈠紩灏哄害
- "_all":{"enabled":"false"}
8.log杈撳嚭鐨勬按骞抽粯璁や负trace,鍗虫煡璇㈣秴杩�500ms鍗充负鎱㈡煡璇紝灏辫鎵撳嵃鏃ュ織锛岄�犳垚cpu鍜宮em,io璐熻浇寰堥珮锛屾妸log杈撳嚭姘村钩鏀逛负info锛屽彲鍑忚交鏈嶅姟鍣ㄧ殑鍘嬪姏
-淇敼ES_HOME/conf/logging.yaml鏂囦欢
-鎴栦慨鏀笶S_HOME/conf/elasticsearch.yaml
鍏紟Elastic瀹夎x-pack
瀹夎锛氳繘鍏lastic 鐨刡in涓�
鍛戒护锛歟lasticsearch-plugin install x-pack
瀹夎瀹屾垚鍚庡惎鍔╡lasticsearch
鍒濆鍖栧瘑鐮侊細
bin/x-pack/setup-passwords auto
鎴栬�呮墜鍔ㄨ缃瘑鐮侊細
bin/x-pack/setup-passwords interactive
娉ㄦ剰锛歺-pack鐩墠閮ㄥ垎鍔熻兘鏄敹璐圭殑锛岃瘯鐢ㄦ湡鏄竴涓湀锛屽熀纭�鐗堥渶瑕佺敵璇凤紝鍩虹鐗堝厤璐�
涓冿紟ElasticSearch瀛︿範绗旇鍙婃煡璇㈣娉�
1.淇℃伅妫�绱笌缁撴灉杩囨护
閫氳繃_source 杩斿洖鎸囧畾鐨勬绱㈠瓧娈甸泦
2.kibana 涓� sort鎺掑簭涓嶅噯纭殑闂
3.match 妫�绱㈠瓙鍙ュ拰_all鍙傛暟鐨勪娇鐢�
鍖归厤鍏ㄩ儴瀛楁鐨勬绱�
4.match_phrase 鍦ㄦ寚瀹氬瓧娈典腑鍖归厤鐗瑰畾鐨勬绱㈠幓
5.瀵逛簬缁欏畾鐨勫唴瀹癸紝query_string鏌ヨ浣跨敤鏌ヨ瑙ｆ瀽鍣ㄦ潵鏋勯�犲疄闄呯殑鏌ヨ
鍏朵腑^2琛ㄧず鍖呭惈鎵撳紑闂ㄩ攣涓斿叾鏉冮噸涓�2
6.prefix 鎵惧埌鏌愪釜瀛楁浠ョ粰瀹氬墠缂�寮�澶寸殑鏂囨。
7.range 鑼冨洿鏌ヨ
gte
澶т簬鎴栫瓑浜�
gt
姣�...鏇存
lte
灏忎簬鎴栫瓑浜�
lt
灏戜簬
8.more_like_this,fuzzy_like_this
鏌ヨ寰楀埌涓庢墍鎻愪緵鐨勬枃鏈浉浼肩殑鏂囨。
fields
like
9.multi_match 澶氬瓧娈典腑缁勫悎鎼滅储
10.杩囨护鏌ヨ
鏌ヨ瀛樺湪鏌愪釜瀛楁鐨勬棩蹇�
11.query_string 鏌ヨ
12.cat鍛戒护
1.GET /_cat
鍒楀嚭鎵�鏈塩at鍛戒护
2.GET /_cat/health?v
妫�鏌ラ泦缇ゅ仴搴风姸鍐�
3.GET /_cat/nodes?v
鑺傜偣缁熻
4.GET /_cat/indices
鑾峰彇鍏ㄩ儴鑺傜偣
13 璁剧疆鏈�灏忚妭鐐癸紝闃叉鑴戣锛岄伩鍏嶄袱涓富鑺傜偣鍚屾椂瀛樺湪涓�涓泦缇や腑
discovery.zen.minimum_master_nodes: 2
閫氳繃api淇敼姝ｅ湪杩愯鐨勮妭鐐癸紝璁剧疆鏈�灏忚妭鐐�
PUT /_cluster/settings
{
聽 聽 聽 聽 聽 "persistent" : {
聽 聽 聽 聽 聽 聽 聽 聽 "discovery.zen.minimum_master_nodes" : 2
聽 聽 }
聽 聽 聽 聽 }
14.闆嗙兢鎭㈠鏂归潰鐨勯厤缃�
闃绘 Elasticsearch 鍦ㄥ瓨鍦ㄨ嚦灏� 8 涓妭鐐癸紙鏁版嵁鑺傜偣鎴栬�� master 鑺傜偣锛変箣鍓嶈繘琛屾暟鎹仮澶�
gateway.recover_after_nodes: 8
閰嶇疆闆嗙兢涓簲璇ユ湁澶氬皯鑺傜偣锛屼互鍙婃垜浠効鎰忎负杩欎簺鑺傜偣绛夊緟澶氶暱鏃堕棿
gateway.expected_nodes: 10聽 聽 聽 gateway.recover_after_time: 5m
璁剧疆鍗曟挱鑺傜偣鍒楄〃锛岄槻姝㈣妭鐐规棤鎰忓姞鍏ラ泦缇�
discovery.zen.ping.unicast.hosts: ["host1", "host2:port"]
闃叉鍚屼竴涓垎鐗囷紙shard锛夌殑涓诲壇鏈瓨鍦ㄥ悓涓�涓墿鐞嗘満涓�
cluster.routing.allocation.same_shard.host: true
鍏佽 JVM 閿佷綇鍐呭瓨锛岀姝㈡搷浣滅郴缁熶氦鎹㈠嚭鍘�
bootstrap.mlockall: true
15.閰嶇疆elasticsearch涓嶈鍘婚厤缃瀮鍦惧洖鏀跺拰绾跨▼姹�
璁剧疆鍫嗗唴瀛�
export ES_HEAP_SIZE=1g 鎴� ./bin/elasticsearch -Xmx1g -Xms1g
纭繚鍫嗗唴瀛樻渶灏忓�硷紙 Xms 锛変笌鏈�澶у�硷紙 Xmx 锛夌殑澶у皬鏄浉鍚岀殑锛岄槻姝㈢▼搴忓湪杩愯鏃舵敼鍙樺爢鍐呭瓨澶у皬锛� 杩欐槸涓�涓緢鑰楃郴缁熻祫婧愮殑杩囩▼銆�
璁剧疆绂佺敤鍐呭瓨浜ゆ崲 鎴� 闄嶄綆swappiness 鐨勫��
sudo swapoff -a聽 聽 vm.swappiness = 1
16.鏌ヨ鏌愪釜瀛楁蹇呴』鍖呭惈浠�涔堬紝蹇呴』鎺掗櫎浠�涔�
17.es鍐呯疆REST鎺ュ彛
1./index/_search 鎼滅储鎸囧畾绱㈠紩涓嬬殑鏁版嵁
2./_aliases聽 鑾峰彇鎴栨搷浣滅储寮曠殑鍒悕
3./index/type/聽 鍒涘缓鎴栨搷浣滅被鍨�
4./index/_mapping 鍒涘缓鎴栨搷浣渕apping
5./index/_settings聽 鍒涘缓鎴栨搷浣滆缃紙number_of_shards 鏄笉鍙洿鏀圭殑锛�
6./index/_open 鎵撳紑鎸囧畾琚叧闂殑绱㈠紩
7./index/_close 鍏抽棴鎸囧畾绱㈠紩
8./index/refresh聽 鍒锋柊绱㈠紩锛堜娇鏂板姞鍐呭瀵圭储寮曞彲瑙侊紝涓嶄繚璇佹暟鎹鍐欏叆纾佺洏锛�
9./index/_flush聽 鍒锋柊绱㈠紩锛堜細瑙﹀彂lucene鎻愪氦锛�
18.term鏌ヨ鎸囧畾鐨勭簿纭��
19.浣跨敤 constant_score 鏌ヨ浠ラ潪璇勫垎妯″紡鏉ユ墽琛� term 鏌ヨ骞朵互涓�浣滀负缁熶竴璇勫垎
鍐呴儴杩囨护鍣ㄧ殑鎿嶄綔锛�
1.鏌ユ壘鍖归厤鏂囨。
term鏌ヨ鍦ㄥ�掓帓绱㈠紩涓煡鎵惧寘鍚term鐨勬墍鏈夋枃妗ｏ紝瀹冩弿杩颁簡鍝釜鏂囨。浼氬寘鍚 term 銆傚尮閰嶆枃妗ｇ殑鏍囧織浣嶆槸 1
2.鍒涘缓bitset锛堜竴涓寘鍚�0鍜�1鐨勬暟缁勶級
3.杩唬bitset(s)
涓�鏃︿负姣忎釜鏌ヨ鐢熸垚浜� bitsets 锛孍lasticsearch 灏变細寰幆杩唬 bitsets 浠庤�屾壘鍒版弧瓒虫墍 鏈夎繃婊ゆ潯浠剁殑鍖归厤鏂囨。鐨勯泦鍚堛�傛墽琛岄『搴忔槸鍚彂寮忕殑锛屼絾涓�鑸潵璇村厛杩唬绋�鐤忕殑 bitset 锛堝洜涓哄畠鍙互鎺掗櫎鎺夊ぇ閲忕殑鏂囨。锛�
4.澧為噺浣跨敤璁℃暟
Elasticsearch 鑳藉缂撳瓨闈炶瘎鍒嗘煡璇粠鑰岃幏鍙栨洿蹇殑璁�
20.bool杩囨护鍣�
must
鎵�鏈夌殑璇彞閮� 蹇呴』锛坢ust锛� 鍖归厤锛屼笌 AND 绛変环銆�
must_not
鎵�鏈夌殑璇彞閮� 涓嶈兘锛坢ust not锛� 鍖归厤锛屼笌 NOT 绛変环銆�
should
鑷冲皯鏈変竴涓鍙ヨ鍖归厤锛屼笌 OR 绛変环銆�
21.terms鏌ユ壘澶氫釜绮剧‘鍊�
22.鍒涘缓鏂囨。绱㈠紩
PUT /gakj
{
聽 "settings": {
聽 聽 "number_of_shards": 5
聽 },
聽 "mappings": {
聽 聽 "guoanjia":{
聽 聽 聽 "properties": {
聽 聽 聽 聽 "id" :
聽 聽 聽 聽 {
聽 聽 聽 聽 聽 "type" : "long",
聽 聽 聽 聽 聽 "store": true
聽 聽 聽 聽 聽
聽 聽 聽 聽 },聽
聽 聽 聽 聽 "url" :
聽 聽 聽 聽 {
聽 聽 聽 聽 聽 "type" : "text",
聽 聽 聽 聽 聽 "index" :true,
聽 聽 聽 聽 聽 "omit_norms": "true",
聽 聽 聽 聽 聽 "store": true
聽 聽 聽 聽 }
聽 聽 聽 }
聽 聽 }
聽 }
}
23銆傚垱寤虹储寮曟椂娣诲姞榛樿杩囨湡鏃堕棿
PUT my_index聽
{聽
聽 "mappings": {聽
聽 聽 "my_type": {聽
聽 聽 聽 "_ttl": {聽
聽 聽 聽 聽 "enabled": true,聽
聽 聽 聽 聽 "default": "5m"聽
聽 聽 聽 }聽
聽 聽 }聽
聽 }聽
}聽
濡傛灉涓嶆坊鍔爐tl杩囨湡鍙傛暟鍒欐枃妗ｉ粯璁や笉杩囨湡
聽 d
澶�
聽 ms锛堥粯璁わ級
姣
m
鍒嗛挓
24.ik鍒嗚瘝鍣ㄩ泦鎴�
https://github.com/medcl/elasticsearch-analysis-ik
涓嬭浇鍚庡湪elastic/plugins涓嬪垱寤篿k鏂囦欢澶癸紝灏嗕笅杞界殑瑙ｅ帇锛岀劧鍚庢牴鐩綍涓嬫墍鏈夋枃浠舵嫹璐濊繘ik鏂囦欢澶癸紝閲嶆柊鍚姩鏈嶅姟锛屽畬鎴愬畨瑁�
PUT /mytest
{
聽 "mappings": {
聽 聽 "iktest":{
聽 聽 聽 聽 "properties": {
聽 聽 聽 聽 聽 聽 "content": {
聽 聽 聽 聽 聽 聽 聽 聽 "type": "text",
聽 聽 聽 聽 聽 聽 聽 聽 "analyzer": "ik_max_word",
聽 聽 聽 聽 聽 聽 聽 聽 "search_analyzer": "ik_max_word"
聽 聽 聽 聽 聽 聽 }
聽 聽 聽 聽 }
聽 聽 }
聽 }
}
GET /mytest/_analyze
{
聽 "text":"鐢熸椿鎬绘槸杩欐牱锛屽氨鍍忕尗鐨勮姳鑹茶繙涓嶅彧涓�绉�",
聽 "tokenizer": "ik_max_word"
}
GET /testik/ik/_search
{
聽 "query": {
聽 聽 "match": {
聽 聽 聽 "content": "鑷"
聽 聽 }
聽 },
聽 "highlight" : {
聽 聽 聽 聽 "pre_tags" : ["
聽 聽 聽 聽 "post_tags" : ["", ""],
聽 聽 聽 聽 "fields" : {
聽 聽 聽 聽 聽 聽 "content" : {}
聽 聽 聽 聽 }
聽 聽 }
}
25.鍏充簬elastic棰勫厛瀹氫箟鏂囨。缁撴瀯鍚庢坊鍏ユ柊瀛楁浼氳嚜鍔ㄦ坊鍔犻棶棰榚lastic缁欏嚭鏄庣‘鐨勫洖绛�
PUT /my_index
{
聽 聽 "mappings": {
聽 聽 聽 聽 "my_type": {
聽 聽 聽 聽 聽 聽 "dynamic":"strict",
聽 聽 聽 聽 聽 聽 "properties": {
聽 聽 聽 聽 聽 聽 聽 聽 "title":聽 { "type": "text"},
聽 聽 聽 聽 聽 聽 聽 聽 "stash":聽 {
聽 聽 聽 聽 聽 聽 聽 聽 聽 聽 "type":聽 聽 "object",
聽 聽 聽 聽 聽 聽 聽 聽 聽 聽 "dynamic":聽 true
聽 聽 聽 聽 聽 聽 聽 聽 }
聽 聽 聽 聽 聽 聽 }
聽 聽 聽 聽 }
聽 聽 }
}
榛樿鎯呭喌涓嬶紝褰撳湪鏂囨。涓壘鍒板厛鍓嶆湭瑙佺殑瀛楁鏃讹紝Elasticsearch浼氬皢鏂板瓧娈垫坊鍔犲埌绫诲瀷鏄犲皠涓�俹bject 閫氳繃灏哾ynamic鍙傛暟璁剧疆涓篺alse锛堝拷鐣ユ柊瀛楁锛夋垨璁剧疆涓簊trict锛堝湪閬囧埌鏈煡瀛楁鏃舵姏鍑哄紓甯革級锛屽彲浠ュ湪鏂囨。鍜岀骇鍒鐢ㄦ琛屼负
Elastic+logstash+head绠�鍗曚粙缁�
涓�. 姒傝堪
ElasticSearch鏄竴涓熀浜嶭ucene鐨勬悳绱㈡湇鍔″櫒銆傚畠鎻愪緵浜嗕竴涓垎甯冨紡澶氱敤鎴疯兘鍔涚殑鍏ㄦ枃鎼滅储寮曟搸锛屽熀浜嶳ESTful web鎺ュ彛銆�
浜岋紟ElasticSearch鐨勫畨瑁呬笌绠�鍗曢厤缃�
1.1. 搴旇濮嬬粓杩愯鏈�鏂扮増鏈殑 Java 铏氭嫙鏈猴紙 JVM 锛夛紝 闄ら潪 Elasticsearch 缃戠珯涓婂彟鏈夎鏄庛�偮� Elasticsearch锛� 鐗瑰埆鏄� Lucene锛屾槸涓�涓珮瑕佹眰鐨勮蒋浠躲�侺ucene 鐨勫崟鍏冩祴璇曞拰闆嗘垚娴嬭瘯缁忓父鏆撮湶鍑� JVM 鏈韩鐨� bug銆傝繖浜� bug 鐨勮寖鍥翠粠杞诲井鐨勯夯鐑﹀埌涓ラ噸娈甸敊璇紝鎵�浠ワ紝鏈�濂藉敖鍙兘鐨勪娇鐢ㄦ渶鏂扮増鏈殑 JVM
1.2. 涓嬭浇ElasticSearch
涓嬭浇鍦板潃锛歨ttps://www.elastic.co/downloads/elasticsearch
濡備笅杞芥渶鏂扮増鏈鐐筎AR锛屽叾浠栫増鏈鐐瑰嚮 past releases閫夋嫨
1.3.聽 瑙ｅ帇瀹夎鍖�
1.4. 鐩綍璁茶В锛�
1. bin:鍖呭惈杩愯ElasticSearch瀹炰緥鍜岀鐞嗘彃浠剁殑涓�浜涜剼鏈�
2. Config: 涓昏鏄竴浜涜缃枃浠讹紝濡俵ogging锛宔lasticsearch.yml,jvm
3. Lib:鍖呭惈鐩稿叧鐨勫寘鏂囦欢绛�
4. Plugins:鍖呭惈鐩稿叧鐨勬彃浠舵枃浠剁瓑
5. Logs锛氭棩蹇楁枃浠�
6. Data锛氭暟鎹瓨鏀惧湴鍧�
1.5. 闆堕厤缃惎鍔�
鐩存帴杩涘叆bin鐩綍涓嬶紝鍚姩鑴氭湰elasticsearch
./elasticsearch
浼氭樉绀哄涓嬮敊璇細
(1).java.lang.RuntimeException: can not run elasticsearch as root
杩欐槸鐢变簬elastic涔嬪墠鍥犱负root瓒呯骇鏉冮檺涓嬪惎鍔ㄥ紩鍙戣繃琛�妗堬紝鎵�鏈夊己鍒朵笉璁╁湪root涓嬪惎鍔� 銆傝В鍐虫柟妗堬細
娣诲姞elastic鍚姩鐢ㄦ埛锛�
useradd syliu
passwd syliu
chown -R syliu:syliu elasticsearch-6.1.3
su syliu #
鐒跺悗杩涘叆bin鐩綍涓嬪惎鍔�./elasticsearch
(2).max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
杩欐槸鐢变簬linux涓嬫渶澶ф墦寮�鏂囦欢鏁伴噺姣攅lastic瑕佹眰鐨勬墦寮�鏁伴噺灏�
鏌ョ湅绯荤粺鏈�澶ф墦寮�鏂囦欢鏁伴噺
- ulimit -a (鏌ョ湅)
- ulimit -n 65536(璁剧疆)
鎴栬�咃細
鍒囨崲鍒皉oot鐢ㄦ埛淇敼
vim /etc/security/limits.conf
(3).max number of threads [3889] for user [syliu] is too low, increase to at least [4096]
vi /etc/security/limits.d/90-nproc.conf
vim /etc/security/limits.conf
(4).max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
淇敼/etc/sysctl.conf閰嶇疆鏂囦欢锛�
cat /etc/sysctl.conf | grep vm.max_map_count
vm.max_map_count=262144
濡傛灉涓嶅瓨鍦ㄥ垯娣诲姞
echo "vm.max_map_count=262144" >>/etc/sysctl.conf
1.6. 鍒版,鐜閰嶇疆瀹屾垚锛岄噸鏂板惎鍔ㄩ」鐩紝椤圭洰鍚姩鎴愬姛
璁块棶http://localhost:9200 鍚庝細鏄剧ず鎴愬姛鎻愮ず
(1) name:elastic瀹炰緥鍚嶏紝璁剧疆鍙互鍦╟onfig/elasticsearch.yml涓厤缃�
(2) Version:鐗堟湰鍙�,浠son鏍煎紡琛ㄧず鐨勪竴缁勪俊鎭��
鈶� Name: 褰撳墠鐗堟湰鍙�
鈶� build_snapshot锛氭槸鍚︿粠婧愮爜鏋勫缓鑰屾潵
鈶� lucene_version锛氬熀浜巐ucene鐨勭増鏈�
1.7. 绠�鍗曢厤缃甧lastic闆嗙兢
1. cluster.name:闆嗙兢鍚嶇О銆傝缃ソ浠ュ悗浼氭牴鎹悓涓�闆嗙兢鍚嶅瓧鑷姩鍙戠幇鍚屼竴缃戞涓嬬殑鑺傜偣锛屽鏋滃湪鍚屼竴缃戞涓嬫湁澶氫釜闆嗙兢锛屽彲浠ユ牴鎹繖涓瓧娈靛尯鍒嗛泦缇�
2. Node.name:鑺傜偣鍚嶇О锛屽彲浠ヨ嚜鍔ㄧ敓鎴愯妭鐐瑰悕绉帮紝杩欓噷閰嶇疆鏄湁鍒╀簬鍒╃敤api璁块棶鍏蜂綋鐨勮妭鐐癸紝寤鸿鑷繁閰嶇疆
3. Node.master:鑺傜偣鏄惁涓簃aster涓昏妭鐐�-----姣忎釜鑺傜偣閮藉彲琚厤缃负涓昏妭鐐广�傞粯璁ゅ�间负true锛岀洰鐨勬槸鎸囧畾璇ヨ妭鐐规槸鍚︽湁璧勬牸琚�変妇鎴愪负node銆傞粯璁ら泦缇や腑绗竴鍙伴泦缇や负master銆傚鏋滆繖鍙版満鍣ㄥ嚭鐜版晠闅滐紝闆嗙兢浼氳嚜鍔ㄩ噸鏂伴�変妇
4. Node.data:璁剧疆鑺傜偣鏄惁瀛樺偍鏁版嵁锛岄粯璁や负true锛屽鏋滀笉甯屾湜瀛樺偍锛屽垯璁剧疆涓篺alse
瀹㈡埛鑺傜偣閰嶇疆锛�
Node.master:true
Node.data:false
鏁版嵁鑺傜偣閰嶇疆锛�
Node.master:false
Node.data:true
5. Network.host: 缁戝畾鐩戝惉IP,鎻愪緵澶栫晫璁块棶
6. Transport.tcp.port:璁剧疆鑺傜偣闂翠氦浜掔殑tcp绔彛
7. Discovery.zen.ping.unicast.hosts:璁剧疆闆嗙兢涓璵aster鑺傜偣鐨勫垵濮嬪垪琛�-閫氳繃杩欎簺鑺傜偣鏉ヨ嚜鍔ㄥ彂鐜版柊鍔犲叆鐨勯泦缇�
8. discovery.zen.ping_timeout锛氳缃泦缇や腑鑷姩鍙戠幇鍏朵粬鑺傜偣鏃秔ing杩炴帴瓒呮椂鏃堕棿-榛樿涓�3绉掋�傚浜庢瘮杈冨樊鐨勭綉缁滅幆澧冨彲浠ュ姞澶у�兼潵闃叉鑷姩鍙戠幇鏃跺嚭閿�
9. client.transport.ping_timeout锛氬鎴风杩炴帴ping鐨勬渶澶ц秴鏃舵椂闂�
10. bootstrap.memory_lock锛歵rue 閿佸畾鍐呭瓨锛岄槻姝㈠唴瀛樹氦鎹�
11. http.port:缁戝畾鐩戝惉ip鐨勭鍙ｅ彿锛岄粯璁�9200
12. path.data:鏁版嵁瀛樻斁浣嶇疆锛屾渶濂戒笉瑕佹斁鍦ㄩ粯璁ゅ畨瑁呯洰褰曚笅锛岃繖鏍峰嵏杞戒簡浼氬紩璧锋暟鎹涪澶憋紝鍙互淇濆瓨鍒颁笉鍚岀殑鐩綍涓嬶紝鏈�濂芥寕杞藉埌涓嶅悓鐨勭鐩樹笂
閰嶇疆鏂规硶濡備笅锛歱ath.data: /path/to/data1,/path/to/data2
鍚姩elastic浼氬彂鐜版姤閿欙細
memory locking requested for elasticsearch process but memory is not locked
瑙ｅ喅鍔炴硶锛�
vim /etc/security/limits.conf
Syliu soft memlock unlimited
Syliu hard memlock unlimited
淇敼锛�
/etc/sysctl.conf
vm.swappiness=0
鍒版锛氶厤缃畬鎴愶紝鐒跺悗灏嗘湰閰嶇疆锛屽皢鏈厤缃嫹璐濆埌鍏朵粬鑺傜偣涓嬶紝娉ㄦ剰淇敼node.name.
濡傛灉閰嶇疆鍦ㄥ悓涓�鍙版湇鍔″櫒涓嬪叾浠栬妭鐐硅繕闇�瑕佷慨鏀圭鍙ｅ彿锛屼互鍏嶇鍙ｈ鍗犵敤
涓夛紟Logstash鐨勭畝鍗曢厤缃笌瀹夎
1.1聽 Logstash鏄竴涓兘鏈夋晥杩涜鏃ュ織澶勭悊鐨勫伐鍏凤紝鍙互瀵规棩蹇楄繘琛屾敹闆嗭紝鍒嗘瀽銆傚叾鏈韩骞朵笉浜х敓鏃ュ織锛屽畠鍙槸涓�涓唴缃垎鏋愬拰杞崲宸ュ叿鐨勬棩蹇楃鐞嗗伐鍏凤紝鏄竴涓帴鏀�,澶勭悊,杞彂鏃ュ織鐨勨�欑閬撯�欍��
1.2 涓嬭浇logstash
涓嬭浇鍦板潃锛歨ttps://www.elastic.co/downloads/logstash
涓嬭浇鍘嬬缉鍖咃紝瑙ｅ帇
1.3 杩涘叆bin鐩綍涓嬪垱寤�.conf鏂囦欢浣滀负鍚姩鏂囦欢
Logstash澶勭悊浜嬩欢鏈変笁涓樁娈碉細杈撳叆Inputs,杩囨护Filters,杈撳嚭OutPuts
鍏蜂綋鍙傞槄锛歨ttp://udn.yyuap.com/doc/logstash-best-practice-cn/input/stdin.html 鎴栬�卙ttps://www.elastic.co/guide/en/logstash/current/input-plugins.html
1.鎸囧畾file涓鸿鍙栨枃浠剁殑鏂瑰紡锛�
input {
file {
type => "guoan"
path => "F:/logs/guoanjia/guoanjia/*.log"
codec => json { charset => "GBK" }
start_position => "beginning"
}
}
type: 鏍囪浜嬩欢绫诲瀷聽 闆嗘垚java鐨勬椂鍊欓渶瑕佺敤鍒般�傝繕鍙互鏍规嵁type鍊煎仛涓�浜涜繃婊ょ殑鎿嶄綔
path:涓烘煇涓粷瀵硅矾寰勬枃浠跺す涓嬫墍鏈変互.log缁撳熬鐨勬枃浠朵负鏁版嵁婧�
codec: 缂栫爜鎻掍欢锛� codec 灏辨槸鐢ㄦ潵 decode銆乪ncode 浜嬩欢鐨�
鎸囧畾杈撳叆鍒發ogstash鐨勬牸寮忎负json鏍煎紡鐨勶紝鎸囧畾瀛楃闆嗕负涓枃
start_position锛氫粠浠�涔堜綅缃紑濮嬭鍙栨枃浠舵暟鎹紝榛樿鏄粨鏉熶綅缃�
鎶婅繖涓瀹氭敼鎴� "beginning"锛宭ogstash 杩涚▼灏变粠澶村紑濮嬭鍙�
闄ゆ涔嬪杩樻湁涓�浜涢厤缃湁鏃跺�欓渶瑕佹墜鍔ㄩ厤涓婏細
discover_interval锛歭ogstash 姣忛殧澶氫箙鍘绘鏌ヤ竴娆¤鐩戝惉鐨� path 涓嬫槸鍚︽湁鏂版枃浠躲�傞粯璁ゅ�兼槸 15 绉�
exclude锛氫笉鎯宠鐩戝惉鐨勬枃浠跺彲浠ユ帓闄ゅ嚭鍘�
stat_interval锛歭ogstash 姣忛殧澶氫箙妫�鏌ヤ竴娆¤鐩戝惉鏂囦欢鐘舵�侊紙鏄惁鏈夋洿鏂帮級锛岄粯璁ゆ槸 1 绉�
******娉ㄦ剰锛歴tart_position 浠呭湪璇ユ枃浠朵粠鏈鐩戝惉杩囩殑鏃跺�欒捣浣滅敤銆傚鏋� sincedb 鏂囦欢涓凡缁忔湁杩欎釜鏂囦欢鐨� inode 璁板綍浜嗭紝閭ｄ箞 logstash 渚濈劧浼氫粠璁板綍杩囩殑 pos 寮�濮嬭鍙栨暟鎹�傛墍浠ラ噸澶嶆祴璇曠殑鏃跺�欐瘡鍥為渶瑕佸垹闄� sincedb 鏂囦欢銆�
2. 閫氳繃TCP濂楁帴瀛楄鍙栦簨浠躲��
input {
tcp {
host =>127.0.0.1
port => 8999
mode => "server"
ssl_enable => false
type => "guoan3"
codec => json { charset => "GBK" }
}
}
鍙互鎺ュ彈鏉ヨ嚜瀹㈡埛绔殑杩炴帴鎴栬繛鎺ュ埌鏈嶅姟鍣紝鍏蜂綋鍙栧喅浜巑ode銆�
缂栬緫
mode
鍊煎彲浠ユ槸浠讳綍鐨勶細server锛宑lient
榛樿鍊间负 "server"
鎿嶄綔妯″紡銆俿erver鐩戝惉瀹㈡埛绔繛鎺ワ紝 client杩炴帴鍒版湇鍔″櫒銆�
ssl_enable => false
鍚敤SSL锛堝繀椤昏缃叾浠杝sl_閫夐」鎵嶈兘鐢熸晥锛�
host:
鍊肩被鍨嬫槸瀛楃涓�
榛樿鍊间负 "0.0.0.0"
褰撴ā寮忔槸server锛屽湴鍧�瑕佺洃鍚�傚綋妯″紡鏄痗lient锛岃繛鎺ュ埌鐨勫湴鍧�
鐒跺悗 闇�瑕佸湪SpringBoot椤圭洰涓姞鍏� 鐩稿簲鐨勪緷璧栵細
鏃ュ織閰嶇疆鏂囦欢(涓嶆槸SpringBoot涔熷彲浠�)锛�
鍒╃敤鏈ā寮忚繘琛屾暟鎹殑瀵煎叆闇�瑕佸湪springboot 閰嶇疆鏂囦欢涓姞鍏�
server:
context-path: /agenthouseCutomer
port:8084
max-http-header-size: 10024
3. 鍒╃敤logstash鐨� logstash-input-jdbc杩涜涓巑ysql鏁版嵁鐨勫悓姝�
杩涘叆logstash鐨刡in鐩綍涓嬪畨瑁呮彃浠�
Logstash-plugin install logstash-input-jdbc
Input閰嶇疆
4. 鍏充簬filter鐨勯厤缃�
4.1 grok filter
127.0.0.1 POST /logs/getLog 12345 0.123
grok { match => {鈥渕essage鈥�=>鈥滐紖{IP锛歝lient}锛厈WORD锛歮ethod}锛厈URIPATHPARAM:url}锛厈NUMBER锛歝ount}锛厈NUMBER锛歮oney}鈥潁 }
鍙互灏嗚偖鑴忕殑闈炵粨鏋勫寲鏃ュ織鏁版嵁瑙ｆ瀽鎴愮粨鏋勫寲鍜屽彲鏌ヨ鐨勬暟鎹�
Grok浣滀负鏁版嵁缁撴瀯鍖栧伐鍏�,鍦╨ogstash涓粯璁や笂鐧句釜grok鍙橀噺锛屽彲浠ョ洿鎺ユ嬁鏉ヤ娇鐢�
閫傚悎瀵箂yslog.apache log绛夊彲璇绘棩蹇楄繘琛屽垎鏋�
4.2 kv filter
瀵逛簬璇稿key-value 杩欑閿�煎鏁版嵁杩涜鍒嗘瀽
濡傦細user=鍥藉畨1&url=111&method=main&ip=124.65.164.98&args=null
kv {
source => "message"
field_split => "&?"
}
瑙ｆ瀽鍑烘潵鏍煎紡
"method": "main",
ip": "124.65.164.98",
"message": "user=鍥藉畨1&url=111&method=main&ip=124.65.164.98&args=null",
"url": "111",
"args": "null",
"user": "鍥藉畨1"
4.3 geoip
geoip {
source => "ip"
fields => ["city_name", "country_code2", "country_name", "latitude", "longitude", "region_name"]
remove_field => ["[geoip][country_code3]", "[geoip][region_name]", "[geoip][continent_code]", "[geoip][timezone]", "[geoip][ip]"]
}
鏍规嵁涓婇潰kv瑙ｆ瀽鍑烘潵鐨刬p瀛楁杩涜鑾峰彇璇︾粏鍦扮悊淇℃伅鍜岀粡绾害
鍝嶅簲锛�
"geoip": {
"city_name": "Beijing",
"latitude": 39.9289,
"country_code2": "CN",
"country_name": "China",
"longitude": 116.3883
},
5. 鍏充簬output閰嶇疆
5.1 elasticsearch
elasticsearch {
hosts => "127.0.0.1:9200"
#index => "guoan88881234"
index => "guoerror-%{+YYYY.MM.dd}"
user => elastic
password => changeme
retry_on_conflict => 5
}
灏唋ogstash涓暟鎹緭鍏ュ埌elasticsearch涓細
hosts:ip鍔犵鍙ｅ彿锛屾垨鑰呭煙鍚�
index锛氫唬琛ㄦ瘡澶╀互guoerror-寮�澶寸敓鎴愮储寮�
user:elasticsearch鐨勭敤鎴峰悕
password锛氫唬琛╡lasticsearch鐨勫瘑鐮�
retry_on_conflict锛氳緭鍑洪噸璇曟鏁�
5.2 email
email {
port => "25" 绔彛
address => "smtp.126.com" 鍦板潃
username => "[email protected]"聽 鐢ㄦ埛
password => "*****" 瀹㈡埛绔巿鏉冨瘑鐮� 閫傜敤浜庣櫥褰曚互涓嬫湇鍔�: POP3/IMAP/SMTP/Exchange/CardDAV/CalDAV鏈嶅姟
authentication => "plain" 鍥哄畾
use_tls => false
from => "[email protected]"
subject=> "Warning: 绯荤粺鍑洪敊浜�!%{@timestamp}-%{type}-%{logger_name}"
to => "[email protected]"
via => "smtp"
body => "%{stack_trace}"
聽 聽 聽 聽 }
6. 鍚姩logstash
閰嶇疆瀹屾垚鍚巜indons涓嬪垱寤� run.bat
鍐欏叆logstash -f guoan.conf 鍚姩鍛戒护
鎴栫洿鎺ュ懡浠よlogstash -f guoan.conf鍚姩
-f 鍙傛暟鎰忔�� 鈥樻枃浠垛��
浣滅敤鏄寚瀹歭ogstash閰嶇疆鏂囦欢
-e 鍙傛暟鐨勪綔鐢ㄤ负鎵ц聽 濡傦細logstash -e 鈥榠nput{stdin{}} output{stout{}}鈥�
闄ゆ涔嬪锛岃繕鏈� -t,-l,-w,-p,-v
鍚姩涔嬪墠娉ㄦ剰锛�
鍦╟onfig/logstash.yml涓厤缃細
#pipeline绠￠亾绾跨▼鏁帮紝瀹樻柟寤鸿鏄瓑浜嶤PU鍐呮牳鏁�
pipeline.workers: 8
#pipeline绠￠亾瀹為檯output鏃剁殑绾跨▼鏁帮紝涓�鑸皬浜庢垨绛変簬绠￠亾绾跨▼鏁帮紝寤鸿绛変簬cpu鍐呮牳鏁�
pipeline.output.workers: 8
#鍗曚釜宸ヤ綔绾跨▼鍦ㄥ皾璇曟墽琛岃繃婊ゅ櫒鍜岃緭鍑轰箣鍓嶆敹闆嗙殑鏈�澶т簨浠舵暟锛岄粯璁�125锛涖�� 鏁板�艰秺澶э紝澶勭悊鍒欓�氬父鏇撮珮鏁堬紝浣嗗鍔犱簡鍐呭瓨寮�閿�锛涖��鏌愪簺纭欢閰嶇疆瑕佹眰閫氳繃璁剧疆LS_HEAP_SIZE鍙橀噺鏉ュ鍔燡VM鍫嗗ぇ灏忥紝浠ラ伩鍏嶄娇鐢ㄦ閫夐」瀵艰嚧鎬ц兘涓嬮檷锛涖��姝ゅ弬鏁扮殑鍊艰秴杩囨渶浣宠寖鍥翠細瀵艰嚧鐢变簬棰戠箒鐨勫瀮鍦惧洖鏀舵垨涓庡唴瀛樹笉瓒冲紓甯哥浉鍏崇殑JVM宕╂簝鑰屽鑷存�ц兘涓嬮檷锛涖��璋冩暣pipeline.batch.size璁剧疆澶у皬鍙皟鏁村彂閫佸埌Elasticsearch鐨勬壒閲忚姹傜殑澶у皬
pipeline.batch.size: 3000
#姝よ缃皟鏁碙ogstash绠￠亾鐨勫欢杩燂紝榛樿5锛涖��娴佹按绾挎壒澶勭悊寤惰繜鏄疞ogstash鍦ㄥ綋鍓嶇閬撳伐浣滅嚎绋嬩腑鎺ユ敹鍒颁簨浠跺悗绛夊緟鏂版秷鎭殑鏈�澶ф椂闂达紙姣锛夛紱銆�鍦ㄦ鏃堕棿杩囧悗锛孡ogstash寮�濮嬫墽琛岃繃婊ゅ櫒鍜岃緭鍑�.Logstash鍦ㄦ帴鏀朵簨浠跺拰鍦ㄨ繃婊ゅ櫒涓鐞嗚浜嬩欢涔嬮棿绛夊緟鐨勬渶澶ф椂闂存槸pipeline.batch.delay鍜宲ipeline.batch.size璁剧疆鐨勪箻绉�
pipeline.batch.delay: 100
鍥涳紟Head 閰嶇疆
1.鍦╡lasticsearch.yml閰嶇疆璺ㄥ煙鏀寔
http.cors.enabled: true
http.cors.allow-origin: "*"
2. 涓嬭浇 elasticsearch-head
https://github.com/mobz/elasticsearch-head
3. Npm install
Npm run start
鎴栬�呭幓google鍟嗗簵鎼滅储ElasticSearch-head鎻掍欢锛岀洿鎺ュ畨瑁呭嵆鍙娇鐢�
鎴戝緱鐧惧害缃戠洏锛歨ttps://pan.baidu.com/s/1dkA1m9XECKSfpH65-qmE3Q
涓嬭浇瀹屼箣鍚庡姞鍏ヨ胺姝屾墿灞曠▼搴忓嵆鍙娇鐢�
浜旓紟ElasticSearch浼樺寲寤鸿
1.璋冨ぇ绯荤粺鐨勨�滄渶澶ф墦寮�鏂囦欢鏁扳�濓紝寤鸿32k鐢氳嚦64k
- ulimit -a (鏌ョ湅)
- ulimit -n 32000(璁剧疆)
2. 淇敼閰嶇疆鏂囦欢璋冩暣es鐨刯vm鍐呭瓨澶у皬
淇敼bin/elasticsearch.in.sh涓璄S_MIN_MEM鍜孍S_MAX_MEM鐨勫ぇ灏忥紝寤鸿璁剧疆涓�鏍峰ぇ锛岄伩鍏嶅钩鍑＄殑鍒嗛厤鍐呭瓨锛屾牴鎹湇鍔″櫒鍐呭瓨澶у皬閰嶇疆锛屼竴鑸垎閰�60%宸﹀彸锛堥粯璁�256锛�
3.璁剧疆mlockall鏉ラ攣瀹氳繘绋嬬殑鐗╃悊鍐呭瓨鍦板潃
- 閬垮厤浜ゆ崲锛坰wapped锛夋潵鎻愰珮鎬ц兘
-淇敼鏂囦欢 conf/elasticsearch.yml 涓� bootstrap.mlockall:true
4.鍒嗙墖澶氱殑璇濆彲浠ユ彁鍗囧缓绔嬬储寮曠殑鑳藉姏锛�5-20涓瘮杈冨悎閫傦紝榛樿5涓�
濡傛灉鍒嗙墖鏁拌繃灏戞垨杩囧锛岄兘浼氬鑷存绱㈡瘮杈冩參銆傚垎鐗囨暟杩囧浼氬鑷存绱㈡椂鎵撳紑姣旇緝澶氱殑鏂囦欢锛屽彟澶栦篃浼氬鑷村鍙版湇鍔″櫒涔嬮棿鐨勯�氳杩囨參銆傝�屽垎鐗囨暟杩囧皯浼氬鑷村崟涓垎鐗囩储寮曡繃澶э紝鎵�浠ユ绱㈤�熷害浼氭參涓嬫潵銆傚缓璁崟涓垎鐗囨渶澶氬偍瀛�20G宸﹀彸鐨勭储寮曟暟鎹紝鎵�浠ワ紝鍒嗙墖鏁�=鏁版嵁鎬婚噺/20G
5.鍓湰澶氱殑璇濓紝鍙互鎻愬崌鎼滅储鑳藉姏锛屼絾鏄鏋滆缃緢澶氬壇鏈殑璇濅篃浼氬鏈嶅姟鍣ㄩ�犳垚棰濆鐨勫帇鍔涳紝鍥犱负闇�瑕佸悓姝ユ暟鎹紝鎵�浠ュ缓璁缃�2-3涓�
PUT /gakj-1 { "settings": { "number_of_shards" : 5, "number_of_replicas" : 2 } }
6.瀹氭椂浼樺寲绱㈠紩锛宻egment瓒婂锛屾煡璇㈤�熷害瓒婂樊
max_num_segments=1
7.鍘绘帀mappings _all鍩燂紝绱㈠紩鍒涘缓鏃堕粯璁や細鏈塤all鍩燂紝杩欐牱浼氱粰鏌ヨ甯︽潵鏂归潰锛屼絾鏄鍔犵储寮曟椂闂村拰绱㈠紩灏哄害
- "_all":{"enabled":"false"}
8.log杈撳嚭鐨勬按骞抽粯璁や负trace,鍗虫煡璇㈣秴杩�500ms鍗充负鎱㈡煡璇紝灏辫鎵撳嵃鏃ュ織锛岄�犳垚cpu鍜宮em,io璐熻浇寰堥珮锛屾妸log杈撳嚭姘村钩鏀逛负info锛屽彲鍑忚交鏈嶅姟鍣ㄧ殑鍘嬪姏
-淇敼ES_HOME/conf/logging.yaml鏂囦欢
-鎴栦慨鏀笶S_HOME/conf/elasticsearch.yaml
鍏紟Elastic瀹夎x-pack
瀹夎锛氳繘鍏lastic 鐨刡in涓�
鍛戒护锛歟lasticsearch-plugin install x-pack
瀹夎瀹屾垚鍚庡惎鍔╡lasticsearch
鍒濆鍖栧瘑鐮侊細
bin/x-pack/setup-passwords auto
鎴栬�呮墜鍔ㄨ缃瘑鐮侊細
bin/x-pack/setup-passwords interactive
娉ㄦ剰锛歺-pack鐩墠閮ㄥ垎鍔熻兘鏄敹璐圭殑锛岃瘯鐢ㄦ湡鏄竴涓湀锛屽熀纭�鐗堥渶瑕佺敵璇凤紝鍩虹鐗堝厤璐�
涓冿紟ElasticSearch瀛︿範绗旇鍙婃煡璇㈣娉�
1.淇℃伅妫�绱笌缁撴灉杩囨护
閫氳繃_source 杩斿洖鎸囧畾鐨勬绱㈠瓧娈甸泦
2.kibana 涓� sort鎺掑簭涓嶅噯纭殑闂
3.match 妫�绱㈠瓙鍙ュ拰_all鍙傛暟鐨勪娇鐢�
鍖归厤鍏ㄩ儴瀛楁鐨勬绱�
4.match_phrase 鍦ㄦ寚瀹氬瓧娈典腑鍖归厤鐗瑰畾鐨勬绱㈠幓
5.瀵逛簬缁欏畾鐨勫唴瀹癸紝query_string鏌ヨ浣跨敤鏌ヨ瑙ｆ瀽鍣ㄦ潵鏋勯�犲疄闄呯殑鏌ヨ
鍏朵腑^2琛ㄧず鍖呭惈鎵撳紑闂ㄩ攣涓斿叾鏉冮噸涓�2
6.prefix 鎵惧埌鏌愪釜瀛楁浠ョ粰瀹氬墠缂�寮�澶寸殑鏂囨。
7.range 鑼冨洿鏌ヨ
gte
澶т簬鎴栫瓑浜�
gt
姣�...鏇存
lte
灏忎簬鎴栫瓑浜�
lt
灏戜簬
8.more_like_this,fuzzy_like_this
鏌ヨ寰楀埌涓庢墍鎻愪緵鐨勬枃鏈浉浼肩殑鏂囨。
fields
like
9.multi_match 澶氬瓧娈典腑缁勫悎鎼滅储
10.杩囨护鏌ヨ
鏌ヨ瀛樺湪鏌愪釜瀛楁鐨勬棩蹇�
11.query_string 鏌ヨ
12.cat鍛戒护
1.GET /_cat
鍒楀嚭鎵�鏈塩at鍛戒护
2.GET /_cat/health?v
妫�鏌ラ泦缇ゅ仴搴风姸鍐�
3.GET /_cat/nodes?v
鑺傜偣缁熻
4.GET /_cat/indices
鑾峰彇鍏ㄩ儴鑺傜偣
13 璁剧疆鏈�灏忚妭鐐癸紝闃叉鑴戣锛岄伩鍏嶄袱涓富鑺傜偣鍚屾椂瀛樺湪涓�涓泦缇や腑
discovery.zen.minimum_master_nodes: 2
閫氳繃api淇敼姝ｅ湪杩愯鐨勮妭鐐癸紝璁剧疆鏈�灏忚妭鐐�
PUT /_cluster/settings
{
聽 聽 聽 聽 聽 "persistent" : {
聽 聽 聽 聽 聽 聽 聽 聽 "discovery.zen.minimum_master_nodes" : 2
聽 聽 }
聽 聽 聽 聽 }
14.闆嗙兢鎭㈠鏂归潰鐨勯厤缃�
闃绘 Elasticsearch 鍦ㄥ瓨鍦ㄨ嚦灏� 8 涓妭鐐癸紙鏁版嵁鑺傜偣鎴栬�� master 鑺傜偣锛変箣鍓嶈繘琛屾暟鎹仮澶�
gateway.recover_after_nodes: 8
閰嶇疆闆嗙兢涓簲璇ユ湁澶氬皯鑺傜偣锛屼互鍙婃垜浠効鎰忎负杩欎簺鑺傜偣绛夊緟澶氶暱鏃堕棿
gateway.expected_nodes: 10聽 聽 聽 gateway.recover_after_time: 5m
璁剧疆鍗曟挱鑺傜偣鍒楄〃锛岄槻姝㈣妭鐐规棤鎰忓姞鍏ラ泦缇�
discovery.zen.ping.unicast.hosts: ["host1", "host2:port"]
闃叉鍚屼竴涓垎鐗囷紙shard锛夌殑涓诲壇鏈瓨鍦ㄥ悓涓�涓墿鐞嗘満涓�
cluster.routing.allocation.same_shard.host: true
鍏佽 JVM 閿佷綇鍐呭瓨锛岀姝㈡搷浣滅郴缁熶氦鎹㈠嚭鍘�
bootstrap.mlockall: true
15.閰嶇疆elasticsearch涓嶈鍘婚厤缃瀮鍦惧洖鏀跺拰绾跨▼姹�
璁剧疆鍫嗗唴瀛�
export ES_HEAP_SIZE=1g 鎴� ./bin/elasticsearch -Xmx1g -Xms1g
纭繚鍫嗗唴瀛樻渶灏忓�硷紙 Xms 锛変笌鏈�澶у�硷紙 Xmx 锛夌殑澶у皬鏄浉鍚岀殑锛岄槻姝㈢▼搴忓湪杩愯鏃舵敼鍙樺爢鍐呭瓨澶у皬锛� 杩欐槸涓�涓緢鑰楃郴缁熻祫婧愮殑杩囩▼銆�
璁剧疆绂佺敤鍐呭瓨浜ゆ崲 鎴� 闄嶄綆swappiness 鐨勫��
sudo swapoff -a聽 聽 vm.swappiness = 1
16.鏌ヨ鏌愪釜瀛楁蹇呴』鍖呭惈浠�涔堬紝蹇呴』鎺掗櫎浠�涔�
17.es鍐呯疆REST鎺ュ彛
1./index/_search 鎼滅储鎸囧畾绱㈠紩涓嬬殑鏁版嵁
2./_aliases聽 鑾峰彇鎴栨搷浣滅储寮曠殑鍒悕
3./index/type/聽 鍒涘缓鎴栨搷浣滅被鍨�
4./index/_mapping 鍒涘缓鎴栨搷浣渕apping
5./index/_settings聽 鍒涘缓鎴栨搷浣滆缃紙number_of_shards 鏄笉鍙洿鏀圭殑锛�
6./index/_open 鎵撳紑鎸囧畾琚叧闂殑绱㈠紩
7./index/_close 鍏抽棴鎸囧畾绱㈠紩
8./index/refresh聽 鍒锋柊绱㈠紩锛堜娇鏂板姞鍐呭瀵圭储寮曞彲瑙侊紝涓嶄繚璇佹暟鎹鍐欏叆纾佺洏锛�
9./index/_flush聽 鍒锋柊绱㈠紩锛堜細瑙﹀彂lucene鎻愪氦锛�
18.term鏌ヨ鎸囧畾鐨勭簿纭��
19.浣跨敤 constant_score 鏌ヨ浠ラ潪璇勫垎妯″紡鏉ユ墽琛� term 鏌ヨ骞朵互涓�浣滀负缁熶竴璇勫垎
鍐呴儴杩囨护鍣ㄧ殑鎿嶄綔锛�
1.鏌ユ壘鍖归厤鏂囨。
term鏌ヨ鍦ㄥ�掓帓绱㈠紩涓煡鎵惧寘鍚term鐨勬墍鏈夋枃妗ｏ紝瀹冩弿杩颁簡鍝釜鏂囨。浼氬寘鍚 term 銆傚尮閰嶆枃妗ｇ殑鏍囧織浣嶆槸 1
2.鍒涘缓bitset锛堜竴涓寘鍚�0鍜�1鐨勬暟缁勶級
3.杩唬bitset(s)
涓�鏃︿负姣忎釜鏌ヨ鐢熸垚浜� bitsets 锛孍lasticsearch 灏变細寰幆杩唬 bitsets 浠庤�屾壘鍒版弧瓒虫墍 鏈夎繃婊ゆ潯浠剁殑鍖归厤鏂囨。鐨勯泦鍚堛�傛墽琛岄『搴忔槸鍚彂寮忕殑锛屼絾涓�鑸潵璇村厛杩唬绋�鐤忕殑 bitset 锛堝洜涓哄畠鍙互鎺掗櫎鎺夊ぇ閲忕殑鏂囨。锛�
4.澧為噺浣跨敤璁℃暟
Elasticsearch 鑳藉缂撳瓨闈炶瘎鍒嗘煡璇粠鑰岃幏鍙栨洿蹇殑璁�
20.bool杩囨护鍣�
must
鎵�鏈夌殑璇彞閮� 蹇呴』锛坢ust锛� 鍖归厤锛屼笌 AND 绛変环銆�
must_not
鎵�鏈夌殑璇彞閮� 涓嶈兘锛坢ust not锛� 鍖归厤锛屼笌 NOT 绛変环銆�
should
鑷冲皯鏈変竴涓鍙ヨ鍖归厤锛屼笌 OR 绛変环銆�
21.terms鏌ユ壘澶氫釜绮剧‘鍊�
22.鍒涘缓鏂囨。绱㈠紩
PUT /gakj
{
聽 "settings": {
聽 聽 "number_of_shards": 5
聽 },
聽 "mappings": {
聽 聽 "guoanjia":{
聽 聽 聽 "properties": {
聽 聽 聽 聽 "id" :
聽 聽 聽 聽 {
聽 聽 聽 聽 聽 "type" : "long",
聽 聽 聽 聽 聽 "store": true
聽 聽 聽 聽 聽
聽 聽 聽 聽 },聽
聽 聽 聽 聽 "url" :
聽 聽 聽 聽 {
聽 聽 聽 聽 聽 "type" : "text",
聽 聽 聽 聽 聽 "index" :true,
聽 聽 聽 聽 聽 "omit_norms": "true",
聽 聽 聽 聽 聽 "store": true
聽 聽 聽 聽 }
聽 聽 聽 }
聽 聽 }
聽 }
}
23銆傚垱寤虹储寮曟椂娣诲姞榛樿杩囨湡鏃堕棿
PUT my_index聽
{聽
聽 "mappings": {聽
聽 聽 "my_type": {聽
聽 聽 聽 "_ttl": {聽
聽 聽 聽 聽 "enabled": true,聽
聽 聽 聽 聽 "default": "5m"聽
聽 聽 聽 }聽
聽 聽 }聽
聽 }聽
}聽
濡傛灉涓嶆坊鍔爐tl杩囨湡鍙傛暟鍒欐枃妗ｉ粯璁や笉杩囨湡
聽 d
澶�
聽 ms锛堥粯璁わ級
姣
m
鍒嗛挓
24.ik鍒嗚瘝鍣ㄩ泦鎴�
https://github.com/medcl/elasticsearch-analysis-ik
涓嬭浇鍚庡湪elastic/plugins涓嬪垱寤篿k鏂囦欢澶癸紝灏嗕笅杞界殑瑙ｅ帇锛岀劧鍚庢牴鐩綍涓嬫墍鏈夋枃浠舵嫹璐濊繘ik鏂囦欢澶癸紝閲嶆柊鍚姩鏈嶅姟锛屽畬鎴愬畨瑁�
PUT /mytest
{
聽 "mappings": {
聽 聽 "iktest":{
聽 聽 聽 聽 "properties": {
聽 聽 聽 聽 聽 聽 "content": {
聽 聽 聽 聽 聽 聽 聽 聽 "type": "text",
聽 聽 聽 聽 聽 聽 聽 聽 "analyzer": "ik_max_word",
聽 聽 聽 聽 聽 聽 聽 聽 "search_analyzer": "ik_max_word"
聽 聽 聽 聽 聽 聽 }
聽 聽 聽 聽 }
聽 聽 }
聽 }
}
GET /mytest/_analyze
{
聽 "text":"鐢熸椿鎬绘槸杩欐牱锛屽氨鍍忕尗鐨勮姳鑹茶繙涓嶅彧涓�绉�",
聽 "tokenizer": "ik_max_word"
}
GET /testik/ik/_search
{
聽 "query": {
聽 聽 "match": {
聽 聽 聽 "content": "鑷"
聽 聽 }
聽 },
聽 "highlight" : {
聽 聽 聽 聽 "pre_tags" : ["
聽 聽 聽 聽 "post_tags" : ["", ""],
聽 聽 聽 聽 "fields" : {
聽 聽 聽 聽 聽 聽 "content" : {}
聽 聽 聽 聽 }
聽 聽 }
}
25.鍏充簬elastic棰勫厛瀹氫箟鏂囨。缁撴瀯鍚庢坊鍏ユ柊瀛楁浼氳嚜鍔ㄦ坊鍔犻棶棰榚lastic缁欏嚭鏄庣‘鐨勫洖绛�
PUT /my_index
{
聽 聽 "mappings": {
聽 聽 聽 聽 "my_type": {
聽 聽 聽 聽 聽 聽 "dynamic":"strict",
聽 聽 聽 聽 聽 聽 "properties": {
聽 聽 聽 聽 聽 聽 聽 聽 "title":聽 { "type": "text"},
聽 聽 聽 聽 聽 聽 聽 聽 "stash":聽 {
聽 聽 聽 聽 聽 聽 聽 聽 聽 聽 "type":聽 聽 "object",
聽 聽 聽 聽 聽 聽 聽 聽 聽 聽 "dynamic":聽 true
聽 聽 聽 聽 聽 聽 聽 聽 }
聽 聽 聽 聽 聽 聽 }
聽 聽 聽 聽 }
聽 聽 }
}
榛樿鎯呭喌涓嬶紝褰撳湪鏂囨。涓壘鍒板厛鍓嶆湭瑙佺殑瀛楁鏃讹紝Elasticsearch浼氬皢鏂板瓧娈垫坊鍔犲埌绫诲瀷鏄犲皠涓�俹bject 閫氳繃灏哾ynamic鍙傛暟璁剧疆涓篺alse锛堝拷鐣ユ柊瀛楁锛夋垨璁剧疆涓簊trict锛堝湪閬囧埌鏈煡瀛楁鏃舵姏鍑哄紓甯革級锛屽彲浠ュ湪鏂囨。鍜岀骇鍒鐢ㄦ琛屼负