Es集群部署
目录
组件全家套
版本说明
主机准备
1.解压安装
2.运行环境配置
2.1修改每个节点linux系统限制
2.2 修改每个节点 linux 系统配置
2.3 调整vm.max_map_count的大小
2.4 重启验证配置
3. 配置ES
3.1 每个节点创建ES用户，ES不能使用root启动
3.2 每个节点创建数据目录，日志目录
3.3 配置es文件目录，跨域请求,修改每个节点特殊的内容
3.4修改JVM启动参数
3.5 配置开机启动，启动服务
组件全家套
- LogStash数据同步组件相关
Lostash同步Mysql数据到Elasticsearch（一）服务介绍及环境配置
Lostash同步Mysql数据到ElasticSearch（二）logstash脚本配置和常见坑点
Lostash同步Mysql数据到Elasticsearch（三）Elasticsearch模板与索引设置
Lostash同步Mysql数据到Elasticsearch（四）通过kibana辅助查看同步情况- kibana开发工具相关
Kibana开发工具安装
版本说明
| 组件 | 版本 | 备注 |
| Elasticsearch | elasticsearch-7.7.1-linux-x86_64.tar.gz |
主机准备
| 主机名 | ip | |
| node144 | 172.168.9.144 | |
| node145 | 172.168.9.146 | |
| node146 | 172.168.9.146 |
1.解压安装
注意：Elasticsearch需要安装JDK环境，配置前请先行安装JDK环境
tar -zxvf elasticsearch-7.7.1-linux-x86_64.tar.gz -C /usr/local/
2.运行环境配置
2.1修改每个节点linux系统限制
vim /etc/security/limits.conf
* hard nproc 4096 * soft nproc 4096 * hard nofile 655300 * soft nofile 655300 * soft memlock unlimited * hard memlokc unlimited
2.2 修改每个节点 linux 系统配置
编辑文件，查找如下参数进行配置 vim /etc/systemd/system.conf
DefaultLimitNOFILE=65536
DefaultLimitNPROC=32000
DefaultLimitMEMLOCK=infinity
ulimit -a #查看重启前系统配置
2.3 调整vm.max_map_count的大小
sysctl -a|grep vm.max_map_count #查看当前系统配置大小，可以设置为262144 建议设置为262144,最大655360 vim /etc/sysctl.conf
vm.max_map_count=655360
sysctl -p #生效配置
2.4 重启验证配置
ulimit -a #重启后查看配置是否生效 sysctl -a|grep vm.max_map_count #重启后查看配置是否生效
3. 配置ES
3.1 每个节点创建ES用户，ES不能使用root启动
| groupadd elasticsearch useradd elasticsearch -g elasticsearch |
3.2 每个节点创建数据目录，日志目录
| mkdir -p /mnt/elk/elasticsearch/{data,logs} chown -R elasticsearch. /mnt/elk/elasticsearch chown -R elasticsearch. /usr/local/elasticsearch-7.7.1/ |
3.3 配置es文件目录，跨域请求,修改每个节点特殊的内容
vi /usr/local/elasticsearch-7.7.1/config/elasticsearch.yml
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # #cluster.name: my-application cluster.name: es_cluster # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # #node.name: node-1 node.name: es_znyg104 node.master: true node.data: true # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /mnt/elk/elasticsearch/data # # Path to log files: # path.logs: /mnt/elk/elasticsearch/logs # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # #network.host: 192.168.0.1 network.host: 0.0.0.0 # # Set a custom port for HTTP: # #http.port: 9200 http.port: 9200 http.cors.enabled: true http.cors.allow-origin: "*" http.cors.allow-credentials: true # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # #discovery.seed_hosts: ["host1", "host2"] discovery.zen.minimum_master_nodes: 2 discovery.zen.ping_timeout: 3s discovery.seed_hosts: ["172.168.9.104", "172.168.9.105", "172.168.9.203", "172.168.9.204", "172.168.9.205"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # #cluster.initial_master_nodes: ["node-1", "node-2"] cluster.initial_master_nodes: ["es_znyg104", "es_znyg204"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true transport.tcp.port: 9300
3.4修改JVM启动参数
vi /usr/local/elasticsearch-7.7.1/config/jvm.options
主要修改的是运行内存，日志文件路径等，按照自己定义的进行处理
| -xms2g -xmx2g -XX:HeapDumpPath=/mnt/elk/elasticsearch -XX:ErrorFile=/mnt/elk/elasticsearch/hs_err_pid%p.log 8:-xloggc:/mnt/elk/elasticsearch/logs/gc.log 9-:-xlog:gc*,gc+age=trace,safepoint:file=/mnt/elk/elasticsearch/logs/gc.log:utctime,pid,tags :filecount=32,filesize=64m |
3.5 配置开机启动，启动服务
cd /etc/init.d vi elasticsearch
#!/bin/bash
#chkconfig: 345 63 37
#description: elasticsearch
#processname: elasticsearch-7.7.1
export ES_HOME=/usr/local/elasticsearch-7.7.1 # elasticsearch所在目录
case $1 in
start)
su elasticsearch<
保存退出，赋予执行权限
chmod +x elasticsearch
添加到开机启动任务
chkconfig --add elasticsearch
启动，重启，停止
service elasticsearch start/stop/restart
遇到could not find java in bundled jdk at /usr/local/elasticsearch-7.7.1/jdk/bin/java，查看自己的es安装目录是否是root权限，修改授权为ES
验证服务启动
ps aux|grep elasticsearch
curl 127.0.0.1:9200