Elasticsearch 集群安装文档
- 1. 说明
- 2. 环境准备
- 2.1. 集群列表
- 2.2. 最大文件打开数
- 2.3. Java 的 JDK
- 3. Elasticsearch 的安装
- 3.1. 从官网下载安装包
- 3.2. 解压到指定的目录下
- 3.3. 角色分配
- 3.4. Elasticsearch.yml 设置
- 3.5. jvm.options 配置
- 3.6. Elasticsearch 启动
- 3.7. Elasticsearch 停止
- 3.8. 查看 Elasticsearch 集群状态
- 3.9. 测试
- 4. Kibana 安装
- 4.1. 下载安装包
- 4.2. 解压
- 4.3. 在各 Elasticsearch 主机下安装 x-pack
- 4.4. 在 Kibana 下安装 x-pack
- 4.5. 根据需要修改 Kibana 配置
- 4.6. 启动 Kibana
- 4.7. 停止
- 4.8. 访问
- 5. Search Guard 安装
- 6. 安装过程中遇到的问题
1. 说明
本安装文档针对 ElasticSearch5.4.3 进行编写。
2. 环境准备
2.1. 集群列表
172.19.7.91 du-es-1
172.19.7.92 du-es-2
172.19.7.93 du-es-3
172.19.7.94 du-es-4
172.19.7.95 du-es-5
2.2. 最大文件打开数
1.在 /etc/security/limits.conf 添加如下内容
* soft nofile 655350
* hard nofile 655350
* soft nproc 655350
* hard nproc 655350
2.注释掉 /etc/security/limits.d/90-nproc.conf 里面的两行
#* soft nproc 1024
#root soft nproc unlimited
3.直接设置文件打开数(避免重启)
ulimit -n 655350
4.修改 max_map_count（操作系统默认限制的内存映射数是比较低的，可能会引起内存溢出异常。）
在 /etc/sysctl.conf 添加
vm.max_map_count=655360
然后执行 sysctl -p
验证：
sysctl -a | grep “vm.max_map_count”
2.3. Java 的 JDK
1．jdk 的版本
因为 Elasticsearch5.4.3 至少需要 Java 8，本文使用的是 oracle jdk 1.8.0_131
可以从 JDK 官网直接下载 jdk-8u131-linux-x64.tar.gz。
2．Java 的垃圾回收器算法
采用 G1 收集器
3. Elasticsearch 的安装
3.1. 从官网下载安装包
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.4.3.tar.gz
3.2. 解压到指定的目录下
tar –zxvf elasticsearch-5.4.3.tar.gz –C /srv/app/es/
3.3. 角色分配
| ip | 主机名 | 实例目录 | 角色 | 数据目录 | 日志目录 |
|---|---|---|---|---|---|
| 172.19.7.91 | du-es-1 | Instance1 | Master:true nodedata:false | /data1/es/data，/data2/es/data，/data3/es/data，/data4/es/data | /data1/es/logs， |
| 172.19.7.91 | du-es-1 | Instance2 | Master:false nodedata:false | /data1/es/data，/data2/es/data，/data3/es/data，/data4/es/data | /data1/es/logs， |
| 172.19.7.91 | du-es-1 | Instance3 | Master:false nodedata:true | /data1/es/data，/data2/es/data，/data3/es/data，/data4/es/data | /data1/es/logs， |
| 172.19.7.91 | du-es-1 | Instance4 | Master: false nodedata:true | /data1/es/data，/data2/es/data，/data3/es/data，/data4/es/data | /data1/es/logs， |
| 172.19.7.91 | du-es-1 | Instance5 | Master: false nodedata:true | /data1/es/data，/data2/es/data，/data3/es/data，/data4/es/data | /data1/es/logs， |
| ip | 主机名 | 实例目录 | 角色 | 数据目录 | 日志目录 |
|---|---|---|---|---|---|
| 172.19.7.92 | du-es-2 | Instance1 | Master:true nodedata:false | /data1/es/data，/data2/es/data，/data3/es/data，/data4/es/data | /data1/es/logs， |
| 172.19.7.92 | du-es-2 | Instance[2-4] | Master:false nodedata:false | /data1/es/data，/data2/es/data，/data3/es/data，/data4/es/data | /data1/es/logs， |
| 172.19.7.93 | du-es-3 | Instance1 | Master:trune nodedata:true | /data1/es/data，/data2/es/data，/data3/es/data，/data4/es/data | /data1/es/logs， |
| 172.19.7.93 | du-es-3 | Instance[2-4] | Master: false nodedata:true | /data1/es/data，/data2/es/data，/data3/es/data，/data4/es/data | /data1/es/logs， |
| 172.19.7.94 | du-es-4 | Instance[1-4] | Master: false nodedata:true | /data1/es/data，/data2/es/data，/data3/es/data，/data4/es/data | /data1/es/logs， |
| 172.19.7.95 | du-es-2 | Instance[1-4] | Master: false nodedata:true | /data1/es/data，/data2/es/data，/data3/es/data，/data4/es/data | /data1/es/logs， |
端口配置
| 集群名称 | ip | 主机名 | 实例 | 节点实例名称 | http.port | Transport.tcp.port |
|---|---|---|---|---|---|---|
| du-es | 172.19.7.91 | du-es-1 | instance1 | du-es-1-1 | 9211 | 9311 |
| 172.19.7.91 | du-es-1 | Instance2 | du-es-1-2 | 9212 | 9312 | |
| 172.19.7.91 | du-es-1 | Instance3 | du-es-1-3 | 9213 | 9313 | |
| 172.19.7.91 | du-es-1 | Instance4 | du-es-1-4 | 9214 | 9314 | |
| 172.19.7.91 | du-es-1 | instance5 | du-es-1-5 | 9215 | 9315 |
其它机器都可以作为数据节点，配置请参照 172.19.7.91
3.4. Elasticsearch.yml 设置
同一个主机上第i个实例配置，注所有的参数配置，参数名:空格值，此处的空格一定要有。每个实例 instance 目录下都要有一个 elasticsearch.yml 文件。
[srvadmin@du-es-1 config]$ more elasticsearch.yml.default
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ————————————————— Cluster —————————————————-
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name:du-es
#
# —————————————————— Node ——————————————————
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: ${HOSTNAME}-i
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#下面4种情况只能选择一个。
#1、每台机器请参照上述角色进行配置
node.master:true
node.data:false
node.ingest:false
search.remote.connect:false
#2、数据节点
node.master: false
node.data: true
node.ingest:false
search.remote.connect:false
#3、ingest节点
node.master: false
node.data: false
node.ingest: true
search.remote.connect:false
#4、协调节点
node.master: false
node.data: false
node.ingest: false
search.remote.connect: false
# —————————————————- Paths ——————————————————
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#注两个目录之间用逗号分隔且逗号后面要有空格
path.data: /data1/es/data, /data2/es/data, /data3/es/data, /data4/es/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#logs目录只能配置一个
path.logs: /data1/es/logs
#
# —————————————————- Memory —————————————————-
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#禁止内存交换，此设置生效 需要问题 4 的解决办法。
bootstrap.memory_lock: true
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ————————————————— Network —————————————————-
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
network.host: 172.19.7.91
#
# Set a custom port for HTTP:
#
#http.port: 9200
#可以为Http传输监听定制端口：
http.port: 921i
#可以定制该节点与其他节点交互的端口：
transport.tcp.port:931i
#
# For more information, consult the network module documentation.
#
# ————————————————- Discovery —————————————————
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is [“127.0.0.1”, “[::1]”]
#
#discovery.zen.ping.unicast.hosts: [“host1”, “host2”]
discovery.zen.ping.unicast.hosts: [“172.19.7.91:9311”,”172.19.7.92:9311”,”172.19.7.93:9311”]
#
# Prevent the “split brain” by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
discovery.zen.minimum_master_nodes: 2
#本地的最大节点数(包括主节点，数据节点，协调节点等节点)
#第一个机器配置如下
node.max_local_storage_nodes:5
# 其它机器配置 node.max_local_storage_nodes:4
# For more information, consult the zen discovery module documentation.
#
# ————————————————— Gateway —————————————————-
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ————————————————— Various —————————————————-
#设置是否可以通过正则或者_all删除或者关闭索引,false是允许，true是禁止
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
# ES 在内存不够 JVM 开启 swapping 的时候，表现得会很差，所以为了避免这个问题，将该属性设为 true，表示锁定 ES 所使用的内存
#请确保 ES_MIN_MEM 和 ES_MAX_MEM 的值是一样的，并且能够为、 ElasticSearch 分配足够的内在，并为系统操作保留足够的内存。
#启动会报错，实例起不来，详细错误见问题 3
#bootstrap.mlockall: true
#分片数，启动会报错，实例起不来，详细错误见问题 3
#index.number_of_shards: 200
#副本数 ，启动会报错，实例起不来，详细错误见问题 3
#index.number_of_replicas: 0
3.5. jvm.options 配置
jvm.options 只能在 config 目录下，不能放在 config/instancei 目录下的。
[srvadmin@du-es-1 config]$ more jvm.options
## JVM configuration
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms14g
-Xmx14g
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don’t tamper with them unless
## you understand what you are doing
##
################################################################
## GC configuration
-XX:+UseG1GC
-XX:G1HeapRegionSize=2
-XX:MaxGCPauseMillis=20
#-XX:ParaleGCThreads=2
#-XX:ConcGCThreads=2
## optimizations
# disable calls to System#gc
-XX:+DisableExplicitGC
# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch
## basic
# force the server VM (remove on 32-bit client JVMs)
-server
# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
-Xss1m
# set to headless, just in case
-Djava.awt.headless=true
# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8
# use our provided JNA always versus the system one
-Djna.nosys=true
# use old-style file permissions on JDK9
-Djdk.io.permissionsUseCanonicalPath=true
# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps
# ensure the directory exists and has sufficient space
#-XX:HeapDumpPath=${heap.dump.path}
## GC logging
#-XX:+PrintGCDetails
#-XX:+PrintGCTimeStamps
#-XX:+PrintGCDateStamps
#-XX:+PrintClassHistogram
#-XX:+PrintTenuringDistribution
#-XX:+PrintGCApplicationStoppedTime
# log GC status to a file with time stamps
# ensure the directory exists
#-Xloggc:${loggc}
# By default, the GC log file will not rotate.
# By uncommenting the lines below, the GC log file
# will be rotated every 128MB at most 32 times.
#-XX:+UseGCLogFileRotation
#-XX:NumberOfGCLogFiles=32
#-XX:GCLogFileSize=128M
# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.
# If documents were already indexed with unquoted fields in a previous version
# of Elasticsearch, some operations may throw errors.
#
# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided
# only for migration purposes.
#-Delasticsearch.json.allow_unquoted_field_names=true
3.6. Elasticsearch 启动
bin/elasticsearch -Epath.conf=/srv/app/elasticsearch/config/instance1 -d -p /tmp/elasticsearch_1.pid
bin/elasticsearch -Epath.conf=/srv/app/elasticsearch/config/instance2 -d -p /tmp/elasticsearch_2.pid
bin/elasticsearch -Epath.conf=/srv/app/elasticsearch/config/instance3 -d -p /tmp/elasticsearch_3.pid
bin/elasticsearch -Epath.conf=/srv/app/elasticsearch/config/instance4 -d -p /tmp/elasticsearch_4.pid
bin/elasticsearch -Epath.conf=/srv/app/elasticsearch/config/instance5 -d -p /tmp/elasticsearch_5.pid
3.7. Elasticsearch 停止
直接杀进程
kill -9 cat /tmp/elasticsearch_1.pid
3.8. 查看 Elasticsearch 集群状态
curl ‘172.19.7.91:9211/_cat/nodes?v’
curl -XGET ‘http://172.19.7.91:9211/_cluster/health?pretty‘
curl -XGET ‘http://172.19.7.91:9211/_cluster/health?level=indices&pretty‘
3.9. 测试
创建索引实例
curl -XPUT ‘http://172.19.7.91:9211/twitter‘ -d ‘{
“settings”:{
“number_of_shards”:3,
“number_of_replicas”:2
}
}’
创建文档
curl -XPUT ‘http://172.19.7.91:9211/twitter/tweet/1‘ -d ‘{
“user”:”kimchy”,
“post_date”:”2012-12-12”,
“message”:”trying out ElasticSearch!”
}’
查询文档
curl -XGET ‘http://172.19.7.91:9211/twitter/tweet/1‘
显示下面结果则表明成功了：
[srvadmin@du-es-1 elasticsearch]$ curl -XGET ‘http://172.19.7.91:9211/twitter/tweet/1‘
{“_index”:”twitter”,”_type”:”tweet”,”_id”:”1”,”_version”:1,”found”:true,”_source”:{
“user”:”kimchy”,
“post_date”:”2012-12-12”,s
“message”:”trying out ElasticSearch!”
}}
4. Kibana 安装
4.1. 下载安装包
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.4.3-linux-x86_64.tar.gz
4.2. 解压
tar -xzf kibana-5.4.3-linux-x86_64.tar.gz
4.3. 在各 Elasticsearch 主机下安装 x-pack
./elasticsearch-plugin install x-pack
#修改 Elasticsearch 配置。如果是用 ambari 安装，则修改 elasticsearch.yml.j2，重启 ambari
#如果要使用 xpack 的安全控制，可以将此属性设置为 true，用户为 elastic/changeme，
但此功能是收费的
xpack.security.enabled: false
重启 Elasticsearch
4.4. 在 Kibana 下安装 x-pack
./kibana-plugin install x-pack
4.5. 根据需要修改 Kibana 配置
kibana.yml
4.6. 启动 Kibana
./kibana &
4.7. 停止
fuser -n tcp 5601
kill -9
4.8. 访问
http:// 172.19.7.91:5601/
elastic/changeme
5. Search Guard 安装
Search Guard 是 Elasticsearch 的一个安全控件，绝大部分功能免费。但难于安装和操作使用，没有管理界面，只能在配置文件中进行操作。在安装此控制后，其它控件如 elasticsearch-head、kibana monitoring 将不可用
#如果 openssl 版本不够高，需要升级 openssl
wget https://www.openssl.org/source/old/1.0.1/openssl-1.0.1k.tar.gz
tar -zvxf openssl-1.0.1k.tar.gz
cd openssl-1.0.1k
./config shared
make
make install
mv /usr/bin/openssl /usr/bin/openssl.bak
mv /usr/include/openssl /usr/include/openssl.bak
ln -s /usr/local/ssl/bin/openssl /usr/bin/openssl
ln -s /usr/local/ssl/include/openssl/ /usr/include/openssl
cp /etc/ld.so.conf /etc/ld.so.conf.bak
echo “/usr/local/ssl/lib” >> /etc/ld.so.conf
ldconfig
ln -s /usr/local/lib64/libssl.so.1.1 /usr/lib64/libssl.so.1.1
ln -s /usr/local/lib64/libcrypto.so.1.1 /usr/lib64/libcrypto.so.1.1
#控件安装，5.4.3 是与 Elasticsearch 对应的版本
./elasticsearch-plugin install -b com.floragunn:search-guard-5:5.4.3-12
#制作证书
git clone https://github.com/floragunncom/search-guard-ssl.git
cd search-guard-ssl/example-pki-scripts
#修改 example.sh
#生成证书
./example.sh
#将 node-0-keystore.jks、truststore.jks 拷贝到 Elasticsearch 各节点的 config 目录下
#将 sgadmin-keystore.jks、truststore.jks 拷贝到 elasticsearch 各节点的控件 search guard 的 sgconfig 目录下
#修改 elasticsearch 追加配置。如果是用 ambari 安装，则修改 elasticsearch.yml.j2，重启 ambari
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: node-0-keystore.jks
searchguard.ssl.transport.keystore_password: 123456
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: 123456
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.authcz.admin_dn:
- CN=sgadmin,OU=client,O=client,L=test, C=de
#如果有安装 xpack
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.watcher.enabled: false
xpack.monitoring.exporters:
my_remote:
type: http
host: “http://{{hostname}}:9200“
auth:
username: admin
password: admin
#在各节点上运行 sgadmin，在运行前需要对脚本授权
plugins/search-guard-5/tools/sgadmin.sh -cd plugins/search-guard-5/sgconfig/ -ks plugins/search-guard-5/sgconfig/sgadmin-keystore.jks -ts plugins/search-guard-5/sgconfig/truststore.jks -kspass 123456 -tspass 123456 -nhnv —diagnose -cn es_cluster -h isunode
#重启 Elasticsearch
#访问进行验证，可以看到访问 Elasticsearch 时需要进行用户名密码验证，输入默认管理员用户 admin/admin 可进入
#用户角色和权限管理，请配置 search guard 控件安装目录下 sgconfig 中的配置文件
6. 安装过程中遇到的问题
问题 1：
main ERROR Could not register mbeans
java.security.AccessControlException:
access denied (“javax.management.MBeanTrustPermission” “register”)
解决办法：
该安装采用的是单节点多实例的方案，在 config 目录下建立多个 instancei（ i 是变量从 1 到 n）目录,每个目录下都要保证有
elasticsearch.yml,log4j2.properties 这 2 个文件。而 jvm.options 是必须要在 config 目录下的
上面报错就是因为，config 目录下没有 jvm.options 文件了。jvm.options 中有这样一个参数设置 -Dlog4j2.disable.jmx=true
问题 2：
java.lang.IllegalArgumentException: node settings must not contain any index level settings
解决办法：
即 Elasticsearch 不支持针对索引的配置，把针对索引的设置都去掉就可以了
问题 3：
java.lang.IllegalArgumentException: unknown setting [bootstrap.mlockall] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
解决办法：
bootstrap.memory_lock: true
问题 4：
[1] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2017-06-30T09:53:49,673][INFO ][o.e.n.Node ][du-es-1-4] stopping …
[2017-06-30T09:53:49,689][INFO ][o.e.n.Node ][du-es-1-4] stopped
[2017-06-30T09:53:49,689][INFO ][o.e.n.Node ][du-es-1-4] closing …
[2017-06-30T09:53:49,697][INFO ][o.e.n.Node ][du-es-1-4] closed
[2017-06-30T09:53:50,448][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
[2017-06-30T09:53:50,449][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
[2017-06-30T09:53:50,450][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
[2017-06-30T09:53:50,450][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
# allow user ‘srvadmin’ mlockall
srvadmin soft memlock unlimited
srvadmin hard memlock unlimited
[2017-06-30T09:53:50,450][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
[2017-06-30T09:53:50,511][INFO ][o.e.n.Node ][du-es-1-5] initializing …
[2017-06-30T09:53:50,580][INFO ][o.e.e.NodeEnvironment ][du-es-1-5] using [4] data paths, mounts [[/data3 (/dev/vdd1), /data2 (/dev/vdc1), /data1 (/dev/vdb1), /data4 (/dev/vde1)]], net usable_space [7.2tb], net total_space [7.6tb], spins? [possibly], types [ext4]
解决办法：
/etc/security/limits.conf
baoshan soft memlock unlimited
baoshan hard memlock unlimited
修改：
/etc/sysctl.conf
vm.swappiness=0
重启机器
问题 5：
第 2 个节点总共有 4 个实例，前 3 个实例都可以正常起来，但当第 4 个实例启动的时候报
Caused by: java.lang.IllegalStateException: handshake failed with
原因：
Jvm.options 参数内存设置过大
解决办法：
Jvm.options 参数内存设置改小即可
问题 6：
java.lang.IllegalStateException: failed to obtain node locks, tried [[/data1/es/data/du-es, /data2/es/data/du-es, /data3/es/data/du-es, /data4/es/data/du-es]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
原因：
每天机器上都有多个实例，node.max_local_storage_nodes 应该配置为本机上的实例个数
解决办法：
node.max_local_storage_nodes：4