Centos7 安装ElasticSearch7 及其Kibana环境安装

1,打开网址 https://www.elastic.co/cn/

点击产品选择ElasticSearch+kibana

image.png

点击了解详情

image.png

点击下载

image.png

点击选择历史版本

image.png

选择7.3.0

image.png

选择对应的系统版本（注Kibana下载和es 一样 要求ElasticSeach的版本一致 ）

image.png

下载后上传到服务器

启动es
./elasticsearch
会遇到

image.png

不能用root权限启动
1，创建用户elasticsearch
adduser elasticsearch
2，创建用户密码，需要输入两次
passwd elasticsearch
3,将对应的文件夹权限赋给该用户
chown -R elasticsearch elasticsearch-7.3.0
4,切换到elasticsearch用户
su elasticsearch
5，进入启动目录启动
elasticsearch-7.3.0/bin 使用后台启动方式：./elasticsearch -d

报java环境不兼容的错误

elasticsearch-7.3.0 要求用jdk11 但是是开发环境是jdk11需要手动隔离

image.png

1下载jdk11的包
https://www.oracle.com/java/technologies/javase-jdk11-downloads.html#license-lightbox
2解压到指定位置
tar -xzvf jdk-11.0.6_linux-x64_bin.tar.gz /data/softwear/es
3,修改配置文件
cd dao elasticsearch/bin
vi elasticsearch

#配置自己的jdk11
export JAVA_HOME=/data/softweare/jdk-11.0.6
export PATH=$JAVA_HOME/bin:$PATH

#添加jdk判断
if [ -x "$JAVA_HOME/bin/java" ]; then
        JAVA="/data/softweare/jdk-11.0.6/bin/java"
else
        JAVA=`which java`
fi

完整的代码

#配置自己的jdk11
export JAVA_HOME=/data/softweare/jdk-11.0.6
export PATH=$JAVA_HOME/bin:$PATH



source "`dirname "$0"`"/elasticsearch-env

if [ -z "$ES_TMPDIR" ]; then
  ES_TMPDIR=`"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.TempDirectory`
fi

ES_JVM_OPTIONS="$ES_PATH_CONF"/jvm.options
JVM_OPTIONS=`"$JAVA" -cp "$ES_CLASSPATH" org.elasticsearch.tools.launchers.JvmOptionsParser "$ES_JVM_OPTIONS"`
ES_JAVA_OPTS="${JVM_OPTIONS//\$\{ES_TMPDIR\}/$ES_TMPDIR}"

#添加jdk判断
if [ -x "$JAVA_HOME/bin/java" ]; then
        JAVA="/data/softweare/jdk-11.0.6/bin/java"
else
        JAVA=`which java`
fi

# manual parsing to find out, if process should be detached
if ! echo $* | grep -E '(^-d |-d$| -d |--daemonize$|--daemonize )' > /dev/null; then
  exec \
    "$JAVA" \
    $ES_JAVA_OPTS \
    -Des.path.home="$ES_HOME" \
    -Des.path.conf="$ES_PATH_CONF" \
    -Des.distribution.flavor="$ES_DISTRIBUTION_FLAVOR" \
    -Des.distribution.type="$ES_DISTRIBUTION_TYPE" \
    -Des.bundled_jdk="$ES_BUNDLED_JDK" \
    -cp "$ES_CLASSPATH" \
    org.elasticsearch.bootstrap.Elasticsearch \
    "$@"
else
  exec \
    "$JAVA" \
    $ES_JAVA_OPTS \
    -Des.path.home="$ES_HOME" \
    -Des.path.conf="$ES_PATH_CONF" \
    -Des.distribution.flavor="$ES_DISTRIBUTION_FLAVOR" \
    -Des.distribution.type="$ES_DISTRIBUTION_TYPE" \
    -Des.bundled_jdk="$ES_BUNDLED_JDK" \
    -cp "$ES_CLASSPATH" \
    org.elasticsearch.bootstrap.Elasticsearch \
    "$@" \
    <&- &
  retval=$?
  pid=$!
  [ $retval -eq 0 ] || exit $retval
  if [ ! -z "$ES_STARTUP_SLEEP_TIME" ]; then
    sleep $ES_STARTUP_SLEEP_TIME
  fi
  if ! ps -p $pid > /dev/null ; then
    exit 1
  fi
  exit 0
fi

exit $?

启动es

[esuser@localhost bin]$ ./elasticsearch
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-08-02T15:33:27,994][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [32.6gb], net total_space [35.7gb], types [rootfs]
[2019-08-02T15:33:28,014][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [1015.6mb], compressed ordinary object pointers [true]
[2019-08-02T15:33:28,017][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [8zGCWQhBS3OpVShqjSgU-w], cluster name [es-application]
[2019-08-02T15:33:28,017][INFO ][o.e.n.Node               ] [node-1] version[7.2.0], pid[1545], build[default/tar/508c38a/2019-06-20T15:54:18.811730Z], OS[Linux/3.10.0-862.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13]
[2019-08-02T15:33:28,017][INFO ][o.e.n.Node               ] [node-1] JVM home [/opt/jdk-11.0.1]
[2019-08-02T15:33:28,018][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-5247006010869253587, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Dio.netty.allocator.type=unpooled, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/local/src/elasticsearch, -Des.path.conf=/usr/local/src/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]

到这里我们又看到另一个警告

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
这是提醒你 cms 垃圾收集器在 jdk9 就开始被标注为 @deprecated

看下JDK11支持的垃圾回收器
HotSpot Virtual Machine Garbage Collection Tuning Guide

修改jvm.options
-XX:+UseConcMarkSweepGC
改为：-XX:+UseG1GC
完整配置如下

## JVM configuration

################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms1g
-Xmx1g

################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################

## GC configuration
## 原有的注释了，添加了G1回收器
## -XX:+UseConcMarkSweepGC
-XX:+UseG1GC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

## G1GC Configuration
# NOTE: G1GC is only supported on JDK version 10 or later.
# To use G1GC uncomment the lines below.
# 10-:-XX:-UseConcMarkSweepGC
# 10-:-XX:-UseCMSInitiatingOccupancyOnly
# 10-:-XX:+UseG1GC
# 10-:-XX:InitiatingHeapOccupancyPercent=75

## DNS cache policy
# cache ttl in seconds for positive DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.ttl; set to -1 to cache forever
-Des.networkaddress.cache.ttl=60
# cache ttl in seconds for negative DNS lookups noting that this overrides the
# JDK security property networkaddress.cache.negative ttl; set to -1 to cache
# forever
-Des.networkaddress.cache.negative.ttl=10

## optimizations

# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch

## basic

# explicitly set the stack size
-Xss1m

# set to headless, just in case
-Djava.awt.headless=true

# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8

# use our provided JNA always versus the system one
-Djna.nosys=true

# turn off a JDK optimization that throws away stack traces for common
# exceptions because stack traces are important for debugging
-XX:-OmitStackTraceInFastThrow

# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0

# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true

-Djava.io.tmpdir=${ES_TMPDIR}

## heap dumps

# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError

# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=data

# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=logs/hs_err_pid%p.log

## JDK 8 GC logging

8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:logs/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m

# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m
# due to internationalization enhancements in JDK 9 Elasticsearch need to set the provider to COMPAT otherwise
# time/date parsing will break in an incompatible way for some date patterns and locals
9-:-Djava.locale.providers=COMPAT

重新启动
bin/elasticsearch -d
然而没成功依然报错

image.png

solutions：
先要切换到root用户；
然后可以执行以下命令，设置 vm.max_map_count ，但是重启后又会恢复为原值。

sysctl -w vm.max_map_count=262144

持久性的做法是在 /etc/sysctl.conf 文件中修改 vm.max_map_count 参数：

sysctl -p

然后重启 还是报错

[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

修改
elasticsearch.yml
取消注释保留一个节点
cluster.initial_master_nodes: ["node-1"]
这个的话，这里的node-1是上面一个默认的记得打开就可以了

继续重启 重启成功但9200访问不了
修改yml
network.host: 0.0.0.0
http.port: 9200

# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.

再次重启 重启不了发现服务已经重启 lsof -i:9200也没东西
ps -ef|grep elastic
kill -9 所有的进程
然后再su elasticsearch下

./elasticsearch -d 成功启动

//访问9200

image.png

设置ES的密码

1设置elasticsearch.yml 添加如下

cluster.name: esbest
node.name: node127-9200
node.master: true
path.data: /data/softweare/es/elasticsearch-7.3.0/data
path.logs: /data/softweare/es/elasticsearch-7.3.0/logs

bootstrap.memory_lock: false
bootstrap.system_call_filter: false

http.port: 9200
transport.tcp.port: 9300

network.host: 0.0.0.0
discovery.type: single-node

xpack.security.enabled: true

重启es
cd 到bin 目录下执行
./elasticsearch-setup-passwords interactive （有可能提示java不兼容参考上边的解决办法） 记下你设置的密码

设置es最大分页限制

PUT /productinfokeypro/_settings
{ "index.max_result_window" :"1000000"}

简直吐血的经历！！！！

2020-09-07更新

优化es性能
报这样的错误type=circuit_breaking_exception, reason=[parent] Data too large, data for [] would be [1663333912/1.5gb], which is larger than the limit of [1662202675/1.5gb], real usage: [1659250080/1.5gb], new bytes reserved: [4083832/3.8mb], usages [request=1359872/1.2mb, fielddata=0/0b, in_flight_requests=17784334/16.9mb, accounting=551349/538.4kb]]]
在jvm.options 里设置

image.png

在 elasticsearch.yml 里设置

# 避免发生OOM，发生OOM对集群影响很大的
indices.breaker.total.limit: 60%
# 有了这个设置，最久未使用（LRU）的 fielddata 会被回收为新数据腾出空间
indices.fielddata.cache.size: 40%
# fielddata 断路器默认设置堆的  作为 fielddata 大小的上限。
indices.breaker.fielddata.limit: 60%
# request 断路器估算需要完成其他请求部分的结构大小，例如创建一个聚合桶，默认限制是堆内存
indices.breaker.request.limit: 60%
# 父级断路器是否应考虑实际内存使用情况（true）或仅考虑子级断路器保留的数量（false）。默认为true
# 整个父级断点器的启动限制，如果indices.breaker.total.use_real_memory: false 默认为JVM堆的70％ of the JVM heap., 否则为95% of the JVM heap
indices.breaker.total.use_real_memory: false
#关闭circuit检查:
#indices.breaker.type: none