Elasticsearch6.2、head插件、x-pack安全模块(security机制)安装

ElasticSearch简介

a.ElasticSearch是一个基于Lucene开发的搜索服务器，具有分布式多用户的能力，ElasticSearch是用Java开发的开源项目(Apache许可条款)，基于Restful Web接口，能够达到实时搜索、稳定、可靠、快速、高性能、安装使用方便，同时它的横向扩展能力非常强，不需要重启服务。
b.ElasticSearch是一个非常好用的实时分布式搜索和分析引擎，可以帮助我们快速的处理大规模数据，也可以用于全文检索，结构化搜索以及分析等。
c.目前很多网站都在使用ElasticSearch进行全文检索，例如：GitHub、StackOverflow、Wiki等。
d.ElasticSearch式建立在全文检索引擎Lucene基础上的，而Lucene是最先进、高效的开元搜索引擎框架，,它可以使你的引用程序添加索引和搜索能力，但是Lucene只是一个框架，要充分利用它的功能，我们需要很高的学习成本，而ElasticSearch使用Lucene作为内部引擎，在其基础上封装了功能强大的Restful API，让开发人员可以在不需要了解背后复杂的逻辑，即可实现比较高效的搜索。

• ElasticSearch安装
  Elasticsearch是基于java写的，所以它的运行环境中需要java的支持。

  
  [root@localhost 40]# java -version
  -bash: java: command not found
  [root@localhost 40]# wget http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.tar.gz
  [root@localhost 40]# tar xvf jdk-8u161-linux-x64.gz
  [root@localhost 40]# vim /etc/profile
  export JAVA_HOME=/opt/40/jdk1.8.0_161
  export JAVA_BIN=/opt/40/jdk1.8.0_161/bin
  export PATH=$PATH:$JAVA_HOME/bin
  export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
  export JAVA_HOME JAVA_BIN PATH CLASSPATH
  [root@localhost 40]# source /etc/profile
  [root@localhost 40]# java -version
  java version "1.8.0_161"
  Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
  Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
  [root@localhost 40]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.0.tar.gz
  [root@localhost 40]# tar zxvf elasticsearch-6.2.0.tar.gz

* ElasticSearch启动

[root@localhost 40]# cd elasticsearch-6.2.0/bin/
[2018-02-05T08:29:10,244][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.0.jar:6.2.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.0.jar:6.2.0]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.0.jar:6.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.0.jar:6.2.0]
... 6 more
[root@localhost bin]#

当使用root账户启动ES会出现以上错误信息，这是因为处于系统安装考虑的设置，由于Elasticsearch可以接收用户输入的脚本并且执行，为了系统安全考虑，不允许root账号启动，所以建议给Elasticsearch单独创建一个用户来运行Elasticsearch。

[root@localhost bin]# groupadd justin
[root@localhost bin]# useradd justin -g justin -p 51cto
[root@localhost bin]# chown -R justin:justin /opt/40/elasticsearch-6.2.0/
[root@localhost bin]# su - justin
[justin@localhost ~]$ cd /opt/40/elasticsearch-6.2.0/bin/
[justin@localhost bin]$ cp ../config/elasticsearch.yml{,.default}
[justin@localhost bin]$ vim ../config/elasticsearch.yml
54 #
55 network.host: 0.0.0.0
56 #
57 # Set a custom port for HTTP:
58 #
59 http.port: 9200
60 #
[justin@localhost bin]$ ./elasticsearch -d
[justin@localhost bin]$ ps -ef|grep -i elas
justin 13348 1 38 08:34 pts/2 00:00:06 /opt/40/jdk1.8.0_161/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.xljvX8Ss -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.home=/opt/40/elasticsearch-6.2.0 -Des.path.conf=/opt/40/elasticsearch-6.2.0/config -cp /opt/40/elasticsearch-6.2.0/lib/* org.elasticsearch.bootstrap.Elasticsearch -d
justin 13366 13282 0 08:34 pts/2 00:00:00 grep --color=auto -i elas
[justin@localhost bin]$

发现启动一会就自动关闭了。启动时候不要后台启动，这样可以打印出日志，方便找问题，等稳定了在后台启动

[justin@localhost bin]$ ./elasticsearch
[2018-02-05T11:39:47,616][INFO ][o.e.b.BootstrapChecks ] [IAWTuJo] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [3] bootstrap checks failed
[1]: max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
[2]: max number of threads [3856] for user [justin] is too low, increase to at least [4096]
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-02-05T11:39:47,643][INFO ][o.e.n.Node ] [IAWTuJo] stopping ...
[2018-02-05T11:39:47,687][INFO ][o.e.n.Node ] [IAWTuJo] stopped
[2018-02-05T11:39:47,687][INFO ][o.e.n.Node ] [IAWTuJo] closing ...
[2018-02-05T11:39:47,723][INFO ][o.e.n.Node ] [IAWTuJo] closed
[justin@localhost bin]$

切换到root用户，做如下修改

[root@localhost ~]# vim /etc/sysctl.conf
vm.max_map_count = 1048576
[root@localhost ~]# sysctl -p
[root@localhost ~]# vim /etc/security/limits.conf

• hard nofile 65536
• soft nofile 65536
  [root@localhost ~]# vim /etc/security/limits.d/20-nproc.conf
• soft nproc 65535
  root soft nproc unlimited
  [root@localhost ~]#

  再次启动

  [root@localhost ~]# su - justin
  [ES@localhost ~]$ cd /opt/elasticsearch-6.2.0/bin/
  ERROR: [1] bootstrap checks failed
  [1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
  [ES@localhost bin]$

Centos6不支持SecComp，而ES5.2.0后的版本默认bootstrap.system_call_filter为true，在elasticsearch.yml中增加bootstrap.system_call_filter为false，注意要在Memory下面: 
#bootstrap.memory_lock: true
bootstrap.system_call_filter: false
看到以下界面说明启动成功
![](https://s1.51cto.com/images/blog/201802/09/a7f945f39f7381f3096338d3b8805d67.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=)

-----
Elasticsearch Head是集群管理、数据可视化、增删改查、查询语句可视化工具，它的安装方式有两种：命令安装、包安装安装。也可以从下面地址下载https://github.com/mobz/elasticsearch-head、
https://github.com/mobz/elasticsearch-head/archive/master.zip 

[root@localhost opt]# git clone git://github.com/mobz/elasticsearch-head.git
Initialized empty Git repository in /opt/elasticsearch-head/.git/
remote: Counting objects: 4224, done.
remote: Total 4224 (delta 0), reused 0 (delta 0), pack-reused 4224
Receiving objects: 100% (4224/4224), 2.16 MiB | 46 KiB/s, done.
Resolving deltas: 100% (2329/2329), done.
[root@localhost opt]# wget https://nodejs.org/dist/v6.10.2/node-v6.10.2-linux-x64.tar.xz
[root@localhost opt]# vim /etc/profile
export PATH=$PATH::/opt/node-v8.9.4-linux-x64/bin
[root@localhost opt]# source !$
[root@localhost opt]# cd elasticsearch-head/
[root@localhost elasticsearch-head]# npm install
npm WARN [email protected] license should be a valid SPDX license expression
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: node install.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2018-02-09T07_03_39_766Z-debug.log
[root@localhost elasticsearch-head]# npm install [email protected] --ignore-scripts
[root@localhost elasticsearch-head]# npm install
[root@localhost elasticsearch-head]# npm run start

或者通过grunt来启动

[root@localhost elasticsearch-head]# npm install -g grunt-cli
[root@localhost elasticsearch-head]# grunt server

修改elasticsearch的参数
            增加新的参数，这样head插件可以访问es

[ES@localhost bin]$ pwd
/opt/elasticsearch-6.2.0/bin
[ES@localhost bin]$ vim ../config/elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: "*" #前引号前，后引号后有空格
[ES@localhost bin]$

然后通过http://10.10.2.83:9100/访问
或者直接用命令安装

[root@localhost opt]# cd /opt/elasticsearch-6.2.1/bin
[root@localhost bin]# ./plugin install mobz/elasticsearch-head

然后可以通过http://127.0.0.1:9200/_plugin/head/访问

修改head插件源码
            修改服务器监听地址:Gruntfile.js

[root@localhost elasticsearch-head]# vim Gruntfile.js
connect: {
server: {
options: {
hostname: '0.0.0.0', #增加hostname属性
port: 9100,
base: '.',
keepalive: true
}
}
}

修改连接地址：_site/app.js

[root@localhost elasticsearch-head]# vim _site/app.js
app.App = ui.AbstractWidget.extend({
defaults: {
base_uri: null
},
init: function(parent) {
this._super();
this.prefs = services.Preferences.instance();
this.base_uri = this.config.base_uri || this.prefs.get("app-base_uri") || "http://10.10.2.83:9200"; #把localhost修改成ES的服务器地址
if( this.base_uri.charAt( this.base_uri.length - 1 ) !== "/" ) {
// XHR request fails if the URL is not ending with a "/"
this.base_uri += "/";
}
if( this.config.auth_user ) {
var credentials = window.btoa( this.config.auth_user + ":" + this.config.auth_password );
$.ajaxSetup({
headers: {
"Authorization": "Basic " + credentials
}
});
}
[root@localhost elasticsearch-head]# nohup grunt server &

-----
x-pack安全模块(security机制)
只要安装好elasticsearch后，输入正确的ip地址和端口，并有head插件就可以看到索引库，所以如果别人知道IP和和端口，那么数据就完全裸奔在别人面前。安装x-pack后用户名和密码默认为elastic和changeme。
官方文档参考地址:https://www.elastic.co/guide/en/x-pack/5.4/logstash.html
1、修改Elasticsearch配置文件

[root@localhost bin]# pwd
/opt/elasticsearch-6.2.0/bin
[root@localhost bin]# ./elasticsearch-plugin install x-pack
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]ybr/>@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: plugin forks a native controller @br/>@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
This plugin launches a native controller that is not subject to the Java
security manager nor to system call filters.

Continue with installation? [y/N]y
Elasticsearch keystore is required by plugin [x-pack-security], creating...
-> Installed x-pack with: x-pack-security,x-pack-logstash,x-pack-deprecation,x-pack-watcher,x-pack-upgrade,x-pack-core,x-pack-ml,x-pack-monitoring,x-pack-graph
[root@localhost bin]#
[root@localhost elasticsearch-head]# vim /opt/elasticsearch-6.2.0/config/elasticsearch.yml
http.cors.allow-headers: "Authorization" #前引号前，后引号后有空格
[ES@localhost bin]$ ./elasticsearch -d

2、页面访问
http://10.10.2.83:9100/?auth_user=elastic&auth_password=changeme

x-pack也可以创建一个权限组，然后创建用户，给这个用户分配权限。
1.设置权限组

http://10.10.2.83:9200/_xpack/security/role/logstash_writer/
{
 "cluster": ["manage_index_templates","monitor"],
 "indices": [
    {
     "names": [ "logstash-*","indexdb*" ],
     "privileges":["write","delete","create_index"]
    }
  ]
}

2.给用户授权,这里的logstash_internal为用户名。
http://10.10.2.83:9200/_xpack/security/user/logstash_internal
{
 "password" : "changeme",
 "roles" : [ "logstash_writer"],
 "full_name" : "Internal Logstash User"
}

如果不需要该插件，也可以通过非常简单的方式删除

[root@localhost bin]# pwd
/opt/elasticsearch-6.2.0/bin
[root@localhost bin]# ./elasticsearch-plugin remove x-pack
-> removing [x-pack]...
-> preserving plugin config files [/opt/elasticsearch-6.2.0/config/x-pack] in case of upgrade; use --purge if not needed
[root@localhost bin]#

---

• 内存配置修改

配置 Elasticsearch 堆大小时需要通过 $ES_HEAP_SIZE 环境变量应用两个规则：
不要超过可用 RAM 的 50%
Lucene 能很好利用文件系统的缓存，它是通过系统内核管理的。如果没有足够的文件系统缓存空间，性能会受到影响。 此外，专用于堆的内存越多意味着其他所有使用 doc values 的字段内存越少。
不要超过 32 GB
如果堆大小小于 32 GB，JVM 可以利用指针压缩，这可以大大降低内存的使用：每个指针 4 字节而不是 8 字节。

[justin@es-2-10-0005 bin]$ vim ../config/jvm.options
#Xms represents the initial size of total heap space
#Xmx represents the maximum size of total heap space

-Xms16g
-Xmx16g

################################################################
#Expert settings

OK