ElasticSearch2.4.0单机版及集群安装

 

编写不易，转载请注明（ http://shihlei.iteye.com/blog/2327971）！

概述

（一）简介

          基于lucene 实现的近实时搜索服务，Restful 接口操作

     

（二）名词解释

Indics：索引库，相当于RDBMS的 数据库，整体控制分片（shard）和副本（Replica），一旦创建，分片不能改变。

 

Document type：索引类型

 

Document：索引记录，由唯一ID区分，ID决定Shard位置

 

Filed：一条索引记录中的组成字段，有类型概念，通过Mapping控制类型，是否分词，索引等

 

一 单机版安装

 

（一）环境

 

环境要求：

     java 7 以上

环境：

    CentOS Linux release 7.2.1511 (Core)

    jre：1.8.0_101

版本：

     elasticsearch-2.4.0.tar.gz

 

核心配置文件：

                elasticsearch-2.4.0/config/elasticsearch.yml : yml协议的默认配置文件

                elasticsearch-2.4.0/config/logging.xml ：yml协议格式的日志配置文件

（二）安装

1）创建用户

 

groupadd es useradd -r -g es -s /bin/bash es chown -hR es:es /opt/elasticsearch-2.4.0

 

注：由于不能用Root用户启动，否则报java.lang.RuntimeException，所以创建用户

 

Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root.     at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:94)     at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:160)     at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)     at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35) Refer to the log for complete error details.

 

2）解压缩

    

tar -xvf elasticsearch-2.4.0.tar.gz

 

3）启动

 

cd /opt/elasticsearch-2.4.0 bin/elasticsearch

 

日志：

 

[2016-09-22 08:45:21,503][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: your kernel is buggy and you should upgrade [2016-09-22 08:45:21,608][INFO ][node                     ] [Deacon Frost] version[2.4.0], pid[205], build[ce9f0c7/2016-08-29T09:14:17Z] [2016-09-22 08:45:21,608][INFO ][node                     ] [Deacon Frost] initializing ... [2016-09-22 08:45:21,977][INFO ][plugins                  ] [Deacon Frost] modules [reindex, lang-expression, lang-groovy], plugins [], sites [] [2016-09-22 08:45:21,994][INFO ][env                      ] [Deacon Frost] using [1] data paths, mounts [[/ (none)]], net usable_space [45gb], net total_space [59gb], spins? [possibly], types [aufs] [2016-09-22 08:45:21,994][INFO ][env                      ] [Deacon Frost] heap size [998.4mb], compressed ordinary object pointers [unknown] [2016-09-22 08:45:23,147][INFO ][node                     ] [Deacon Frost] initialized [2016-09-22 08:45:23,148][INFO ][node                     ] [Deacon Frost] starting ... [2016-09-22 08:45:23,251][INFO ][transport                ] [Deacon Frost] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300} [2016-09-22 08:45:23,255][INFO ][discovery                ] [Deacon Frost] elasticsearch/6TSRp8FFQ3aXBPtkS4sROQ [2016-09-22 08:45:26,316][INFO ][cluster.service          ] [Deacon Frost] new_master {Deacon Frost}{6TSRp8FFQ3aXBPtkS4sROQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received) [2016-09-22 08:45:26,341][INFO ][http                     ] [Deacon Frost] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200} [2016-09-22 08:45:26,343][INFO ][node                     ] [Deacon Frost] started [2016-09-22 08:45:26,359][INFO ][gateway                  ] [Deacon Frost] recovered [0] indices into cluster_state

 

     注1 ：后台启动方法：bin/elasticsearch -d

     注2 :  这里绑定了两个端口

               9200：默认的Http端口，用于Restful API的调用

               9300：默认TCP端口，用户ES Node 之间的通讯，以及TCP 基本调用 Restful API ， Java API TransportClient 需要连接该端口才能访问

     注3：默认绑定127.0.0.1 本机访问，

               远程访问需修改 ${ES_HOME}/config/elasticsearch.yml 的 network.host绑定指定的IP

               

# ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 0.0.0.0

 

 

（三）验证

 

浏览器：http://127.0.0.1:9200 返回

 

{   "name" : "Abomination",   "cluster_name" : "elasticsearch",   "version" : {     "number" : "2.4.0",     "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55",     "build_timestamp" : "2016-08-29T09:14:17Z",     "build_snapshot" : false,     "lucene_version" : "5.5.2"   },   "tagline" : "You Know, for Search" }

 

由节点名称等，说明安装成功

 

（四）简单操作

 

（1）插入：

 

$ curl -X PUT 'http://localhost:9200/indexdb/type/1' -d ' > { >    "name":"test", >    "desc":"test" > } > ' {"_index":"indexdb","_type":"type","_id":"1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":

 

（2）查询：

 

$ curl -XGET 'http://localhost:9200/indexdb/type/1' {"_index":"indexdb","_type":"type","_id":"1","_version":1,"found":true,"_source": {    "name":"test",    "desc":"test" } }

 

（五）关闭

for pid in `jps | grep Elasticsearch | awk '{print $1}'` do kill -9 $pid done;

 

（六）常用插件安装

（1）Head：

方便对es进行各种操作的客户端（ https://github.com/mobz/elasticsearch-head）

 

bin/plugin install mobz/elasticsearch-head

 

访问：

 

http://127.0.0.1:9200/_plugin/head

 

 

（2）bigdesk：

监控es状态的插件（ https://github.com/lukas-vlcek/bigdesk）

 

bigdesk 的源码开发在 2015 年夏天就已经停止，所以默认是无法支持 Elasticsearch 2.x 的监控的

 

 

二 集群搭建

 

（一）概述

（1）ES  Node 的角色：

Master Node :  通过选举出来，控制集群元数据和状态，压力较小，CRUD不需要通过 （ node.master=true）

 

Data Node : 拥有数据，并控制数据的CRUD，有查询压力 （node.data=true）

 

Client Node： 只能路由，转发响应的请求给Master Node 或者Data Node ，做聚合操作，可以分担部分Data Node 的压力，成本合适，可以创建。

 

Trib Node : 联合节点，控制到多个集群的访问

 

（2）架构

ES 的所有Node 节点是对等的，可以服务于客户端的查询，索引请求。结点通过选举成为Master，维持Cluster 状态，Master 所有的Node 周知。

 

（二）集群搭建

 

（1）架构规划

 

 

说明：客户端通过 ClientNode 访问集群，ClientNode开放tcp，http访问，

          所有DataNode置于后端，只开发TCP进行通讯

 

（2）目录规划

 

ES_HOME : /opt/elasticsearch-2.4.0

 

结点名称	配置文件目录	数据目录	日志目录	pid目录
ClientNode	cluster/config/ClientNode	cluster/data/ClientNode	cluster/logs/ClientNode	cluster/ClientNode.pid
DataNode1	cluster/config/DataNode1	cluster/data/DataNode1	cluster/logs/DataNode1	cluster/DataNode1.pid
DataNode2	cluster/config/DataNode2	cluster/data/DataNode2	cluster/logs/DataNode2	cluster/DataNode2.pid
DataNode3	cluster/config/DataNode3	cluster/data/DataNode3	cluster/logs/DataNode3	cluster/DataNode3.pid

 

 

（3）端口规划

 

结点名称	Http端口	TCP端口
ClientNode	9200	9300
DataNode1	9201	9301
DataNode2	9202	9302
DataNode2	9203	9303

 

（4）安装

步骤1：拷贝配置文件到约定目录：

 

cd /opt/elasticsearch-2.4.0 mkdir -p cluster/config/ClientNode && cp config/logging.yml cluster/config/ClientNode mkdir -p cluster/config/DataNode1 && cp config/logging.yml cluster/config/DataNode1 mkdir -p cluster/config/DataNode2 && cp config/logging.yml cluster/config/DataNode2 mkdir -p cluster/config/DataNode3 && cp config/logging.yml cluster/config/DataNode3

 

 

步骤2：编辑个结点配置文件：

 

特别注意：ES的集群是通过自发现机制，需要cluster.name配置相同才会自动加入相同的集群，这里集群统一命名为ES_CLUSTER

下面的配置文件已经过删减，可以直接替换elasticsearch.yml

 

ClientNode：

ClientNode ：node.master , node.data标志都为false，并开启http访问

 

vi cluster/config/ClientNode/elasticsearch.yml

 

# ---------------------------------- Cluster ----------------------------------- cluster.name: ES_CLUSTER # ------------------------------------ Node ------------------------------------ node.name: ClientNode http.enabled: true node.master: false node.data: false # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /opt/elasticsearch-2.4.0/cluster/data/ClientNode # # Path to log files: # path.logs: /opt/elasticsearch-2.4.0/cluster/logs/ClientNode # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 0.0.0.0 # # Set a custom port for HTTP: # http.port: 9200 # # Set a custom port for TCP: transport.tcp.port: 9300   # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.zen.ping.unicast.hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]

 

DataNode1：

可选举成为Master 的DataNode ： node.master,node.data都为true，由于客户端方位从ClientNode口出，http.enabled禁掉了下面同

 

vi cluster/config/DataNode1/elasticsearch.yml

 

# ---------------------------------- Cluster ----------------------------------- cluster.name: ES_CLUSTER # ------------------------------------ Node ------------------------------------ node.name: DataNode1 http.enabled: false node.master: true node.data: true # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /opt/elasticsearch-2.4.0/cluster/data/DataNode1 # # Path to log files: # path.logs: /opt/elasticsearch-2.4.0/cluster/logs/DataNode1 # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 0.0.0.0 # # Set a custom port for HTTP: # http.port: 9201 # # Set a custom port for TCP: transport.tcp.port: 9301   # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.zen.ping.unicast.hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]

 

DataNode2

 

vi cluster/config/DataNode2/elasticsearch.yml

 

# ---------------------------------- Cluster ----------------------------------- cluster.name: ES_CLUSTER # ------------------------------------ Node ------------------------------------ node.name: DataNode2 http.enabled: false node.master: true node.data: true # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /opt/elasticsearch-2.4.0/cluster/data/DataNode2 # # Path to log files: # path.logs: /opt/elasticsearch-2.4.0/cluster/logs/DataNode2 # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 0.0.0.0 # # Set a custom port for HTTP: # http.port: 9202 # # Set a custom port for TCP: transport.tcp.port: 9302   # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.zen.ping.unicast.hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]

 

DataNode3

 

vi cluster/config/DataNode3/elasticsearch.yml

 

# ---------------------------------- Cluster ----------------------------------- cluster.name: ES_CLUSTER # ------------------------------------ Node ------------------------------------ node.name: DataNode3 http.enabled: false node.master: true node.data: true # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /opt/elasticsearch-2.4.0/cluster/data/DataNode3 # # Path to log files: # path.logs: /opt/elasticsearch-2.4.0/cluster/logs/DataNode3 # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 0.0.0.0 # # Set a custom port for HTTP: # http.port: 9203 # # Set a custom port for TCP: transport.tcp.port: 9303   # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.zen.ping.unicast.hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]

 

特别注意：——坑：

 

transport.tcp.port   参考：http://knktc.com/2016/06/10/elasticsearch-multiple-instances/?utm_source=tuicool&utm_medium=referral 由于到了2.x版本之后，ES取消了默认的广播模式来发现master节点，需要使用该配置来指定发现master节点。这个配置在单机双实例的配置中需要特别注意下，因为习惯上我们配置时并未指定master节点的tcp端口，如果实例的transport.tcp.port配置为9301，那么实例启动后会认为discovery.zen.ping.unicast.hosts中指定的主机tcp端口也是9301，可能导致这些节点无法找到master节点。因此在该配置中需要指定master节点提供服务的tcp端口。   所以配置了 discovery.zen.ping.unicast.hosts: ["127.0.0.1:9301","127.0.0.1:9302","127.0.0.1:9303"]

 

 

（5）启动

启动命令为：

 

bin/elasticsearch -Des.path.conf=cluster/config/ClientNode  -d -p cluster/ClientNode.pid bin/elasticsearch -Des.path.conf=cluster/config/DataNode1  -d -p cluster/DataNode1.pid bin/elasticsearch -Des.path.conf=cluster/config/DataNode2  -d -p cluster/DataNode2.pid bin/elasticsearch -Des.path.conf=cluster/config/DataNode3  -d -p cluster/DataNode3.pid

 

（6）状态：

 

bash-4.2$ jps -lm 3815 org.elasticsearch.bootstrap.Elasticsearch start -Des.path.conf=cluster/config/DataNode1 -d -p cluster/DataNode1.pid 4152 sun.tools.jps.Jps -lm 4089 org.elasticsearch.bootstrap.Elasticsearch start -Des.path.conf=cluster/config/ClientNode -d -p cluster/ClientNode.pid 3836 org.elasticsearch.bootstrap.Elasticsearch start -Des.path.conf=cluster/config/DataNode2 -d -p cluster/DataNode2.pid 3885 org.elasticsearch.bootstrap.Elasticsearch start -Des.path.conf=cluster/config/DataNode3 -d -p cluster/DataNode3.pid

 

ClientNode

 

{   "name" : "ClientNode",   "cluster_name" : "ES_CLUSTER",   "version" : {     "number" : "2.4.0",     "build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55",     "build_timestamp" : "2016-08-29T09:14:17Z",     "build_snapshot" : false,     "lucene_version" : "5.5.2"   },   "tagline" : "You Know, for Search" }

 

 

（7）停止

 

for pid in `jps | grep Elasticsearch | awk '{print $1}'` do kill -9 $pid done;