Docker笔记四：Elasticsearch实例部署

• 在运行或启动elasticsearch容器前，先在宿主机上执行 sudo sysctl -w vm.max_map_count=262144：

1. 解决“ max virtual memory areas vm.maxmapcount [65530] is too low ”错误问题。
2. 解决容器中/etc/sysctl.conf只读，sysctl -w vm.max_map_count=262144无效问题。
3. 本人也尝试过在docker run 时使用--sysctl vm.max_map_count=262144选项，但提示not whitelisted。

• 关于JAVA的安装，本机采用default-jre 。参考 How To Install Java with Apt-Get on Debian 8，不要求设置环境变量CLASSPATH：

  export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

  也不要求设置环境变量PATH：

  export PATH=$JAVA_HOME/bin:$PATH

  只需要设置环境变量JAVA_HOME：红色字体部分根据具体版本进行替换

  export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

• 聚合操作： ElasticsearchDSL / Aggregation
• Elasticsearch PHP客户端：Elasticsearch-PHP
• Elasticsearch Docs：Elasticsearch Reference
• Elasticsearch指南：Elasticsearch 权威指南

 

安装Elasticsearch

1. 编写Dockerfile：

FROM ubuntu
MAINTAINER cenze <272666745@qq.com>

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
ADD conf/sources.list /etc/apt/

RUN apt-get update \
&& apt-get install -y apt-utils vim unzip default-jre

ENV ESEARCH /usr/local/elasticsearch-6.0.1

ADD packages/elasticsearch-6.0.1.zip /usr/local/
RUN  unzip /usr/local/elasticsearch-6.0.1 -d /usr/local \
&& adduser --disabled-login --disabled-password --no-create-home es

ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
ENV PATH $ESEARCH/bin:$PATH
ADD conf/elasticsearch.yml /usr/local/elasticsearch-6.0.1/config/
RUN chown -R es:es /usr/local/elasticsearch-6.0.1

EXPOSE 9200 9300

CMD ["elasticsearch"]

 

2. build镜像：

sudo docker build -t cenze/esearch -f Dockerfile-ESearch .

3. 以非root用户run容器，否则elasticsearch无法启动：

sudo docker run -d -p 9200:9200 -p 9300:9300 --name esearch --user es cenze/esearch

4. curl http://localhost:9200：

{
  "name" : "D_5lS8A",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "wS2pUZ95TC2jCRJ5GMvNsA",
  "version" : {
    "number" : "6.0.1",
    "build_hash" : "601be4a",
    "build_date" : "2017-12-04T09:29:09.525Z",
    "build_snapshot" : false,
    "lucene_version" : "7.0.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

 

安装中文分词插件ik

也可以是smartcn。ik的版本要与Elasticsearch版本匹配，本人用的6.0.1：

 

es@053b32506e8d:/$ elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.0.1/elasticsearch-analysis-ik-6.0.1.zip
es@053b32506e8d:/$ elasticsearch-plugin list
analysis-ik

 

测试中文关键词搜索

下列操作参考 https://github.com/medcl/elasticsearch-analysis-ik/

1. 创建索引：curl -X PUT http://localhost:9200/index。

2. mapping：

curl -XPOST -H'content-type: application/json' http://localhost:9200/index/fulltext/_mapping -d'
{
        "properties": {
            "content": {
                "type": "text",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_max_word"
            }
        }
    
}'

3. 索引多个文档：如果不加文档id号，ES将自动为你生成文档id

curl -XPOST -H'content-type: application/json' http://localhost:9200/index/fulltext/1 -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
'
curl -XPOST -H'content-type: application/json' http://localhost:9200/index/fulltext/2 -d'
{"content":"公安部：各地校车将享最高路权"}
'
curl -XPOST -H'content-type: application/json' http://localhost:9200/index/fulltext/3 -d'
{"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}
'
curl -XPOST -H'content-type: application/json' http://localhost:9200/index/fulltext/4 -d'
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
'

4. 搜索测试：1）采用POST方法请求

curl -XPOST -H'content-type: application/json' http://localhost:9200/index/fulltext/_search  -d'
{
    "query" : { "match" : { "content" : "中国" }},
    "highlight" : {
        "pre_tags" : ["", ""],
        "post_tags" : ["", ""],
        "fields" : {
            "content" : {}
        }
    }
}
'

2）采用GET方法请求：

(1)q参数查询：curl http://localhost:9200/index/fulltext/_search?q=content:中国
(2)当前index + Multi-Type：curl http://localhost:9200/index/fulltext,fultxt2/_search?q=content:中国
(3)Multi-Index + 当前type：curl http://localhost:9200/index,index1/fulltext/_search?q=content:中国
(4)all indices + all types：curl http://localhost:9200/_search?q=content:中国&size=2
(5)当前index：curl http://localhost:9200/index/_search?q=*:中国

5. 查看索引： curl localhost:9200/_cat/indices?v

[miaohe@GZILXPMH02 ~]$ curl 'elasticsearch:9200/_cat/indices?v'
health status index            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   dashboard_syslog WrlJqmBuRjyRFhfEcU27TQ   5   1(副本)       6822            0       11mb           11mb

绿色：最健康的状态，代表所有的分片包括备份都可用

黄色：基本的分片可用，但是备份不可用（也可能是没有备份）

红色：部分的分片可用，表明分片有一部分损坏。此时执行查询部分数据仍然可以查到，遇到这种情况，还是赶快解决比较好。

6. 删除索引：curl -X DELETE elasticsearch:9200/dashboard_syslog

$ curl elasticsearch:9200/dashboard_syslog/_mapping?pretty
{}

7. cluster_block_exception：ES集群新节点的数据目录data存储空间不足，导致从master主节点接收同步数据的时候失败，为了保护数据，会自动把索引分片index置为只读read-only

{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}],"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"},"status":403}

此时，需解除锁定

curl -XPUT 'elasticsearch:9200/_settings' -H'Content-Type:application/json' -d'
{
    "index": {
    "blocks": {
    "read_only_allow_delete": "false"
    }
    }
}'

8. 批量删除文档：删除文档不会立即将文档从磁盘中删除，只是将文档标记为已删除状态。随着你不断的索引更多的数据，Elasticsearch 将会在后台清理标记为已删除的文档。

curl -XPOST 'elasticsearch:9200/test_dashboard_syslog/notype/_delete_by_query?pretty' -H'Content-Type:application/json' -d '{"query": {"bool": {"filter": {"range":{"MSTIME":{"lte":15361182000000}}}}}}'

query的结构形如：

{
  "query": {
    "bool": {
      "filter": {
        "range": {
          "score": {
            "lte": 60
          }
        }
      },
      "must": {
        "match": {
          "last_name": "Smith"
        }
      }
    }
  }
}

9. elasticdump：数据导入导出

 

非Docker容器中的Elasticsearch服务管理

1. 创建/etc/init.d/elasticsearch(可通过/etc/init.d/elasticsearch start来启动 elasticsearch)：

#!/bin/bash
#
# elasticsearch 

#
# chkconfig:   2345 80 20
# description: Starts and stops a single elasticsearch instance on this system
#

### BEGIN INIT INFO
# Provides: Elasticsearch
# Required-Start: $network $named
# Required-Stop: $network $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: This service manages the elasticsearch daemon
# Description: Elasticsearch is a very scalable, schema-free and high-performance search solution supporting multi-tenancy and near realtime search.
### END INIT INFO

#
# init.d / servicectl compatibility (openSUSE)
#
if [ -f /etc/rc.status ]; then
    . /etc/rc.status
    rc_reset
fi

#
# Source function library.
#
if [ -f /etc/rc.d/init.d/functions ]; then
    . /etc/rc.d/init.d/functions
fi

# Sets the default values for elasticsearch variables used in this script
ES_HOME="/usr/share/elasticsearch"
MAX_OPEN_FILES=65536
MAX_MAP_COUNT=262144
ES_PATH_CONF="/etc/elasticsearch"

PID_DIR="/var/run/elasticsearch"

# Source the default env file
ES_ENV_FILE="/etc/sysconfig/elasticsearch"
if [ -f "$ES_ENV_FILE" ]; then
    . "$ES_ENV_FILE"
fi

# ES_USER and ES_GROUP settings were removed
if [ ! -z "$ES_USER" ] || [ ! -z "$ES_GROUP" ]; then
    echo "ES_USER and ES_GROUP settings are no longer supported. To run as a custom user/group use the archive distribution of Elasticsearch."
    exit 1
fi

exec="$ES_HOME/bin/elasticsearch"
prog="elasticsearch"
pidfile="$PID_DIR/${prog}.pid"

export ES_JAVA_OPTS
export JAVA_HOME
export ES_PATH_CONF
export ES_STARTUP_SLEEP_TIME

lockfile=/var/lock/subsys/$prog

if [ ! -x "$exec" ]; then
    echo "The elasticsearch startup script does not exists or it is not executable, tried: $exec"
    exit 1
fi

checkJava() {
    if [ -x "$JAVA_HOME/bin/java" ]; then
        JAVA="$JAVA_HOME/bin/java"
    else
        JAVA=`which java`
    fi

    if [ ! -x "$JAVA" ]; then
        echo "Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME"
        exit 1
    fi
}

start() {
    checkJava
    [ -x $exec ] || exit 5

    if [ -n "$MAX_OPEN_FILES" ]; then
        ulimit -n $MAX_OPEN_FILES
    fi
    if [ -n "$MAX_LOCKED_MEMORY" ]; then
        ulimit -l $MAX_LOCKED_MEMORY
    fi
    if [ -n "$MAX_MAP_COUNT" -a -f /proc/sys/vm/max_map_count ]; then
        sysctl -q -w vm.max_map_count=$MAX_MAP_COUNT
    fi

    # Ensure that the PID_DIR exists (it is cleaned at OS startup time)
    if [ -n "$PID_DIR" ] && [ ! -e "$PID_DIR" ]; then
        mkdir -p "$PID_DIR" && chown elasticsearch:elasticsearch "$PID_DIR"
    fi
    if [ -n "$pidfile" ] && [ ! -e "$pidfile" ]; then
        touch "$pidfile" && chown elasticsearch:elasticsearch "$pidfile"
    fi

    cd $ES_HOME
    echo -n $"Starting $prog: "
    # if not running, start it up here, usually something like "daemon $exec"
    daemon --user elasticsearch --pidfile $pidfile $exec -p $pidfile -d
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    # stop it here, often "killproc $prog"
    killproc -p $pidfile -d 86400 $prog
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}
                              
restart() {
    stop
    start
}

reload() {
    restart
}

force_reload() {
    restart
}

rh_status() {
    # run checks to determine if the service is running or use generic status
    status -p $pidfile $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}


case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
                                                                                   
                status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
        restart
        ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
        exit 2
esac
exit $?
                                   
            

 

2. 创建/usr/lib/systemd/system/elasticsearch.service：

[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
RuntimeDirectory=elasticsearch
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/sysconfig/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet

# StandardOutput is configured to redirect to journalctl since
# some error messages may be logged in standard output before
# elasticsearch logging system is initialized. Elasticsearch
# stores its logs in /var/log/elasticsearch and does not use
# journalctl by default. If you also want to enable journalctl
# logging, you can simply remove the "quiet" option from ExecStart.
StandardOutput=journal
StandardError=inherit

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Specifies the maximum number of processes
LimitNPROC=4096

# Specifies the maximum size of virtual memory
LimitAS=infinity

# Specifies the maximum file size
LimitFSIZE=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0

# SIGTERM signal is used to stop the Java process
KillSignal=SIGTERM

# Send the signal only to the JVM rather than its control group
KillMode=process

# Java process is never killed
SendSIGKILL=no

# When a JVM receives a SIGTERM signal it exits with code 143
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target

# Built for distribution-6.2.3 (distribution)
                                                                                                                   

 

3. service elasticsearch [start | restart | status | stop]

4. systemctl -a：查看系统所有服务及状态

5. systemctl restart elasticsearch

转载于:https://www.cnblogs.com/XiongMaoMengNan/p/8073852.html