docker部署es+kibana+ik分词器

保持es、ik、kibana版本一致不然容器启动时会报错

一、启动es容器

先在https://github.com/medcl/elasticsearch-analysis-ik/releases下载es对应版本的ik分词器插件zip
然后解压到主机的/ysl/elasticsearch/plugins/ik目录下，最后一级目录名需要为ik
然后将该目录挂在给容器及下面指令中的
-v /ysl/elasticsearch/plugins:/usr/share/elasticsearch/plugins

docker pull elasticsearch:7.12.0

docker run -d \
	-v /ysl/elasticsearch/plugins:/usr/share/elasticsearch/plugins\
	-v /ysl/elasticsearch/data:/usr/share/elasticsearch/data\
	-v /ysl/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml\
	--name es \
	--net host \
	-e "discovery.type=single-node" \
	-e "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"\
	 elasticsearch:7.12.0

elasticsearch.yml默认内容如下

cluster.name: "docker-cluster"
network.host: 0.0.0.0

在yml后添加下列内容即可开启认证

http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

添加后进入容器执行下面的命令进入修改密码面板(暂未测试改在data目录后重启密码是否还保存)

cd /usr/share/elasticsearch/bin
elasticsearch-setup-passwords interactive

关于–net=host，加了该参数后就不需要再做端口映射了会自动将主机该端口映射给容器(当然直接通过 -p 8080:8080后docker ps -a 可以看到端口映射关系，会比较直观)

9300是集群节点指点的tcp通讯端口，9200是http协议的web客户端RESTful端口
discovery.type=single-node 表示，如果你正在使用单个节点开发，那就要添加这句话避开引导检查

二、启动kibana容器（默认端口为5601）

 docker pull kibana:7.12.0
 
docker run -d --name=kibana --env ELASTICSEARCH_HOSTS=http://es的ip:9200/ --net host kibana:7.12.0

–env ELASTICSEARCH_HOSTS=http://10.0.0.88:9200指定es的http的服务路径不指定会报

server is not ready yet

三、分词器

ik分词器

创建索引：

PUT /index_name

给索引添加mapping

PUT /index_name/_mapping?pretty
{
	  "properties" : {
		  "content" : {
				  "type" : "text",
				  "analyzer": "ik_max_word",
	        	  "search_analyzer": "ik_smart",
				  "fields" : {
					"keyword" : {
					  "type" : "keyword",
					  "ignore_above" : 256
					}
				  }
			}
		}
}

查询测试

POST /index_name/_search
{
    "query" : { "match" : { "content" : "中国" }},
    "highlight" : {
        "pre_tags" : ["", ""],
        "post_tags" : ["", ""],
        "fields" : {
            "content" : {}
        }
    }
}

可参考ik文档https://github.com/medcl/elasticsearch-analysis-ik

pinyin分词器

PUT  test_pinyin
{
  "settings" : {
    "analysis": {
      "analyzer": {
        "ik_pinyin_analyzer": {
          "tokenizer": "my_ik_pinyin",
          "filter": "pinyin_first_letter_and_full_pinyin_filter"
        },
        "pinyin_analyzer": {
          "tokenizer": "my_pinyin"
        }
      },
      "tokenizer": {
        "my_ik_pinyin": {
          "type": "ik_max_word"
        },
        "my_pinyin": {
          "type": "pinyin",
          "keep_first_letter": true,
          "keep_separate_first_letter": false,
          "keep_full_pinyin": false,
          "keep_joined_full_pinyin": true,
          "keep_none_chinese": true,
          "none_chinese_pinyin_tokenize": false,
          "keep_none_chinese_in_joined_full_pinyin": true,
          "keep_original": false,
          "limit_first_letter_length": 16,
          "lowercase": true,
          "trim_whitespace": true,
          "remove_duplicated_term": true
        }
      },
      "filter": {
        "pinyin_first_letter_and_full_pinyin_filter": {
          "type": "pinyin",
          "keep_first_letter": true,
          "keep_separate_first_letter": false,
          "keep_full_pinyin": false,
          "keep_joined_full_pinyin": true,
          "keep_none_chinese": true,
          "none_chinese_pinyin_tokenize": false,
          "keep_none_chinese_in_joined_full_pinyin": true,
          "keep_original": false,
          "limit_first_letter_length": 16,
          "lowercase": true,
          "trim_whitespace": true,
          "remove_duplicated_term": true
        }
      }
    }
  }
}
测试分词器
GET /test_pinyin/_analyze
{
  "analyzer": "ik_pinyin_analyzer", 
  "text": ["平安银行股份有限公司"]
}

四、logstash导入全量测试es

从mysql全表数据导入数据，logstash官网下载
下载解压后修改conf文件夹下的配置文件即可，此处新建mysql.conf文件（名字随意）,解压后的路径中不能存在中文
test.conf文件如下

input {
    stdin {
    }
    jdbc {
      # 数据库连接
      jdbc_connection_string => "jdbc:mysql://127.0.0.1:3360/test?characterEncoding=utf-8&useSSL=false&useUnicode=true&zeroDateTimeBehavior=convertToNull&allowMultiQueries=true&serverTimezone=Asia/Shanghai"
      # 用户名密码
      jdbc_user => "root"
      jdbc_password => "root"
      # 驱动jar包的位置
      jdbc_driver_library => "D:\mysql-connector-java-5.1.47.jar"
      # mysql的Driver
      jdbc_driver_class => "com.mysql.jdbc.Driver"
      jdbc_paging_enabled => "true"
      # 此处可以写查表sql，按照自己的需求查询出数据导入es,+8/24用于时区处理
      statement => "select * from test where update_time >= :sql_last_value + 8/24"
      # logstash确定启动时的时间点存储路径 （会从该路径加载sql_last_value的值）
      last_run_metadata_path => "D:\Tool\logstash\logstash-7.17.3\config\sql_last_value.txt"
	  #cron表达式
      schedule => "* * * * *"
	  clean_run => true
    }
}
 
#filter {
#    json {
#       source => "message"
#        remove_field => ["message"]
#    }
#}
 
output {
    elasticsearch {
        hosts => "http://127.0.0.1:9200/"
        # index名
        index => "index_name"
    # 需要关联的数据库中有有一个id字段，对应索引的id号
        document_id => "%{id}"
    }
    #stdout {
    #    codec => json_lines
    #}
}

然后bin目录下运行

logstash -f ../config/mysql.conf

docker启动挂载路径

logstash.yml

node.name: logstash-203
# 日志文件目录配置
path.logs: /usr/share/logstash/logs
# 验证配置文件及存在性
config.test_and_exit: false
# 配置文件改变时是否自动加载
config.reload.automatic: true
# 重新加载配置文件间隔
config.reload.interval: 60s
# debug模式 开启后会打印解析后的配置文件 包括密码等信息 慎用
# 需要同时配置日志等级为debug
config.debug: false
log.level: info
# The bind address for the metrics REST endpoint.
http.host: 0.0.0.0

pipelines.yml

 - pipeline.id: ent_base_info
   pipeline.workers: 4
   pipeline.batch.size: 2000
   queue.type: persisted
   path.config: "/usr/share/logstash/config/test.conf"

/usr/share/logstash/config:/挂载路径/config
/usr/share/logstash/pipeline:/挂载路径/pipeline
/usr/share/logstash/data:/挂载路径/data
/usr/share/logstash/logs:/挂载路径/logs
docker启动logstash可以参考文章[docker安装logstash]
(https://cloud.tencent.com/developer/article/1703928)

logstash同步mysql数据本质为定时任务，定时器去同步数据
canal是监听MySql的binlog日志，进而做到数据几乎实时同步

logstash github路径https://github.com/elastic/logstash/tree/7.12
canal github路径https://github.com/alibaba/canal

四、logstash导入日志至es

docker启动es,kibana

#1:来取镜像
docker pull logstash:7.12.1
#2:启动容器
docker run -d --name=logstash logstash:7.12.1
#3:将容器内的logstash 目录赋值至宿主机指定挂载目录（eg:/home/logstash-7.12.0）
docker cp logstash:/usr/share/logstash /home/logstash-7.12.0/
#4:创建配置文件目录
mkdir /home/logstash-7.12.0/logstash/config/conf.d
#5:赋予容器修改挂载目录的权限，否则启动会报错
chmod 777 -R /home/logstash-7.12.0/logstash
#6:设置/home/logstash-7.12.0/logstash/config/logstash.yml文件为
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://esip:es端口" ]
path.config: "/usr/share/logstash/config/conf.d/*.conf"
path.logs: /usr/share/logstash/logs
#7:在conf.d目录下创建.conf文件(eg:logback-es.conf)内容如下
input {
    tcp {
    mode => "server"
    #端口号
    port => 4567
    codec => json_lines
  }
}
output {
    elasticsearch {  
        hosts => "http://127.0.0.1:9200/"
		index => "app_log"
    } 
    stdout{
       codec => json_lines 
    }
}
#8:启动容器（挂载messages目录的目的是后续可以通过该目录来向es输出日志文件上传）
docker run -d \
  --name=logstash \
  --restart=always \
  --net host \
  -v /home/logstash-7.12.0/logstash:/usr/share/logstash \
  -v /home/log/messages:/var/log/messages \
  logstash:7.12.0